Emergency vehicle priority passing control method based on parameterized reinforcement learning

Abstract

The invention relates to the technical field of intelligent traffic, and discloses an emergency vehicle priority passing control method based on parameterized reinforcement learning, and the method comprises the following steps: firstly constructing an urban road network; according to the position of the emergency vehicle, using the established dynamic decision model to determine a target lane to which the vehicle is about to drive and the continuous driving time in the lane; then, a multi-agent dynamic collaborative lane changing model is used for controlling vehicles on a target lane to change lanes and give way; and finally, coupling the two models, and verifying the actual effect of the two models on an urban road with a certain length. Compared with the prior art, the method solves the problem that emergency vehicles on urban roads are blocked by social vehicles and cannot pass through, ensures that the emergency vehicles quickly pass through the road sections at low delay, reduces secondary accidents while minimizing interference to the social vehicles, and provides an efficient system architecture for priority passing of the emergency vehicles on the urban roads.

Description

Technical Field

[0001] The invention relates to the field of intelligent transportation technology, and mainly discloses an emergency vehicle priority passage control method based on parameterized reinforcement learning. Background Art

[0007] After an accident occurs, it is extremely important for emergency vehicles to quickly pass through the starting and ending points. In terms of medical assistance, the rapid passage of ambulances can send patients to the hospital for treatment, and can also arrive at the scene faster to implement rescue and preliminary treatment; after a fire occurs, fire trucks can quickly arrive to implement firefighting and rescue, and strive for hope of life for the victims. Based on autonomous driving technology, social vehicles can detect the appearance of emergency vehicles at the first time, and make corresponding adaptive operations based on their behavior, such as changing lanes to give way, automatically accelerating and driving away, etc., to maximize precious time for emergency rescue. At the same time, on the basis of autonomous driving vehicles equipped with advanced information and communication technology, vehicles can receive and transmit collaborative information with low latency, enhancing vehicle controllability and safety. However, there is still relatively little research on the rapid passage of emergency vehicles in urban traffic under the intelligent driving environment.

[0008] Therefore, the emergency vehicle priority control method based on parameterized reinforcement learning proposed in the present invention is very necessary, which can solve the problem of emergency vehicles passing safely and quickly without delay to reduce the secondary harm of accidents, and mainly starts from the following aspects. On the one hand, by deeply analyzing the control of autonomous driving vehicles under the deep reinforcement learning method, the mechanism of mutual coordination between emergency vehicles and social vehicles is realized, and a multi-agent dynamic collaborative lane change control model is established to consider sacrificing a small part of the interests of social vehicles to reduce the loss of people's lives and property; and considering the safe and fast driving of emergency vehicles, the number of lane changes of emergency vehicles is reduced on the basis of social vehicles changing lanes and reducing the travel time, and a dynamic decision model for emergency vehicles is established. This has important theoretical significance for improving the optimization control theory of emergency vehicles on urban roads. On the other hand, with the help of fully autonomous driving vehicles in the future, it is possible and necessary for emergency vehicles to respond quickly and pass through the passing sections; furthermore, by actively changing lanes and giving way to social vehicles, the number of lane changes of emergency vehicles is reduced, which not only improves driving safety, but also reduces the adverse effects on social vehicles. Summary of the invention

[0009] The present invention relates to the field of intelligent transportation and automatic driving control, and in particular to an emergency vehicle priority passage control method based on parameterized reinforcement learning, which is committed to solving the above limitations, facilitating emergency vehicles to pass through a certain section of a city road quickly and without delay, and minimizing interference with social vehicles.

[0010] To achieve the above object, the present invention adopts the following technical solutions:

[0011] The emergency vehicle priority control method based on parameterized reinforcement learning includes the following steps:

[0012] Step 1 is to establish a city road network and set the driving characteristic parameters of various types of vehicles;

[0013] In step 1, SUMO—Simulation of Urban Mobility is used. The urban road constructed by the simulation platform is 2,000 meters long, with three lanes for straight driving in one direction and a lane width of 3.2 meters. The vehicle travels from west to east, and the lane IDs are 2, 1, and 0 respectively. The emergency vehicle parameters are: the probability of appearing within a certain time range is 10%, the speed factor conforms to the normal distribution, the vehicle shape is an ambulance, the vehicle following model is the Intelligent Driver Model (IDM), and the emergency deceleration is -4.5 m / s 2 etc.; all social vehicles are considered to be self-driving vehicles, denoted as CAV, and the corresponding parameters are set as follows: the probability of appearing within a certain time range is 90%, the traffic flow is 1500 vehicles per hour, the vehicle arrival conforms to the Poisson distribution, the speed factor conforms to the normal distribution, the vehicle shape is a passenger car, the vehicle following model is the Intelligent Driver Model (IDM), and the emergency deceleration is -4.5m / s 2 , the vehicle color is blue, etc.

[0014] Step 2 is to establish a dynamic decision-making model for emergency vehicles and a dynamic cooperative lane-changing model for social vehicles respectively;

[0015] Step 2.1 describes the establishment of a dynamic decision-making model for emergency vehicles. In this layer of the model, the emergency vehicle is considered as an intelligent agent, and the multi-channel deep Q network reinforcement learning algorithm - MP-DQN is used to train it to select the target lane and the duration of driving in the lane. At the same time, the lane change strategy of the emergency vehicle to the target lane is constructed. The Markov decision construction model is described as follows: the first layer of the environment is simulated by a Markov decision process (MDP) M = {S, A, p, p0, γ, r}, where S is the state space, A is the action space, p is the Markov transition probability distribution, p0 is the transition probability distribution of the initial state, r(s, a) is the reward function, and γ∈[0,1] is the discount factor; the modeling process is as follows:

[0016] Step 1: The agent observes the current environment state, including dynamic traffic flow with time-varying characteristics and local dynamic vehicle state information;

[0017] Step 2: The agent uses a certain strategy to take specific actions and corresponding time parameters according to the current state;

[0018] Step 3: Use the reward function to measure the value of the agent taking a specific action in the current state;

[0019] Step 4: Train the agent’s dynamic decision-making ability, with the goal of maximizing the cumulative reward, and optimize the agent’s action selection strategy in each state.

[0020] The specific action space, state space, and reward function are described as follows:

[0021] Action Space Design:

[0022] The goal of the present invention is to reduce the travel time of emergency vehicles. Therefore, it is necessary to consider which lane to drive faster and safer according to the distribution of vehicles in other lanes ahead of the vehicle. Here, the front zone is demarcated according to the location of the emergency vehicle. length = 200 meters, by counting the number of vehicles N in each lane i , i∈(0,1,2); traffic density k b , Average speed When there is no vehicle in a lane in the interval, the average speed is set to v mesn_i =Emv speed , Emv speed Speed ​​of emergency vehicles; Lane clearance in represents the x-coordinate of the position of vehicle j+1 in lane i within the interval, represents the x-coordinate of the position of vehicle j in lane i in the interval, and the gap of the last vehicle in the interval is calculated as the distance from the vehicle to the end of the interval, which is recorded as: is the x-coordinate of the first vehicle in the interval, is the x-coordinate of the last vehicle in the interval;

[0023] The above statistics are for the state space in front of the emergency vehicle, with a total of three lanes, four features for each lane, and a total of 12 features. However, in this part, we also need to consider controlling the emergency vehicle to change lanes into the target lane with the help of other social vehicles. Therefore, in this part, the information of vehicles around the emergency vehicle is also included in the first-level state space, which is specifically described as: the emergency vehicle's own position Emv pos , speed Emv speed ; The position of the rear vehicle in the adjacent lane on the right relative to the emergency vehicle is the x coordinate of the vehicle behind the emergency vehicle, and its speed is the speed of the vehicle behind the emergency vehicle to the right; similarly, the x-coordinate of the right front vehicle speed The x coordinate of the left front vehicle speed The x coordinate of the left rear vehicle speed Therefore, this part has 12 features. Note that if there is no vehicle in the right rear, right front, left front, or left rear, the relative speed is set to 0, which is the same as the speed of the emergency vehicle, and the safe relative distance Dis security =51m, which is mainly calculated by vehicle dynamics. The calculation process is as follows:

[0024] The vehicle braking distance is calculated as:

[0025]

[0026] Vehicle sensor delay distance:

[0027] d declay =v*t delay =1.11m,

[0028] Safe distance:

[0029] d safe =d declay +d braking =55.97m,

[0030] Where v = 22.22m / s is the maximum speed limit of the road, a dece =4.5m / s is the maximum deceleration of social vehicles. Although the present invention controls an autonomous driving vehicle, there will still be delays in actual operation. Therefore, the system delay t delay = 0.05s, considering the vehicle length is 5m, therefore, the final safe distance between vehicles is Dis security =51m;

[0031] Reward function design:

[0032] In the first layer, the agent is an emergency vehicle. The ultimate goal of the present invention is to allow the emergency vehicle to pass the set road section faster. The local goal of the first layer is to maximize the distance moved by the emergency vehicle within a certain interval. Therefore, the reward is set to the same length as the entire road section Edge. length = 2000m and the initial position of the emergency vehicle when determining the interval Related, calculated:

[0033]

[0034] in is the real-time x-coordinate of the emergency vehicle’s position in the qth interval;

[0035] Parametric Action Space Design

[0036] The MP-DQN algorithm is used in the first layer. In this algorithm, discrete actions and corresponding continuous parameters need to be determined. In the present invention, the emergency vehicle is driven at a faster speed by lane selection. Therefore, the parameterized action space is A k ={(a k ,x k )|x k ∈X k ∈[100,360],for all k∈[0,2],where a k ∈(0,1,2) is a discrete action, i.e., the lane ID that the emergency vehicle is about to drive to, x k Corresponding to X k Select the time parameter in , i.e., the duration of the emergency vehicle’s continuous travel in lane k;

[0037] The model is trained through techniques such as experience replay and target network update. Based on the collected experience, the agent's strategy and value function are iteratively updated to adapt to the dynamic environment and improve the stability and efficiency of the training process.

[0038] Step 2.1 controls the emergency vehicle to change lanes and enter the target lane according to the set lane change strategy; according to the previous acquisition of status information, local information around the emergency vehicle can be obtained, including the relative position and speed of the left front, left rear, right front, and right rear vehicles and the emergency vehicle, and the target lane is also determined according to the first layer model. Then, based on this information, the emergency vehicle can be controlled to change lanes according to the designed lane change strategy. The lane change strategy is designed as follows:

[0039]

[0040]

[0041]

[0042] in is the distance between the emergency vehicle and the front or rear vehicle other in the target lane at time t, The safe distance between the emergency vehicle and the vehicle in front or behind the target lane; is the speed of the emergency vehicle at time t, is the speed of other vehicles at time t; τ is the safe headway; a other,max,brake is the maximum braking deceleration of other vehicles, is the braking deceleration of the emergency vehicle at time t; a emv,min,brake is the minimum deceleration; a emv,max,brake is the maximum braking deceleration of the emergency vehicle; v emv,max The maximum speed of the emergency vehicle on the road, which is set to 1.2 times the road speed limit here; when the distance between the emergency vehicle and the vehicle in front of the target lane does not meet the safety distance, it needs to accelerate until the requirement is met:

[0043]

[0044]

[0045] Similarly, the following vehicle should slow down until the clearance requirement is met:

[0046]

[0047] in is the speed of the vehicle ahead of the emergency vehicle in the target lane at the next moment, is the current speed of the vehicle in front of the emergency vehicle in the target lane, a other,acceleration is the acceleration of the social vehicle, a other,acceleration is the deceleration of social vehicles.

[0048] Step 2.2 establishes a multi-social vehicle cooperative lane-changing model. In this layer of the model, the social vehicles on the target lane within the interval determined by the emergency vehicle are considered as intelligent agents. The proximal policy optimization reinforcement learning algorithm - MAPPO is used to train social vehicles to perform lane changes based on local information while ensuring safety and efficiency. The model is established as a Markov decision. The first layer of the environment is simulated by a Markov decision process (MDP) M = {S, A, p, p0, γ, r}, where S is the state space, A is the action space, p is the Markov transition probability distribution, p0 is the transition probability distribution of the initial state, r(s, a) is the reward function, and γ∈[0,1] is the discount factor; the specific modeling process is as follows:

[0049] Step 1: Each agent observes the local environment in which it is currently located, including the dynamic traffic flow with time-varying characteristics and the status information of the surrounding local vehicles;

[0050] Step 2: Each agent uses a certain strategy to select a specific action based on the current state;

[0051] Step 3: Use the reward function to measure the value of the agent taking a specific action in a certain state;

[0052] Step 4: Train the dynamic decision-making capabilities of multiple agents, with the goal of maximizing the cumulative reward, and optimize the action selection strategies of all agents in each state corresponding to themselves.

[0053] The specific action space, state space, and reward function are described as:

[0054] Action Space Design All agents can randomly choose between changing lanes left, changing lanes right, staying, accelerating, and decelerating, so the action space is set to:

[0055] a t ∈A i ={turn left, turn right, keep, speed up, slow down}, then the set of all agent actions is A = A1×A2×···×A N , where A N It means there are N agents, corresponding to the action of the Nth agent;

[0056] State Space Design

[0057] The state S of agent i i is defined as a dimension The matrix of is the number of observed vehicles, and W is the number of features used to represent the vehicle status, including:

[0058] ispresent: a binary variable indicating whether other vehicles can be observed near the agent;

[0059] x: the longitudinal position of the observed vehicle relative to the agent;

[0060] y: lateral position of the observed vehicle relative to the agent;

[0061] v x : The lateral velocity of the observed vehicle relative to the agent;

[0062] v y : The longitudinal velocity of the observed vehicle relative to the agent;

[0063] The agent can only observe the nearest vehicle within 50 meters, so the state space is defined as:

[0064] S=S1×S2×···×S N ,

[0065] Reward function design

[0066] Reward function R iIt is the key to train the agent to follow the desired behavior. Since the goal is to train the agent to change lanes safely and efficiently, the reward return of the i-th agent at time step t is defined as follows:

[0067]

[0068] where w c , w speed , w success and w t The collision evaluation w c , speed impact evaluation w speed , lane change success evaluation w success and timeout for incomplete lane change evaluation w t The corresponding positive weight scalar, since safety is the most important criterion, this weight is much greater than other weights to give priority to safety; if a collision occurs, the collision assessment r c Set to -1, otherwise r c =0; The impact on the speed of the vehicle behind the target lane during the lane change process: r speed =min{v t+2 -v t ,0}; The reward for successful lane change is set to: Vehicles that fail to change lanes within the time parameters passed in the first layer will be penalized accordingly:

[0069] The second-layer multi-agent lane changing mainly receives the two parameters of lane clearing and total clearing time passed in by the first layer, determines the agent that needs to change lanes, and then calls the multi-agent lane changing model to implement the process of the agent changing lanes and giving way; finally, the model is trained through technologies such as experience replay and target network update, and the agent's strategy and value function are iteratively updated based on the collected experience to adapt to the dynamic environment and improve the stability and efficiency of the training process.

[0070] The model established in step 3 is coupled after training and the model effect is verified using a sliding interval on the set road section. The proposed sliding interval, specifically, is to determine the target lane to be driven to and the continuous driving time in the lane in the first layer model. In this step, it is determined according to the position of the emergency vehicle and a certain range in front of it. Then this range is defined as the interval. Then the emergency vehicle will enter the target lane according to the set lane change strategy. At the same time, the second layer will determine the vehicle-agent that needs to change lanes based on the interval and the target lane; the emergency vehicle drives forward while the social vehicles change lanes to give way until the time parameters determined by the first layer end. At this time, the interval is determined again according to the position of the emergency vehicle and a certain range in front of it. Note that the two intervals may overlap or may be separated by a distance; the two-layer model is repeatedly executed, and the emergency vehicle is finally driven out of the set section. Through continuous optimization, the road driving time and safety of the emergency vehicle can be minimized, and the adverse effects of the emergency vehicle on social vehicles can be reduced through active avoidance of social vehicles.

[0071] Compared with the existing urban road emergency vehicle priority technology, the present invention has the following outstanding features:

[0072] 1. The present invention controls the autonomous driving vehicle to realize the mechanism of mutual coordination between emergency vehicles and social vehicles, considers sacrificing a small part of the interests of social vehicles to reduce the loss of people's lives and property, and establishes a multi-agent dynamic collaborative lane-changing and yielding control model; and considers the safe and fast driving of emergency vehicles, reduces the number of lane changes of emergency vehicles and reduces the travel time on the basis of lane-changing and yielding of social vehicles, and establishes a dynamic decision-making model for emergency vehicles. This has important theoretical significance for improving the theory of optimal control of emergency vehicles on urban roads.

[0073] Second, the present invention considers that with the help of future fully autonomous driving vehicles, it is possible and necessary for emergency vehicles to respond quickly and pass through the passing sections; secondly, by having social vehicles actively change lanes to give way and thus reducing the number of lane changes of emergency vehicles, it not only improves driving safety but also reduces the adverse effects on social vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 The present invention is a flowchart of a specific implementation of the emergency vehicle priority control method based on parameterized reinforcement learning in the implementation of the present invention.

[0075] Figure 2 Schematic diagram of SUMO simulation platform parameters and file settings in an embodiment of the present invention.

[0076] Figure 3 Schematic diagram of the first-level dynamic decision-making of emergency vehicles in an embodiment of the present invention.

[0077] Figure 4 Schematic diagram of the lane changing strategy of an emergency vehicle in an embodiment of the present invention.

[0078] Figure 5 This is the second-level social vehicle cooperative lane-changing diagram in an embodiment of the present invention.

[0079] Figure 6 Schematic diagram of the sliding interval in an embodiment of the present invention. DETAILED DESCRIPTION

[0080] Next, the embodiments of the present invention are explained and described in detail in conjunction with the accompanying drawings of the present invention.

[0081] The present invention relates to the field of intelligent transportation and automatic driving control, and in particular to an emergency vehicle priority passage control method based on parameterized reinforcement learning, which is committed to solving the above limitations, facilitating emergency vehicles to pass through a certain section of a city road quickly and without delay, and minimizing interference with social vehicles.

[0082] To achieve the above object, the present invention adopts the following technical solutions:

[0083] The emergency vehicle priority control method based on parameterized reinforcement learning includes the following steps:

[0084] Step 1 is to establish a city road network and set the driving characteristic parameters of various types of vehicles;

[0085] like Figure 2 As shown in Figure 1, SUMO—Simulation of Urban Mobility is used in step 1. The urban road constructed by the simulation platform is 2000 meters long, with three lanes for straight driving in one direction and a lane width of 3.2 meters. The vehicle travels from west to east, and the lane IDs are 2, 1, and 0 respectively. The emergency vehicle parameters are: the probability of appearing within a certain time range is 10%, the speed factor conforms to the normal distribution, the vehicle shape is an ambulance, the vehicle following model is the intelligent driver model, namely the Intelligent Driver Model (IDM), and the emergency deceleration is -4.5m / s 2 etc.; all social vehicles are considered to be self-driving vehicles, denoted as CAV, and the corresponding parameters are set as follows: the probability of appearing within a certain time range is 90%, the traffic flow is 1500 vehicles per hour, the vehicle arrival conforms to the Poisson distribution, the speed factor conforms to the normal distribution, the vehicle shape is a passenger car, the vehicle following model is the Intelligent Driver Model (IDM), and the emergency deceleration is -4.5m / s 2 , the vehicle color is blue, etc.

[0086] Step 2 is to establish a dynamic decision-making model for emergency vehicles and a dynamic cooperative lane-changing model for social vehicles respectively;

[0087] Step 2.1 establishes a dynamic decision-making model for emergency vehicles. The specific implementation form is as follows: Figure 3 As shown in Figure 1, in this layer of the model, the emergency vehicle is considered as an intelligent agent, and the multi-channel deep Q network reinforcement learning algorithm - MP-DQN is used to train it to select the target lane and the duration of driving in the lane. At the same time, the lane change strategy of the emergency vehicle to the target lane is constructed, as shown in Figure 1. Figure 4 As shown in the figure, the Markov decision model is described as follows: the first layer environment is simulated by a Markov decision process (MDP) M = {S, A, p, p0, γ, r}, where S is the state space, A is the action space, p is the Markov transition probability distribution, p0 is the transition probability distribution of the initial state, r(s, a) is the reward function, and γ∈[0,1] is the discount factor; the modeling process is as follows:

[0088] Step 1: The agent observes the current environment state, including dynamic traffic flow with time-varying characteristics and local dynamic vehicle state information;

[0089] Step 2: The agent uses a certain strategy to take specific actions and corresponding time parameters according to the current state;

[0090] Step 3: Use the reward function to measure the value of the agent taking a specific action in the current state;

[0091] Step 4: Train the agent's dynamic decision-making ability, optimize the agent's action selection strategy in each state with the goal of maximizing the cumulative reward;

[0092] The specific action space, state space, and reward function are described as follows:

[0093] Action Space Design:

[0094] The goal of the present invention is to reduce the travel time of emergency vehicles. Therefore, it is necessary to consider which lane to drive faster and safer according to the distribution of vehicles in other lanes ahead of the vehicle. Here, the front zone is demarcated according to the location of the emergency vehicle. length = 200 meters, by counting the number of vehicles N in each lane i , i∈(0,1,2); traffic density k b , Average speed When there is no vehicle in a lane in the interval, the average speed is set to v mean_i =Emv speed , Emv speed Speed ​​of emergency vehicles; Lane clearance in represents the x-coordinate of the position of vehicle j+1 in lane i within the interval, represents the x-coordinate of the position of vehicle j in lane i in the interval, and the gap of the last vehicle in the interval is calculated as the distance from the vehicle to the end of the interval, which is recorded as: is the x-coordinate of the first vehicle in the interval, is the x-coordinate of the last vehicle in the interval;

[0095] The above statistics are for the state space in front of the emergency vehicle, with a total of three lanes, four features for each lane, and a total of 12 features. However, in this part, we also need to consider controlling the emergency vehicle to change lanes into the target lane with the help of other social vehicles. Therefore, in this part, the information of vehicles around the emergency vehicle is also included in the first-level state space, which is specifically described as: the emergency vehicle's own position Emv pos , speed Emv speed ; The position of the rear vehicle in the adjacent lane on the right relative to the emergency vehicle is the x coordinate of the vehicle behind the emergency vehicle, and its speed is the speed of the vehicle behind the emergency vehicle to the right; similarly, the x-coordinate of the right front vehicle speed The x coordinate of the left front vehicle speed The x coordinate of the left rear vehicle speed Therefore, this part has 12 features. Note that if there is no vehicle in the right rear, right front, left front, or left rear, the relative speed is set to 0, which is the same as the speed of the emergency vehicle, and the safe relative distance Dis security =51m, which is mainly calculated by vehicle dynamics. The calculation process is as follows:

[0096] The vehicle braking distance is calculated as:

[0097]

[0098] Vehicle sensor delay distance:

[0099] d declay =v*t delay =1.11m,

[0100] Safe distance:

[0101] d safe =d declay +d braking =55.97m,

[0102] Where v = 22.22m / s is the maximum speed limit of the road, a dece =4.5m / s is the maximum deceleration of social vehicles. Although the present invention controls an autonomous driving vehicle, there will still be delays in actual operation. Therefore, the system delay t delay = 0.05s, considering the vehicle length is 5m, therefore, the final safe distance between vehicles is Dis security =51m;

[0103] Reward function design:

[0104] In the first layer, the agent is an emergency vehicle. The ultimate goal of the present invention is to allow the emergency vehicle to pass the set road section faster. The local goal of the first layer is to maximize the distance moved by the emergency vehicle within a certain interval. Therefore, the reward is set to the same length as the entire road section Edge. length = 2000m and the initial position of the emergency vehicle when determining the interval Related, calculated:

[0105]

[0106] in is the real-time x-coordinate of the emergency vehicle’s position in the qth interval;

[0107] Parametric Action Space Design

[0108] The MP-DQN algorithm is used in the first layer. In this algorithm, discrete actions and corresponding continuous parameters need to be determined. In the present invention, the emergency vehicle is driven at a faster speed by lane selection. Therefore, the parameterized action space is A k ={(a k ,x k )|x k ∈X k ∈[100,360],for all k∈[0,2],where a k ∈(0,1,2) is a discrete action, i.e., the lane ID that the emergency vehicle is about to drive to, x k Corresponding to X k Select the time parameter in , i.e., the duration of the emergency vehicle’s continuous travel in lane k;

[0109] The model is trained through techniques such as experience replay and target network update. Based on the collected experience, the agent's strategy and value function are iteratively updated to adapt to the dynamic environment and improve the stability and efficiency of the training process.

[0110] Step 2.1 controls the emergency vehicle to change lanes and enter the target lane according to the set lane change strategy; according to the previous acquisition of status information, local information around the emergency vehicle can be obtained, including the relative position and speed of the left front, left rear, right front, and right rear vehicles and the emergency vehicle, and the target lane is also determined according to the first layer model. Then, based on this information, the emergency vehicle can be controlled to change lanes according to the designed lane change strategy. The lane change strategy is designed as follows:

[0111]

[0112]

[0113]

[0114] in is the distance between the emergency vehicle and the front or rear vehicle other in the target lane at time t, The safe distance between the emergency vehicle and the vehicle in front or behind the target lane; is the speed of the emergency vehicle at time t, is the speed of other vehicles at time t; τ is the safe headway; a other,max,brake is the maximum braking deceleration of other vehicles, is the braking deceleration of the emergency vehicle at time t; a emv,min,brake is the minimum deceleration; a emv,max,brake is the maximum braking deceleration of the emergency vehicle; v emv,max The maximum speed of the emergency vehicle on the road, which is set to 1.2 times the road speed limit here; when the distance between the emergency vehicle and the vehicle in front of the target lane does not meet the safety distance, it needs to accelerate until the requirement is met:

[0115]

[0116]

[0117] Similarly, the following vehicle should slow down until the clearance requirement is met:

[0118]

[0119] in is the speed of the vehicle ahead of the emergency vehicle in the target lane at the next moment, is the current speed of the vehicle in front of the emergency vehicle in the target lane, a other,acceleration is the acceleration of the social vehicle, a other,acceleration is the deceleration of social vehicles.

[0120] Step 2.2 establishes a multi-vehicle cooperative lane-changing model. The specific implementation form is as follows: Figure 5In this layer of the model, the social vehicles on the target lane within the interval determined by the emergency vehicle are considered as intelligent agents. The proximal policy optimization reinforcement learning algorithm - MAPPO is used to train social vehicles to perform lane changes based on local information while ensuring safety and efficiency. The model is established as a Markov decision. The first layer of the environment is simulated by a Markov decision process (MDP) M = {S, A, p, p0, γ, r}, where S is the state space, A is the action space, p is the Markov transition probability distribution, p0 is the transition probability distribution of the initial state, r(s, a) is the reward function, and γ∈[0,1] is the discount factor; the specific modeling process is as follows:

[0121] Step 1: Each agent observes the local environment in which it is currently located, including the dynamic traffic flow with time-varying characteristics and the status information of the surrounding local vehicles;

[0122] Step 2: Each agent uses a certain strategy to select a specific action based on the current state;

[0123] Step 3: Use the reward function to measure the value of the agent taking a specific action in a certain state;

[0124] Step 4: Train the dynamic decision-making capabilities of multiple agents, with the goal of maximizing the cumulative reward, and optimize the action selection strategies of all agents in each state corresponding to themselves;

[0125] The specific action space, state space, and reward function are described as follows:

[0126] Action Space Design All agents can randomly choose between changing lanes left, changing lanes right, staying, accelerating, and decelerating, so the action space is set to:

[0127] a t ∈A i ={turn left, turn right, keep, speed up, slow down}, then the set of all agent actions is A = A1×A2×···×A N , where A N It means there are N intelligent agents, corresponding to the action of the Nth intelligent agent;

[0128] State Space Design

[0129] The state S of agent i i is defined as a dimension The matrix of is the number of observed vehicles, and W is the number of features used to represent the vehicle status, including:

[0130] ispresent: a binary variable indicating whether other vehicles can be observed near the agent;

[0131] x: the longitudinal position of the observed vehicle relative to the agent;

[0132] y: lateral position of the observed vehicle relative to the agent;

[0133] v x : The lateral velocity of the observed vehicle relative to the agent;

[0134] v y : The longitudinal velocity of the observed vehicle relative to the agent;

[0135] The agent can only observe the nearest vehicle within 50 meters, so the state space is defined as:

[0136] S=S1×S2×···×S N ,

[0137] Reward function design

[0138] Reward function R i It is the key to train the agent to follow the desired behavior. Since the goal is to train the agent to change lanes safely and efficiently, the reward return of the i-th agent at time step t is defined as follows:

[0139]

[0140] where w c , w speed , w success and w t The collision evaluation w c , speed impact evaluation w speed , lane change success evaluation w success and timeout for incomplete lane change evaluation w t The corresponding positive weight scalar, since safety is the most important criterion, this weight is much greater than other weights to give priority to safety; if a collision occurs, the collision assessment r c Set to -1, otherwise r c =0; The impact on the speed of the vehicle behind the target lane during the lane change process: r speed =min{v t+2 -v t ,0}; The reward for successful lane change is set to: Vehicles that fail to change lanes within the time parameters passed in the first layer will be penalized accordingly:

[0141] The second-layer multi-agent lane changing mainly receives the two parameters of lane clearing and total clearing time passed in by the first layer, determines the agent that needs to change lanes, and then calls the multi-agent lane changing model to implement the process of the agent changing lanes and giving way; finally, the model is trained through technologies such as experience replay and target network update, and the agent's strategy and value function are iteratively updated based on the collected experience to adapt to the dynamic environment and improve the stability and efficiency of the training process.

[0142] The established model described in step 3 is coupled after training and the sliding interval is used on the set road section to verify the model effect. The proposed sliding interval is as follows Figure 6 As shown, specifically, the target lane to be driven into and the duration of driving in the lane are determined in the first-layer model. In this step, it is determined based on the position of the emergency vehicle and a certain range in front of it. Then this range is defined as an interval. Then the emergency vehicle will drive into the target lane according to the set lane change strategy. At the same time, the second layer will determine the vehicle that needs to change lanes based on the interval and the target lane - the intelligent agent; the emergency vehicle drives forward while the social vehicles change lanes to give way until the time parameters determined by the first layer are over. At this time, the interval is determined again based on the position of the emergency vehicle and a certain range in front of it. Note that the two intervals may overlap or may be separated by a distance; the two-layer model is repeatedly executed, and the emergency vehicle is finally driven out of the set section. Through continuous optimization, the road driving time and safety of the emergency vehicle can be minimized, and the adverse effects of the emergency vehicle on social vehicles can be reduced through active avoidance by social vehicles.

Claims

1. Emergency vehicle priority control method based on parameterized reinforcement learning, characterized in that: The steps are as follows: Step 1 is to establish a city road network and set the driving characteristic parameters of various types of vehicles; Step 2 is to establish a dynamic decision-making model for emergency vehicles and a dynamic cooperative lane-changing model for social vehicles respectively; Step 3 is to couple the established model after training and use the sliding interval to verify the model effect on the set road section; The two models established in step 2 are described as follows: Step 2.1 Establish a dynamic decision-making model for emergency vehicles. Initially, based on the location of the emergency vehicle, detect the number of vehicles in each lane in a certain interval ahead, traffic density, average speed, gaps between adjacent lanes, and relative positions and speeds of vehicles around the emergency vehicle. Then, use the multi-channel deep Q network reinforcement learning algorithm—MP-DQN—to train the emergency vehicle to select the target lane to be driven to and the time parameters for continuing to drive in the lane based on the current state. At the same time, control the vehicle to change lanes and drive into the target lane according to the set lane change strategy. Step 2.2 establishes a dynamic cooperative lane-changing model for multiple social vehicles. According to the target lane and time parameters determined by the upper-level model, in this model, the social vehicles in the target lane of the interval perform lane-changing and yielding operations within the specified time parameters. The proximal strategy optimization algorithm - PPO is mainly used to train multiple vehicles to perform lane changes.

2. The emergency vehicle priority control method based on parameterized reinforcement learning according to claim 1 is characterized in that: Step 1: Use SUMO—Simulation of Urban Mobility. The urban road constructed by the simulation platform is 2,000 meters long, with three lanes in one direction and a lane width of 3.2 meters. The vehicle travels from west to east, and the lane IDs are 2, 1, and 0 respectively. The emergency vehicle parameters are: the probability of appearing within a certain time range is 10%, the speed factor conforms to the normal distribution, the vehicle shape is an ambulance, the vehicle following model is the Intelligent Driver Model (IDM), and the emergency deceleration is -4.5 m / s. 2 etc.; all social vehicles are considered to be self-driving vehicles, denoted as CAV, and the corresponding parameters are set as follows: the probability of appearing within a certain time range is 90%, the traffic flow is 1500 vehicles per hour, the vehicle arrival conforms to the Poisson distribution, the speed factor conforms to the normal distribution, the vehicle shape is a passenger car, the vehicle following model is the Intelligent Driver Model (IDM), and the emergency deceleration is -4.5m / s 2 , the vehicle color is blue, etc.

3. The emergency vehicle priority control method based on parameterized reinforcement learning according to claim 1 is characterized in that: Step 2.1 establishes a dynamic decision-making model for emergency vehicles. In this layer of the model, the emergency vehicle is considered as an intelligent agent, and the multi-channel deep Q network reinforcement learning algorithm - MP-DQN is used to train it to select the target lane and the duration of driving in the lane. At the same time, the lane change strategy of the emergency vehicle to the target lane is constructed. The Markov decision construction model is described as follows: The first layer of the environment is simulated by a Markov decision process (MDP) M = {S, A, p, p0, γ, r}, where S is the state space, A is the action space, p is the Markov transition probability distribution, p0 is the transition probability distribution of the initial state, r(s, a) is the reward function, and γ∈[0,1] is the discount factor; the modeling process is as follows: Step 1: The agent observes the current environment state, including dynamic traffic flow with time-varying characteristics and local dynamic vehicle state information; Step 2: The agent uses a certain strategy to take specific actions and corresponding time parameters according to the current state; Step 3: Use the reward function to measure the value of the agent taking a specific action in the current state; Step 4: Train the agent’s dynamic decision-making ability, with the goal of maximizing the cumulative reward, and optimize the agent’s action selection strategy in each state.

4. The emergency vehicle priority control method based on parameterized reinforcement learning according to claim 3 is characterized in that: The specific action space, state space, and reward function are described as follows: 4.1 Action Space Design The goal of the present invention is to reduce the travel time of emergency vehicles. Therefore, it is necessary to consider which lane to drive faster and safer according to the distribution of vehicles in other lanes ahead of the vehicle. Here, the front zone is demarcated according to the location of the emergency vehicle. length = 200 meters, by counting the number of vehicles N in each lane i , i∈(0,1,2); traffic density k b , i∈(0,1,2); average vehicle speed j∈(0,···,N i ), i∈(0,1,2), when there is no car in a lane in the interval, the average speed is set to v mean_i =Emv speed , Emv speed Speed ​​of emergency vehicles; Lane clearance j∈(0,···,N i ), i∈(0,1,2), where represents the x-coordinate of the position of vehicle j+1 in lane i within the interval, represents the x-coordinate of the position of vehicle j in lane i in the interval, and the gap of the last vehicle in the interval is calculated as the distance from the vehicle to the end of the interval, which is recorded as: is the x-coordinate of the first vehicle in the interval, is the x-coordinate of the last vehicle in the interval; The above statistics are for the state space in front of the emergency vehicle, with a total of three lanes, four features for each lane, and a total of 12 features. However, in this part, we also need to consider controlling the emergency vehicle to change lanes into the target lane with the help of other social vehicles. Therefore, in this part, the information of vehicles around the emergency vehicle is also included in the first-level state space, which is specifically described as: the emergency vehicle's own position Emv pos , speed Emv speed ; The position of the rear vehicle in the adjacent lane on the right relative to the emergency vehicle is the x coordinate of the vehicle behind the emergency vehicle, and its speed is the speed of the vehicle behind the emergency vehicle to the right; similarly, the x-coordinate of the right front vehicle speed The x coordinate of the left front vehicle speed The x coordinate of the left rear vehicle speed Therefore, this part has 12 features. Note that if there is no vehicle in the right rear, right front, left front, or left rear, the relative speed is set to 0, which is the same as the speed of the emergency vehicle, and the safe relative distance Dis security =51m, which is mainly calculated by vehicle dynamics. The calculation process is as follows: The vehicle braking distance is calculated as: Vehicle sensor delay distance: d declay =v*t delay =1.11m, Safe distance: d safe =d declay +d braking =55.97m, Where v = 22.22m / s is the maximum speed limit of the road, a dece =4.5m / s is the maximum deceleration of social vehicles. Although the present invention controls an autonomous driving vehicle, there will still be delays in actual operation. Therefore, the system delay t delay = 0.05s, considering the vehicle length is 5m, therefore, the final safe distance between vehicles is Dis security =51m; 4.2 Reward Function Design In the first layer, the agent is an emergency vehicle. The ultimate goal of the present invention is to allow the emergency vehicle to pass the set road section faster. The local goal of the first layer is to maximize the distance moved by the emergency vehicle within a certain interval. Therefore, the reward is set to the same length as the entire road section Edge. length = 2000m and the initial position of the emergency vehicle when determining the interval q∈(0,···,n), we can calculate: in is the real-time x-coordinate of the emergency vehicle’s position in the qth interval; 4.3 Parameterized Action Space Design The MP-DQN algorithm is used in the first layer. In this algorithm, discrete actions and corresponding continuous parameters need to be determined. In the present invention, the emergency vehicle is driven at a faster speed by lane selection. Therefore, the parameterized action space is A k ={(a k ,x k )|x k ∈X k ∈[100,360],for all k∈[0,2],where a k ∈(0,1,2) is a discrete action, i.e., the lane ID that the emergency vehicle is about to drive to, x k Corresponding to X k Select the time parameter in , i.e., the duration of the emergency vehicle’s continuous travel in lane k; The model is trained through techniques such as experience replay and target network update. Based on the collected experience, the agent's strategy and value function are iteratively updated to adapt to the dynamic environment and improve the stability and efficiency of the training process.

5. The emergency vehicle priority control method based on parameterized reinforcement learning according to claim 1 is characterized in that: Step 2.1 controls the emergency vehicle to change lanes and enter the target lane according to the set lane change strategy; according to the previous acquisition of status information, local information around the emergency vehicle can be obtained, including the relative position and speed of the left front, left rear, right front, and right rear vehicles and the emergency vehicle, and the target lane is also determined according to the first layer model. Then, based on this information, the emergency vehicle can be controlled to change lanes according to the designed lane change strategy. The lane change strategy is designed as follows: in is the distance between the emergency vehicle and the front or rear vehicle other in the target lane at time t, The safe distance between the emergency vehicle and the vehicle in front or behind the target lane; is the speed of the emergency vehicle at time t, is the speed of other vehicles at time t; τ is the safe headway; a other,max,max is the maximum braking deceleration of other vehicles, is the braking deceleration of the emergency vehicle at time t; a emv,min,brake is the minimum deceleration; a emv,max,brake is the maximum braking deceleration of the emergency vehicle; v emv,max The maximum speed of emergency vehicles on the road, which is set here to 1.2 times the road speed limit; When the distance between the emergency vehicle and the vehicle ahead in the target lane does not meet the safety distance, it needs to accelerate until the requirement is met: Similarly, the following vehicle should slow down until the clearance requirement is met: in is the speed of the vehicle ahead of the emergency vehicle in the target lane at the next moment, is the current speed of the vehicle in front of the emergency vehicle in the target lane, a other,acceleration is the acceleration of the social vehicle, a other,acceleration is the deceleration of social vehicles.

6. The emergency vehicle priority control method based on parameterized reinforcement learning according to claim 1 is characterized in that: Step 2.2 establishes a multi-social vehicle cooperative lane-changing model. In this layer of the model, the social vehicles on the target lane within the interval determined by the emergency vehicle are considered as intelligent agents. The proximal policy optimization reinforcement learning algorithm - MAPPO is used to train social vehicles to perform lane changes based on local information while ensuring safety and efficiency. The model is established as a Markov decision. The first layer of the environment is simulated by a Markov decision process (MDP) M = {S, A, p, p0, γ, r}, where S is the state space, A is the action space, p is the Markov transition probability distribution, p0 is the transition probability distribution of the initial state, r(s, a) is the reward function, and γ∈[0,1] is the discount factor; the specific modeling process is as follows: Step 1: Each agent observes the local environment in which it is currently located, including the dynamic traffic flow with time-varying characteristics and the status information of the surrounding local vehicles; Step 2: Each agent uses a certain strategy to select a specific action based on the current state; Step 3: Use the reward function to measure the value of the agent taking a specific action in a certain state; Step 4: Train the dynamic decision-making capabilities of multiple agents, with the goal of maximizing the cumulative reward, and optimize the action selection strategies of all agents in each state corresponding to themselves.

7. The emergency vehicle priority control method based on parameterized reinforcement learning according to claim 6 is characterized in that: The specific action space, state space, and reward function are described as follows: 7.1 Action Space Design All agents can randomly choose between changing lanes left, changing lanes right, staying, accelerating, and decelerating, so the action space is set to: a t ∈A i ={turnleft,turn right,keep,speed up,slow down}, Then the set of all agent actions is A=A1×A2×···×A N , where A N It means there are N intelligent agents, corresponding to the action of the Nth intelligent agent; 7.2 State Space Design The state S of agent i i is defined as a dimension The matrix of is the number of observed vehicles, and W is the number of features used to represent the vehicle status, including: ispresent: a binary variable indicating whether other vehicles can be observed near the agent; x: the longitudinal position of the observed vehicle relative to the agent; y: lateral position of the observed vehicle relative to the agent; v x : The lateral velocity of the observed vehicle relative to the agent; v y : The longitudinal velocity of the observed vehicle relative to the agent; The agent can only observe the nearest vehicle within 50 meters, so the state space is defined as: S = S1×S2×···×S N , 7.3 Reward Function Design Reward function R i It is the key to train the agent to follow the desired behavior. Since the goal is to train the agent to change lanes safely and efficiently, the reward return of the i-th agent at time step t is defined as follows: where w c , w speed , w success and w t The collision evaluation w c , speed impact evaluation w speed , lane change success evaluation w success and timeout for incomplete lane change evaluation w t The corresponding positive weight scalar, since safety is the most important criterion, this weight is much greater than other weights to give priority to safety; if a collision occurs, the collision assessment r c Set to -1, otherwise r c =0; The impact on the speed of the vehicle behind the target lane during the lane change process: r speed =min{v t+2 -v t ,0}; The reward for successful lane change is set to: Vehicles that fail to change lanes within the time parameters passed in the first layer will be penalized accordingly: The second-layer multi-agent lane changing mainly receives the two parameters of lane clearing and total clearing time passed in by the first layer, determines the agent that needs to change lanes, and then calls the multi-agent lane changing model to implement the process of the agent changing lanes and giving way; finally, the model is trained through technologies such as experience replay and target network update, and the agent's strategy and value function are iteratively updated based on the collected experience to adapt to the dynamic environment and improve the stability and efficiency of the training process.

8. The emergency vehicle priority control method based on parameterized reinforcement learning according to claim 1 is characterized in that: The model established in step 3 is coupled after training and the model effect is verified using a sliding interval on the set road section. The proposed sliding interval, specifically, is to determine the target lane to be driven to and the continuous driving time in the lane in the first layer model. In this step, it is determined according to the position of the emergency vehicle and a certain range in front of it. Then this range is defined as the interval. Then the emergency vehicle will enter the target lane according to the set lane change strategy. At the same time, the second layer will determine the vehicle-agent that needs to change lanes based on the interval and the target lane; the emergency vehicle drives forward while the social vehicles change lanes to give way until the time parameters determined by the first layer end. At this time, the interval is determined again according to the position of the emergency vehicle and a certain range in front of it. Note that the two intervals may overlap or may be separated by a distance; the two-layer model is repeatedly executed, and the emergency vehicle is finally driven out of the set section. Through continuous optimization, the road driving time and safety of the emergency vehicle can be minimized, and the adverse effects of the emergency vehicle on social vehicles can be reduced through active avoidance of social vehicles.

Images