A ramp collaborative control system and method based on driving style

Abstract

The present invention discloses a ramp collaborative control system and method based on driving style, including: an on-board control subsystem, a vehicle-to-vehicle communication subsystem, and a collaborative decision-making subsystem based on driving style; the present invention fully considers the state and reaction decision of competing vehicles in the merging process, establishes a ramp collaborative merging interactive decision-making model, and designs a framework for multi-vehicle deep reinforcement learning; in order to consider the impact of random driving style on the decision-making process, the optimal merging strategy is defined based on Bayesian Nash equilibrium, and an experience pool is established through interactive learning with the environment to train and learn the optimal merging strategy. The method of the present invention can adjust the driving performance weights such as safety and efficiency based on driving style to achieve personalized driving and improve the acceptability of autonomous driving vehicles; the decision-making strategy is learned through interactive learning with the environment to improve the adaptability of the decision-making algorithm to dynamic scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent driving technology, and specifically relates to a ramp collaborative control system and method based on driving style. Background Art

[0002] Among the various causes of traffic congestion, ramp area vehicle merging is a typical cause, especially when ramp vehicles meet the rear vehicles on the main line. In the merging area, ramp vehicles try to merge with the main line, and there is a competition for right of way between ramp vehicles and main line vehicles, which causes vehicles to slow down or stop and go, or even conflict with each other.

[0003] In the current research on ramp cooperative control technology, most studies focus on optimizing the merging order of connected vehicles to improve the efficiency of traffic flow;

[0004] In terms of cooperative driving strategies, they can be divided into three categories, including: 1) Rule-based: The merging sequence in rule-based methods is usually defined by some heuristic rules, such as the "first in, first out" (FIFO) rule. Rule-based methods are computationally efficient, but cannot consider many possible merging sequences; 2) Optimal control-based: The optimal method evaluates all or part of the selected merging sequence through performance indicators related to traffic operation and determines the best sequence for the merging of entrance ramp vehicles. However, the computational complexity grows exponentially with the number of passing vehicles, which makes these strategies very difficult for real-time applications, especially for high traffic volume scenarios; 3) Game-based: Game theory provides useful tools to simulate situations where decision makers (called players) interact with each other. Each participant (referring to the interacting vehicles in the merging area) should have a strategy to take actions that bring greater utility value to the participant. Unfortunately, game-based methods lack adaptability to dynamic traffic (such as changes in traffic flow), and most existing studies ignore the impact of vehicle driving style on ramp merging strategies.

[0005] In addition, the driving style in the merging strategy is different from that in the common situation. Existing research on driving style mainly focuses on self-driving cars, where the information of driving style is known and determined, while the driving styles of surrounding vehicles are random, because the main line vehicles encountered by ramp vehicles are random, and the driving style cannot be directly measured by sensors. Summary of the invention

[0006] In view of the deficiencies of the above-mentioned prior art, the purpose of the present invention is to provide a ramp collaborative control system and method based on driving style to solve the problems of poor adaptability and difficulty in achieving safe and personalized driving in the existing ramp control technology; for ramp merging conditions, the present invention fully considers the state and reaction decisions of competing vehicles during the merging process, establishes a ramp collaborative merging interactive decision model, and designs a framework for multi-vehicle deep reinforcement learning; to consider the impact of random driving style on the decision-making process, the optimal merging strategy is defined based on Bayesian Nash equilibrium, and an experience pool is established through interactive learning with the environment to train and learn the optimal merging strategy. This method can adjust the driving performance weights such as safety and efficiency based on driving style to achieve personalized driving and improve the acceptability of autonomous driving vehicles; and learn decision strategies through interactive learning with the environment to improve the adaptability of decision algorithms to dynamic scenarios.

[0007] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0008] A ramp collaborative control system based on driving style of the present invention comprises: a vehicle-mounted control subsystem, a vehicle-to-vehicle communication subsystem and a collaborative decision-making subsystem based on driving style;

[0009] The vehicle communication subsystem is used to collect the traffic status S in the ramp merging area. t and sends it to the collaborative decision-making subsystem based on driving style; and receives the optimal control instruction a of the collaborative decision-making subsystem based on driving style t , and sent to the vehicle control subsystem;

[0010] The vehicle control subsystem, according to the optimal control instruction a sent by the vehicle communication subsystem t , driving the vehicle actuator to control the movement of the vehicle in the merging area;

[0011] The driving style-based collaborative decision-making subsystem is configured to determine the traffic status S sent by the vehicle-to-vehicle communication subsystem. t , establish a ramp collaborative merging interactive decision model under different driving styles, and analyze the driving benefits of vehicles in the merging area under different driving styles; based on the established collaborative merging interactive decision model, design a multi-vehicle deep reinforcement learning algorithm, establish a behavioral strategy network for competing vehicles, and convert the acquired traffic state S t The Bayesian Nash equilibrium combined driving action under the corresponding state is recorded in the driving experience pool as training data, optimizing the behavior strategy network parameters and outputting the optimal control command a under the given state. t .

[0012] Furthermore, the vehicles in the ramp merging area include: the ramp vehicle v0, the vehicle v1 closest to the merging vehicle on the main lane adjacent to the ramp, and the vehicles v1 in front and behind the vehicle v1. 1f 、v1r , the front and rear vehicles v1 adjacent to the vehicle v1 on the inner lane of the main lane 2f 、v 2r ;

[0013] The traffic state S t Including the motion state of vehicle i in the merging area and the relative motion between the two vehicles A collection of, that is:

[0014]

[0015] Among them, the motion state of vehicle i Includes: The longitudinal position of the vehicle in the road coordinate system Lateral position speed and yaw angle Relative motion state Including: relative longitudinal displacement Relative lateral displacement and relative speed

[0016] Furthermore, the optimal control instruction a t It is used to coordinate the control of the ramp vehicle v0 and the vehicle v1 on the main lane adjacent to the ramp that is closest to the merging vehicle. The control quantity includes: the longitudinal motion control command is the acceleration α t and lateral motion control command β t β t Indicates the lane-changing behavior of the vehicle. When β = 0, it indicates lane keeping behavior, and when β = 1, it indicates lane-changing behavior;

[0017] a t =(α t ,β t ).

[0018] Furthermore, the ramp collaborative merging interactive decision model under different driving styles includes: the behavior space under different driving styles Θ:=Θ0×Θ1 refers to the action space of vehicle 0, Refers to the action space of vehicle 1; the state transfer function T represents the motion state S based on the current vehicle t The predicted vehicle motion state after executing a certain behavior; and the profit functions R0 and R1 under different styles to evaluate the driving benefits of each behavior; Θ0 is the main road vehicle driving style, and Θ1 is the ramp vehicle driving style.

[0019] Furthermore, the different driving styles refer to the main road vehicle driving style θ0 and the ramp vehicle driving style θ1, including the aggressive type and conservative The driving style space of vehicle i is defined as,

[0020]

[0021] Aggressive means driving in an unsafe or impolite manner. For vehicles on the ramp, aggressive drivers tend to merge or accelerate in advance when entering the ramp to force vehicles on the main road to slow down and stop. For vehicles on the main road, aggressive drivers tend to speed up through the merge area when encountering merging vehicles to prevent merging vehicles from merging.

[0022] Analyze the driving style distribution of other vehicles based on Bayesian principle ri (θ -i |θ i ) to maximize the benefits of self-driving:

[0023]

[0024] In the formula, θ i is the driving style of vehicle i, θ -i is the driving style of vehicles other than vehicle i, P r represents probability;

[0025] The behavior space under different driving styles For the ramp merging scenario, the behavior space of the aggressive driver is i (θ 1 ) includes: Acceleration a 1 , change lane a 2 ; Behavior space of conservative style drivers a i (θ 2 ) includes: deceleration a 3 , lane keeping 4 ;

[0026]

[0027] a i (θ 1 )∈{a 1 ,a 2 = {acceleration, lane change}

[0028] a i (θ 2 )∈{a 3 ,a 4}={deceleration,keeping}.

[0029] Furthermore, the state transfer function T is based on the motion state S of all vehicles at the current time t t, according to the driving action a selected by each vehicle i,t , through the vehicle kinematic model, predict the motion state S of all vehicles at the next moment t+1 , used to evaluate the benefits of the currently selected behavior;

[0030] T:P r (S t+1 |S t ,a t )=P r (S t+1 |S t ,a 0,t ,a 1,t )

[0031] Among them, a 0,t 、a 1,t They represent the driving actions selected by vehicle 0 and vehicle 1 at time t respectively;

[0032] The vehicle kinematic model is based on a linear kinematic model, taking into account longitudinal and lateral motions; given a time interval T p , t+T p The motion state at a moment is calculated by the following formula:

[0033]

[0034] Among them, x t ,y t They represent the longitudinal position and lateral position at time t, respectively, and v xt ,v yt They represent the longitudinal and lateral speeds at time t, represents the heading angle at time t, represents t+T p The heading angle at time, T p represents the sampling time; α t Indicates that the longitudinal motion control command is acceleration, β t is the lateral motion control command, Δy is the lane change behavior T p The lateral displacement in the time domain,

[0035] Calculation of t+T based on mixed trigonometric lane-changing model p The lateral displacement and lateral velocity of the vehicle at time:

[0036]

[0037] Among them, d is the distance between the center line of the current lane and the target lane, and D is the longitudinal distance to complete the lane change action, which is a fixed value and is adjusted according to the vehicle speed or road structure.

[0038] Furthermore, the revenue function under different driving styles takes into account that vehicles with different driving styles have different priorities for driving performance, and different weights are designed in the reward function. The total revenue of vehicle i at time t is expressed as follows:

[0039]

[0040]

[0041] in, represents the collision safety benefit, represents the driving task completion benefit, Indicates driving efficiency benefits; a i (θ i ) indicates that when the driving style of vehicle i is θ i , the selected driving action; a -i (θ -i ) indicates that the driving style of vehicles other than vehicle i is θ -i When , the weight coefficient of different driving styles of the selected driving action is ω; represents the safety return weight, represents the driving task completion benefit weight, represents the driving efficiency benefit weight;

[0042] The collision safety benefit is evaluated by calculating the collision time between the vehicle and the preceding vehicle;

[0043]

[0044] Where, Δd i is the distance between vehicle i and the vehicle in front, Δv i is the speed difference between vehicle i and the preceding vehicle;

[0045] The driving task completion benefit considers whether the ramp vehicle can complete the merging task; whether the ramp vehicle has completed the merging task is determined based on the error between the vehicle's lateral position and heading angle and the lane centerline; the closer the ramp vehicle is to the main road centerline and the closer the heading angle is to 0, the greater the driving task completion benefit;

[0046]

[0047]

[0048] The driving efficiency benefit is to evaluate the driving efficiency of the selected behavior by calculating the time required to reach the target point, assuming that the vehicle maintains the currently selected behavior;

[0049]

[0050]

[0051] Where a represents the longitudinal acceleration of vehicle i; v i represents the speed of vehicle i; x i Indicates the longitudinal position of vehicle i. D1 represents the length of the acceleration lane in the ramp area, and D2 is used to represent the distance between the target point where the ramp vehicle completes the merge and the acceleration lane, which is set to 50m;

[0052] Calculate the expected return for subsequent solution of game equilibrium; determine the driving style θ of vehicle i i , for vehicle i, the action a is chosen i , other vehicles in the driving style distribution P ri (θ -i |θ i ) is calculated by the following formula:

[0053]

[0054] Furthermore, the multi-vehicle deep reinforcement learning algorithm is specifically as follows: each vehicle i is based on the current traffic state S t , randomly select a driving action Perform a joint action (a 0,t ,a 1,t ), observe the traffic state S at the next moment according to the state transfer function T and benefit function R in the above interactive decision model t+1 and the expected return E(R i,t+1 ), as a sample taken from the environment {S t ,a i,t ,E(R i,t+1 ),S i,t+1}, after multiple sampling, sufficient empirical samples are obtained and a set M is formed i ; Define the value function to evaluate the traffic status S at time t t and action (a 0,t ,a 1,t ) and learn the value function of each state and action from the experience samples obtained in the interaction process; in the actual decision-making process, according to the input state, the joint action with the largest utility is selected as the collaborative decision-making amount to control the movement of ramp vehicles and main road vehicles in the merging area;

[0055] The value function takes into account the randomness of each vehicle's behavior caused by the uncertainty of driving style, and the joint action of multi-vehicle decision-making (a 0,t ,a 1,t ) obeys the probability distribution π, also known as the decision strategy:

[0056]

[0057] Where π0 and π1 represent the driving action probability distributions of vehicle 0 and vehicle 1, respectively.

[0058] Furthermore, the total return during the sampling time under strategy π is Calculated by the following formula:

[0059]

[0060] Among them, R i,τ represents the return of vehicle i at time τ; γ∈[0,1) is the discount factor, which means that the return at a farther time has less impact on the current state value.

[0061] Furthermore, the value of the state S is defined as the expectation of future rewards obtained by executing the strategy π(a|S,θ) based on the current traffic state S:

[0062]

[0063] The action-value function of the policy is defined as:

[0064]

[0065] In particular, the present invention establishes a behavior strategy network of competing vehicles based on a deep neural network, learns the value function of each state and action, and uses samples M obtained by interacting with the environment i As training data, the network parameters are optimized; in the actual decision-making process, the action strategy corresponding to the optimal value of the output value function is used as the decision control quantity.

[0066] Furthermore, considering the uncertainty of driving style, the optimal value function is defined based on the Bayesian Nash equilibrium theory, and the equilibrium strategy is defined as the optimal strategy;

[0067] The optimal value function is defined by:

[0068]

[0069]

[0070]

[0071] in, represents the Bayesian Nash equilibrium strategy of vehicle i; V i * , Represents the optimal state value and action value of vehicle i under the equilibrium strategy.

[0072] In particular, in order to reduce the excessive fluctuation of model parameters caused by sample differences, two network models with exactly the same structure are established, namely the behavior value network and the target value network; by introducing the target value network, the model parameters for calculating the target value will be fixed for a certain period of time to reduce the volatility of the model; at the beginning of training, the two models use exactly the same parameters; during the training process, the target value is calculated by the target value network; then the state value under the Bayesian Nash equilibrium is solved and compared with the estimated value of the behavior value network to obtain the target value and update the network parameters of the behavior value network; whenever a certain number of iterations of training are completed, the parameters of the behavior value network model are synchronized to the target value network for the next stage of learning.

[0073] Furthermore, the loss function of the value function learning is defined as:

[0074]

[0075]

[0076] Among them, Q i,tar represents the target value, represents the value estimated by the behavior value network, It means that the target value network estimates the target value of each behavior and solves the optimal value after Nash equilibrium.

[0077] A ramp coordinated control method based on driving style of the present invention is based on the above system and comprises the following steps:

[0078] 1) Initialize the driving experience pool M of each vehicle i with a capacity of N i , N is a constant;

[0079] 2) Initialize the behavior value network Q of each vehicle i i (S) and network parameters

[0080] 3) Initialize the target value network for each vehicle i and network parameters

[0081] 4) Repeat steps 5)-15) for each process episode:

[0082] 5) Initialize the environment, obtain the initial state S1, and randomly initialize the optimal equilibrium strategy

[0083] 6) Repeat steps 8)-15) for each time t:

[0084] 7) According to the ∈-greedy exploration strategy, vehicle i follows the Bayesian Nash equilibrium strategy with the probability of ∈ From the current vehicle motion state S t Take action i,t , otherwise randomly choose an action a i,t ;

[0085] 8) Carry out joint actions (a) 0,t ,a 1,t ), based on the state transfer function T and the benefit function R in the combined interactive decision model, predict the state S at the next moment t+1 and expected report E(R i,t+1 );

[0086] 9) {S t ,a i,t ,E(R i,t+1 ),S i,t+1}Stored in driving experience pool M i middle;

[0087] 10) From the driving experience pool M i Take a batch of samples

[0088] 11) Calculate the target state value at the next moment

[0089]

[0090] 12) Use quadratic programming to solve the optimal state value under Bayesian Nash equilibrium at the next moment and Bayesian Nash equilibrium strategy

[0091] 13) Calculate the state value network output target value:

[0092]

[0093] 14) According to the loss function Use Adam optimizer to update the behavior value network Q i (S) Network Parameters

[0094] 15) Complete parameter updates every C rounds: C is a constant;

[0095] 16)End.

[0096] Beneficial effects of the present invention:

[0097] The present invention aims at ramp conditions, considers the influence of vehicle driving styles in the merging area on the collaborative merging strategy, and establishes an interactive decision-making model for mainline vehicles and ramp vehicles; designs the action space and benefit function of the vehicle according to different driving styles, realizes personalized driving under the premise of ensuring safety, and improves user acceptance of autonomous driving vehicles; designs a collaborative control algorithm based on multi-vehicle deep reinforcement learning, collects driving experience samples by interacting with the environment, and uses them as training data for learning decision model parameters, and solves the optimal strategy under Bayesian Nash equilibrium as the behavior output in the actual driving process, thereby improving the adaptability and reliability of the decision algorithm to dynamic scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] Figure 1 It is a framework diagram of the control system of the present invention.

[0099] Figure 2 This is a flow chart of the ramp collaborative control algorithm based on driving style proposed in the present invention.

[0100] Figure 3 This is a schematic diagram of a typical scenario for application of the control system proposed in the present invention. DETAILED DESCRIPTION

[0101] In order to facilitate the understanding of those skilled in the art, the present invention is further described below in conjunction with embodiments and drawings. The contents mentioned in the implementation modes are not intended to limit the present invention.

[0102] Reference Figure 1 As shown, a ramp collaborative control system based on driving style of the present invention comprises: a vehicle-mounted control subsystem, a vehicle-to-vehicle communication subsystem and a collaborative decision-making subsystem based on driving style;

[0103] The vehicle communication subsystem is used to collect the traffic status S in the ramp merging area. t and sends it to the collaborative decision-making subsystem based on driving style; and receives the optimal control instruction a of the collaborative decision-making subsystem based on driving style t , and sent to the vehicle control subsystem;

[0104] The vehicles in the ramp merging area include: the ramp vehicle v0, the vehicle v1 closest to the merging vehicle on the main lane adjacent to the ramp, and the vehicles v1 in front and behind the vehicle v1. 1f 、v 1r , the front and rear vehicles v1 adjacent to the vehicle v1 on the inner lane of the main lane 2f 、v 2r ;

[0105] The traffic state S t Including the motion state of vehicle i in the merging area and the relative motion between the two vehicles A collection of, that is:

[0106]

[0107] Among them, the vehicle i motion state Includes: The longitudinal position of the vehicle in the road coordinate system Lateral position speed and yaw angle Relative motion state Including: relative longitudinal displacement Relative lateral displacement and relative speed

[0108] The optimal control instruction a t It is used to coordinate the control of the ramp vehicle v0 and the vehicle v1 on the main lane adjacent to the ramp that is closest to the merging vehicle. The control quantity includes: the longitudinal motion control command is the acceleration α t and lateral motion control command β t β t Indicates the lane-changing behavior of the vehicle. When β = 0, it indicates lane keeping behavior, and when β = 1, it indicates lane-changing behavior;

[0109] a t =(α t ,β t ).

[0110] The vehicle control subsystem, according to the optimal control instruction a sent by the vehicle communication subsystem t , driving the vehicle actuator to control the movement of the vehicle in the merging area;

[0111] The driving style-based collaborative decision-making subsystem is configured to determine the traffic status S sent by the vehicle-to-vehicle communication subsystem. t , establish a ramp collaborative merging interactive decision model under different driving styles, and analyze the driving benefits of vehicles in the merging area under different driving styles; based on the established collaborative merging interactive decision model, design a multi-vehicle deep reinforcement learning algorithm, establish a behavioral strategy network for competing vehicles, and convert the acquired traffic state S t and Bayesian Nash equilibrium joint driving action in the corresponding state Recorded in the driving experience pool as training data, the behavior strategy network parameters are optimized, and the optimal control command a is output under a given state t ;

[0112] The ramp collaborative merging interactive decision model under different driving styles includes: the behavior space under different driving styles Θ:=Θ0×Θ1 refers to the action space of vehicle 0, Refers to the action space of vehicle 1; the state transfer function T represents the motion state S based on the current vehicle t The predicted vehicle motion state after executing a certain behavior; and the profit functions R0 and R1 under different styles to evaluate the driving benefits of each behavior; Θ0 is the main road vehicle driving style, and Θ1 is the ramp vehicle driving style.

[0113] The different driving styles refer to the main road vehicle driving style θ0 and the ramp vehicle driving style θ1, including the aggressive type and conservative The driving style space of vehicle i is defined as,

[0114]

[0115] Aggressive means driving in an unsafe or impolite manner. For vehicles on the ramp, aggressive drivers tend to merge or accelerate in advance when entering the ramp to force vehicles on the main road to slow down and stop. For vehicles on the main road, aggressive drivers tend to speed up through the merge area when encountering merging vehicles to prevent merging vehicles from merging.

[0116] Analyze the driving style distribution of other vehicles based on Bayesian principle ri (θ -i |θ i ) to maximize the benefits of self-driving:

[0117]

[0118] In the formula, θ i is the driving style of vehicle i, θ -i is the driving style of vehicles other than vehicle i, P r represents probability;

[0119] The behavior space under different driving styles For the ramp merging scenario, the behavior space of the aggressive driver is i (θ 1 ) includes: Acceleration a 1 , change lane a 2 ; Behavior space of conservative style drivers a i (θ 2 ) includes: deceleration a 3 , lane keeping 4 ;

[0120]

[0121] a i (θ 1 )∈{a 1 ,a 2= {acceleration, lane change}

[0122] a i (θ 2 )∈{a 3 ,a 4}={deceleration,keeping}.

[0123] The state transfer function T is based on the motion state S of all vehicles at the current time t. t , according to the driving action a selected by each vehicle i,t , through the vehicle kinematic model, predict the motion state S of all vehicles at the next moment t+1 , used to evaluate the benefits of the currently selected behavior;

[0124] T:P r (S t+1 |S t ,a t )=P r (S t+1 |S t ,a 0,t ,a 1,t )

[0125] Among them, a 0,t 、a 1,t They represent the driving actions selected by vehicle 0 and vehicle 1 at time t respectively;

[0126] The vehicle kinematic model is based on a linear kinematic model, taking into account longitudinal and lateral motions; given a time interval T p , t+T p The motion state at a moment is calculated by the following formula:

[0127]

[0128] Among them, x t ,y t They represent the longitudinal position and lateral position at time t, respectively, and v xt ,v yt They represent the longitudinal and lateral speeds at time t, represents the heading angle at time t, represents t+T p The heading angle at time, T p represents the sampling time; α t Indicates that the longitudinal motion control command is acceleration, β t is the lateral motion control command, Δy is the lane change behavior T p The lateral displacement in the time domain,

[0129] Calculation of t+T based on mixed trigonometric function lane-changing model p The lateral displacement and lateral velocity of the vehicle at time:

[0130]

[0131] Among them, d is the distance between the center line of the current lane and the target lane, and D is the longitudinal distance to complete the lane change action, which is a fixed value and is adjusted according to the vehicle speed or road structure.

[0132] The profit function under different driving styles takes into account the different priorities of vehicles with different driving styles on driving performance. Different weights are designed in the reward function. The total profit of vehicle i at time t is expressed as follows:

[0133]

[0134]

[0135] in, represents the collision safety benefit, represents the driving task completion benefit, Represents the driving efficiency benefit; a i (θ i ) indicates that when the driving style of vehicle i is θ i , the selected driving action; represents the safety return weight, represents the driving task completion benefit weight, represents the driving efficiency benefit weight; a -i (θ -i ) indicates that the driving style of vehicles other than vehicle i is θ -i When the driving action is selected, the weight coefficient ω of different driving styles is given by the following Table 1:

[0136] Table 1

[0137]

[0138]

[0139] The collision safety benefit is evaluated by calculating the collision time between the vehicle and the preceding vehicle;

[0140]

[0141] Where, Δd i is the distance between vehicle i and the vehicle in front, Δv i is the speed difference between vehicle i and the preceding vehicle;

[0142] The driving task completion benefit considers whether the ramp vehicle can complete the merging task; whether the ramp vehicle has completed the merging task is determined based on the error between the vehicle's lateral position and heading angle and the lane centerline; the closer the ramp vehicle is to the main road centerline and the closer the heading angle is to 0, the greater the driving task completion benefit;

[0143]

[0144]

[0145] The driving efficiency benefit is to evaluate the driving efficiency of the selected behavior by calculating the time required to reach the target point, assuming that the vehicle maintains the currently selected behavior;

[0146]

[0147]

[0148] Calculate the expected return for subsequent solution of game equilibrium; determine the driving style θ of vehicle i i , for vehicle i, the action a is chosen i , other vehicles in the driving style distribution P ri (θ -i |θ i ) is calculated by the following formula:

[0149]

[0150] The multi-vehicle deep reinforcement learning algorithm is specifically as follows: each vehicle i is based on the current traffic state S t , randomly select a driving action Perform a joint action (a 0,t ,a 1,t ), observe the traffic state S at the next moment according to the state transfer function T and benefit function R in the above interactive decision model t+1 and the expected return E(R i,t+1 ), as a sample taken from the environment {S t ,a i,t ,E(R i,t+1 ),S i,t+1}, after multiple sampling, sufficient empirical samples are obtained and a set M is formed i ; Define the value function to evaluate the traffic status S at time t t and action (a 0,t ,a 1,t) and learn the value function of each state and action from the experience samples obtained in the interaction process; in the actual decision-making process, according to the input state, the joint action with the largest utility is selected as the collaborative decision-making amount to control the movement of ramp vehicles and main road vehicles in the merging area;

[0151] The value function takes into account the randomness of each vehicle's behavior caused by the uncertainty of driving style, and the joint action of multi-vehicle decision-making (a 0,t ,a 1,t ) obeys the probability distribution π, also known as the decision strategy:

[0152]

[0153] Where π0 and π1 represent the driving action probability distributions of vehicle 0 and vehicle 1, respectively.

[0154] The total return during the sampling period under policy π Calculated by the following formula:

[0155]

[0156] Among them, R i,t represents the return of vehicle i at time τ; γ∈[0,1) is the discount factor, which means that the return at a farther time has less impact on the current state value.

[0157] The value of the state S is defined as the expected future return obtained by executing the strategy π(a|S,θ) based on the current traffic state S:

[0158]

[0159] The action-value function of the policy is defined as:

[0160]

[0161] In particular, the present invention establishes a behavior strategy network of competing vehicles based on a deep neural network, learns the value function of each state and action, and uses samples M obtained by interacting with the environment i As training data, the network parameters are optimized; in the actual decision-making process, the action strategy corresponding to the optimal value of the output value function is used as the decision control quantity.

[0162] Specifically, considering the uncertainty of driving style, the optimal value function is defined based on the Bayesian Nash equilibrium theory, and the equilibrium strategy is defined as the optimal strategy;

[0163] The optimal value function is defined by:

[0164]

[0165]

[0166]

[0167] in, represents the Bayesian Nash equilibrium strategy of vehicle i; V i * , Represents the optimal state value and action value of vehicle i under the equilibrium strategy.

[0168] In particular, in order to reduce the excessive fluctuation of model parameters caused by sample differences, two network models with exactly the same structure are established, namely the behavior value network and the target value network; by introducing the target value network, the model parameters for calculating the target value will be fixed for a certain period of time to reduce the volatility of the model; at the beginning of training, the two models use exactly the same parameters; during the training process, the target value is calculated by the target value network; then the state value under the Bayesian Nash equilibrium is solved and compared with the estimated value of the behavior value network to obtain the target value and update the network parameters of the behavior value network; whenever a certain number of iterations of training are completed, the parameters of the behavior value network model are synchronized to the target value network for the next stage of learning.

[0169] The loss function of the value function learning is defined as:

[0170]

[0171]

[0172] Among them, Q i,tar represents the target value, represents the value estimated by the behavior value network, It means that the target value network estimates the target value of each behavior and solves the optimal value after Nash equilibrium.

[0173] Reference Figure 2 As shown, a ramp coordinated control method based on driving style of the present invention is based on the above system, and the steps are as follows:

[0174] 1) Initialize the driving experience pool M of each vehicle i with a capacity of N i , N is a constant;

[0175] 2) Initialize the behavior value network Q of each vehicle i i (S) and network parameters

[0176] 3) Initialize the target value network for each vehicle i and network parameters

[0177] 4) Repeat steps 5)-15) for each process episode:

[0178] 5) Initialize the environment, obtain the initial state S1, and randomly initialize the optimal equilibrium strategy

[0179] 6) Repeat steps 8)-15) for each time t:

[0180] 7) According to the ∈-greedy exploration strategy, vehicle i follows the Bayesian Nash equilibrium strategy with the probability of ∈ From the current vehicle motion state S t Take action i,t , otherwise randomly choose an action a i,t ;

[0181] 8) Carry out joint actions (a) 0,t ,a 1,t ), according to the state transfer function T and the benefit function R in the combined interactive decision model, the state S at the next moment is obtained t+1 and expected report E(R i,t+1 );

[0182] 9) {S t ,a i,t ,E(R i,t+1 ),S i,t+1}Stored in driving experience pool M i middle;

[0183] 10) From the driving experience pool M i Take a batch of samples

[0184] 11) Calculate the target state value at the next moment

[0185]

[0186] 12) Use quadratic programming to solve the optimal state value under Bayesian Nash equilibrium at the next moment and Bayesian Nash equilibrium strategy

[0187] 13) Calculate the state value network output target value:

[0188]

[0189] 14) According to the loss function Use Adam optimizer to update the behavior value network Q i (S) Network Parameters

[0190] 15) Complete parameter update every C rounds: C is a constant;

[0191] 16)End.

[0192] Consider Figure 3 In the typical ramp merging area shown in the figure, vehicle V0 is a vehicle on the ramp and needs to complete the merging task and drive to the main lane. When vehicle V0 drives to the main lane and reaches the target position as shown in the figure, the merging task is considered to be completed. When ramp vehicle V0 reaches the observation point, that is, the end of the ramp in the figure, the rear vehicle on the main lane is defined as vehicle V1. When vehicle V0 performs the merging task, it needs to compete with vehicle V1 for the right of way, consider its reaction, and select the optimal driving action.

[0193] The present invention has many specific application paths. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements can be made without departing from the principle of the present invention. These improvements should also be regarded as the protection scope of the present invention.

Claims

1. A ramp cooperative control system based on driving style, comprising: On-board control subsystem, vehicle-to-vehicle communication subsystem, and collaborative decision-making subsystem based on driving style; The vehicle communication subsystem is used to collect the traffic status S in the ramp merging area. t And send it to the collaborative decision-making subsystem based on driving style; and receiving optimal control instructions from the collaborative decision-making subsystem based on driving style and sending them to the vehicle control subsystem; The vehicle control subsystem, according to the optimal control instruction a sent by the vehicle communication subsystem t , driving the vehicle actuator to control the movement of the vehicle in the merging area; The driving style-based collaborative decision-making subsystem is configured to determine the traffic status S sent by the vehicle-to-vehicle communication subsystem. t , establish a ramp collaborative merging interactive decision model under different driving styles, and analyze the driving benefits of vehicles in the merging area under different driving styles; based on the established collaborative merging interactive decision model, design a multi-vehicle deep reinforcement learning algorithm, establish a behavioral strategy network for competing vehicles, and convert the acquired traffic state S t The Bayesian Nash equilibrium combined driving action under the corresponding state is recorded in the driving experience pool as training data, optimizing the behavior strategy network parameters and outputting the optimal control command a under the given state. t ; The ramp collaborative merging interactive decision model under different driving styles includes: behavior space under different driving styles Θ:=Θ0×Θ1 refers to the action space of vehicle 0, Refers to the action space of vehicle 1; the state transfer function T represents the motion state S based on the current vehicle t The predicted vehicle motion state after executing a certain behavior; and the profit functions R0 and R1 under different styles to evaluate the driving benefits of each behavior; Θ0 is the main road vehicle driving style, and Θ1 is the ramp vehicle driving style; The different driving styles refer to the main road vehicle driving style θ0 and the ramp vehicle driving style θ1, including the aggressive type θ i 1 and conservative The driving style space of vehicle i is defined as: Aggressive means driving in an unsafe or impolite manner. For vehicles on the ramp, aggressive drivers tend to merge or accelerate in advance when entering the ramp to force vehicles on the main road to slow down and stop. For vehicles on the main road, aggressive drivers tend to speed up through the merge area when encountering merging vehicles to prevent merging vehicles from merging. Analyze the driving style distribution of other vehicles based on Bayesian principle ri (θ -i |θ i ) to maximize the benefits of self-driving: In the formula, θ i is the driving style of vehicle i, θ -i is the driving style of vehicles other than vehicle i, P r represents probability; The behavior space under different driving styles For the ramp merging scenario, the behavior space of the aggressive driver is i (θ 1 ) includes: Acceleration a 1 , change lane a 2 ; Behavior space of conservative style drivers a i (θ 2 ) includes: deceleration a 3 , lane keeping 4 ; a i (θ 1 )∈{a 1 ,a 2 }={acceleration,lanechange} a i (θ 2 )∈{a 3 ,a 4 }={deceleration,keeping}; The state transfer function T is based on the motion state S of all vehicles at the current time t. t , according to the driving action a selected by each vehicle i,t , through the vehicle kinematic model, predict the motion state S of all vehicles at the next moment t+1 , used to evaluate the benefits of the currently selected behavior; T:P r (S t+1 |S t ,a t )=P r (S t+1 |S t ,a 0,t ,a 1,t ) Among them, a 0,t 、a 1,t They represent the driving actions selected by vehicle 0 and vehicle 1 at time t respectively; The vehicle kinematic model is based on a linear kinematic model, taking into account longitudinal and lateral motions; given a time interval T p , t+T p The motion state at a moment is calculated by the following formula: Among them, x t ,y t They represent the longitudinal position and lateral position at time t, respectively, and v xt ,v yt They represent the longitudinal and lateral speeds at time t, represents the heading angle at time t, represents t+T p The heading angle at time, T p represents the sampling time; α t Indicates that the longitudinal motion control command is acceleration, β t is the lateral motion control command, Δy is the lane change behavior T p Lateral displacement in the time domain, calculated based on the hybrid trigonometric lane-changing model t+T p The lateral displacement and lateral velocity of the vehicle at time: Where d is the distance between the centerline of the current lane and the target lane, and D is the longitudinal distance to complete the lane change action, which is a fixed value and is adjusted according to the vehicle speed or road structure; The profit function under different driving styles takes into account the different priorities of vehicles with different driving styles on driving performance. Different weights are designed in the reward function. The total profit of vehicle i at time t is expressed as follows: in, represents the collision safety benefit, represents the driving task completion benefit, Represents the driving efficiency benefit; a i (θ i ) indicates that when the driving style of vehicle i is θ i , the selected driving action; a -i (θ -i ) indicates that the driving style of vehicles other than vehicle i is θ -i When , the weight coefficient of different driving styles of the selected driving action is ω; represents the safety return weight, represents the driving task completion benefit weight, represents the driving efficiency benefit weight; The collision safety benefit is evaluated by calculating the collision time between the vehicle and the preceding vehicle; Where, Δd i is the distance between vehicle i and the vehicle in front, Δv i is the speed difference between vehicle i and the preceding vehicle; The driving task completion benefit considers whether the ramp vehicle can complete the merging task; whether the ramp vehicle has completed the merging task is determined based on the error between the vehicle's lateral position and heading angle and the lane centerline; the closer the ramp vehicle is to the main road centerline and the closer the heading angle is to 0, the greater the driving task completion benefit; Among them, ω y , Represent the weight coefficients of lateral displacement and heading angle respectively; y tar represents the lateral position of the desired path; y i represents the lateral position of vehicle i; represents the heading angle of vehicle i; The driving efficiency benefit is to evaluate the driving efficiency of the selected behavior by calculating the time required to reach the target point, assuming that the vehicle maintains the currently selected behavior; Where a represents the longitudinal acceleration of vehicle i; v i represents the speed of vehicle i; x i Indicates the longitudinal position of vehicle i; D1 indicates the length of the acceleration lane in the ramp area, and D2 is used to indicate the distance between the target point where the ramp vehicle completes the merge and the acceleration lane; Calculate the expected return for subsequent solution of game equilibrium; determine the driving style θ of vehicle i i , for vehicle i, the action a is chosen i , other vehicles in the driving style distribution P ri (θ -i |θ i ) is calculated by the following formula:

2. The ramp cooperative control system based on driving style according to claim 1, characterized in that: The vehicles in the ramp merging area include: ramp vehicle v0, vehicle v1 on the main lane adjacent to the ramp and closest to the merging vehicle, and vehicles v1 in front of and behind vehicle v1. 1f 、v 1r , the front and rear vehicles v1 adjacent to the vehicle v1 on the inner lane of the main lane 2f 、v 2r ; The traffic state S t Including the motion state of vehicle i in the merging area and the relative motion between the two vehicles A collection of, that is: Among them, the vehicle i motion state Includes: The longitudinal position of the vehicle in the road coordinate system Lateral position speed and heading angle Relative motion state Including: relative longitudinal displacement Relative lateral displacement and relative speed 3. The ramp cooperative control system based on driving style according to claim 2, characterized in that: The optimal control instruction a t It is used to coordinate the control of the ramp vehicle v0 and the vehicle v1 on the main lane adjacent to the ramp that is closest to the merging vehicle. The control quantity includes: the longitudinal motion control command is the acceleration α t and lateral motion control command β t β t Indicates the lane-changing behavior of the vehicle. When β = 0, it indicates lane keeping behavior, and when β = 1, it indicates lane-changing behavior; a t ＝(α t ,β t )。 4. The ramp cooperative control system based on driving style according to claim 1, characterized in that: The multi-vehicle deep reinforcement learning algorithm is specifically as follows: each vehicle i is based on the current traffic state S t , randomly select a driving action Perform a joint action (a 0,t ,a 1,t ), observe the traffic state S at the next moment according to the state transfer function T and benefit function R in the above interactive decision model t+1 and the expected return E(R i,t+1 ), as a sample taken from the environment {S t ,a i,t ,E(R i,t+1 ),S i,t+1 }, after multiple sampling, sufficient empirical samples are obtained and a set M is formed i ; Define the value function to evaluate the traffic status S at time t t and action (a 0,t ,a 1,t )’s reward value, and learns the value function of each state and action from the experience samples obtained during the interaction; In the actual decision-making process, according to the input state, the joint action with the greatest utility is selected as the collaborative decision-making quantity to control the movement of ramp vehicles and main road vehicles in the merging area; The value function takes into account the randomness of each vehicle's behavior caused by the uncertainty of driving style, and the joint action of multi-vehicle decision-making (a 0,t ,a 1,t ) obeys the probability distribution π, also known as the decision strategy: Where π0 and π1 represent the driving action probability distributions of vehicle 0 and vehicle 1, respectively.

Images