A hierarchical decision-making control method for autonomous driving in ramp merging scenarios

Abstract

The present invention relates to an autonomous driving hierarchical decision control method for ramp merging scenarios, which belongs to the technical field of autonomous driving vehicles, and includes the following steps: S1: constructing an interactive highway parallel ramp merging scenario, using a two-degree-of-freedom kinematic model to describe the movement of vehicles in the training scenario, and using a rule-based intelligent decision control model to control other vehicles, so that the environment is interactive; S2: constructing a decision model based on maximum entropy deep reinforcement learning; S3: constructing a trajectory planning algorithm based on polynomial curves; S4: constructing an action mapping function and path tracking control model based on ramp scenarios, converting vehicle trajectories and actions into Cartesian coordinate systems, and tracking trajectories through a PID algorithm to control vehicle movement; S5: training the decision model based on maximum entropy deep reinforcement learning. The present invention reduces the overall training difficulty of reinforcement learning and improves the stability and safety of ramp merging of autonomous driving vehicles.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving vehicles and relates to an autonomous driving hierarchical decision control method for ramp merging scenarios. Background Art

[0002] With the rapid development of autonomous driving technology, the safety and efficiency of autonomous driving vehicles in complex road environments have become a hot topic of research. As one of the typical complex traffic scenarios, the ramp merging scenario requires the autonomous driving system to make accurate decisions and control in the mixed traffic flow of main roads and ramp vehicles to ensure the safety of vehicles and smooth traffic flow. In the ramp merging scenario, the autonomous driving system needs to take into account the dynamic interaction of multiple vehicles from the main roads and ramps, which places high demands on the real-time and accuracy of decision-making.

[0003] Traditional autonomous driving decision-making and control methods mostly rely on pre-set rules and models. These methods perform well when dealing with structured roads and simple traffic conditions. However, in complex dynamic environments such as ramp merging, these methods often appear to be inflexible and inefficient. In recent years, reinforcement learning technology has received extensive attention in autonomous driving decision-making. Reinforcement learning continuously learns the optimal strategy through interaction with the environment, and can achieve more flexible and efficient decision-making in complex traffic environments. However, there are some challenges in practical applications of relying solely on reinforcement learning for decision-making. First, reinforcement learning algorithms may require a large amount of training data and computing resources during the training process, and the training time is long. Second, reinforcement learning algorithms converge slowly in high-dimensional continuous spaces, and it is difficult to guarantee convergence to the global optimal or better strategy, especially in complex and dynamic ramp merging scenarios. In practical applications, existing methods often find it difficult to balance efficient path planning and accurate path tracking control, resulting in insufficient adaptability and robustness when dealing with complex ramp merging scenarios. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide an autonomous driving hierarchical decision-making control method for ramp merging scenarios, which makes strategic decisions through a maximum entropy deep reinforcement learning model, plans the path based on a polynomial trajectory planning algorithm, and finally tracks and controls it through the lower layer, thereby reducing the overall training difficulty of reinforcement learning and improving the stability and safety of autonomous driving vehicles merging into ramps.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] An autonomous driving hierarchical decision control method for ramp merging scenarios includes the following steps:

[0007] S1: Construct an interactive highway parallel ramp merge scenario, use a two-degree-of-freedom kinematic model to describe the motion of vehicles in the training scenario, and use a rule-based intelligent decision-making control model to control other vehicles, making the environment interactive;

[0008] S2: Build a decision model based on maximum entropy deep reinforcement learning, including building state mapping, state space, action space, reward function, strategy model, evaluation model and experience replay pool based on ramp scenarios;

[0009] S3: constructing a trajectory planning algorithm based on a polynomial curve, including constructing a lane-changing action correction model, calculating a lateral trajectory of the vehicle based on a quintic polynomial, calculating a longitudinal trajectory of the vehicle based on a quartic polynomial, and splicing the lateral trajectory and the longitudinal trajectory to obtain a three-dimensional trajectory of the vehicle;

[0010] S4: Construct an action mapping function and path tracking control model based on the ramp scenario, convert the vehicle trajectory and action planned by the polynomial curve into a Cartesian coordinate system, and track the trajectory through the PID algorithm to control the vehicle motion;

[0011] S5: Training the decision model based on maximum entropy deep reinforcement learning.

[0012] Furthermore, in the interactive highway parallel ramp merging scenario, a two-degree-of-freedom kinematic model is used to describe the motion of the vehicle in the training scenario, as follows:

[0013]

[0014] Among them, x, y, v are the longitudinal position, lateral position, yaw angle and velocity of the vehicle, respectively, x', y', v' is the first-order derivative of the longitudinal position, lateral position, yaw angle and velocity, a is the acceleration, l r ,l f is the distance from the center of gravity of the vehicle to the front axle and the rear axle, β is the sideslip angle at the center of gravity of the vehicle, and δ is the front wheel steering angle.

[0015] Furthermore, the rule-based intelligent decision-making control model in step S1 includes: the longitudinal control of the environmental vehicle adopts the intelligent driving model (Intelligent Driver Model), and the lateral control adopts the minimizing overall braking decelerations induced by lane changes model (Minimizing Overall Braking Decelerations Induced by Lane Changes); randomly initialize the initial position, initial speed and expected speed of the environmental vehicle on the straight road in the environment, as well as the initial position and initial speed of the merging vehicle; ensure that the initial spacing of the environmental vehicles is within a certain range, and the initial speed satisfies the normal distribution with respect to the optimal target vehicle speed.

[0016] Further, step S2 specifically includes the following steps:

[0017] S21: Construct the state mapping based on the ramp scenario as follows:

[0018]

[0019] Among them, x e ,y e are the horizontal and vertical coordinates of the vehicle after observation transformation, s kb ,x kb are the arc length and section length of the kb section respectively, s r is the arc length from the merging vehicle to the starting point, d is the lane width, and Δy is the deviation value relative to the center line of the road in the Frenet coordinate system;

[0020] S22: Build the state space S, which consists of the vehicle state, surrounding vehicle state and obstacle information, namely:

[0021] S=(S e ,S i ,S b )

[0022] Among them, S e Indicates the state of the incoming vehicle, S i Indicates the state of surrounding vehicles, S b Indicates obstacle information;

[0023] The imported vehicle state S e It is expressed as follows:

[0024]

[0025] Among them, p e =1, t e The vehicle strategy type is set to 0. e ,y eThey represent the horizontal and vertical positions of the merging vehicle after state mapping, v xe ,v ye , are the lateral and longitudinal speeds and heading angles of the incoming vehicle in the Frenet coordinate system;

[0026] The surrounding vehicle state S i The specific expressions are as follows:

[0027]

[0028] stp i ∈{0,1},t i ∈{1,2,3},i≤K

[0029] Among them, p i Indicates whether vehicle i is within the observation range of the merging vehicle. If the longitudinal position is within the observation range of the merging vehicle, then p i =1, otherwise, p i =0. t i is the strategy type of vehicle i. If it is an aggressive strategy, then t i =1, which is a normal strategy, then t i =2, which is a conservative strategy, then t i =3;x i ,y i Represent the lateral and longitudinal positions of vehicle i, v xi ,v yi are the lateral and longitudinal velocities of vehicle i, respectively; is the heading angle of vehicle i; the number of surrounding vehicles to be observed is set to K. During observation, the longitudinal distances from surrounding vehicles to the incoming vehicle are sorted, and the status information of the K vehicles closest to them is selected and added; if there are less than K surrounding vehicles, the insufficient part is filled with 0;

[0030] When the distance from the obstacle set at the end of the ramp to the merging vehicle is less than the distance to the farthest vehicle in the state space, the state information of the obstacle is added to the state space; the obstacle at the end of the ramp is regarded as a stationary vehicle with only position information and the rest of the states are 0, that is:

[0031] S b =(p b ,0,-x e ,-y e ,-v xe ,-v ye ,0),p b ∈{0,1}

[0032] S23: Determine the action space A, which is composed of the lane change action lc, the target speed x′(t x) and trajectory duration T y Composition, namely:

[0033] A=(lc,x′(t x ),T y ),lc∈{left,middle,right}

[0034] Among them, lc represents the lane change behavior, left, middle, and right represent left lane change, straight driving, and right lane change, respectively;

[0035] S24: Construct a reward function R, which is a security indicator R safe , high efficiency index R fast , comfort index R comfort , Compliance Index R rule and completion index R success The weighted sum of the five indicators is:

[0036] R=w1R safe +w2R fast +w3R comfort +w4R rule +w5R success

[0037] Among them, w1, w2, w3, w4, and w5 are the safety indicators R safe , high efficiency index R fast , comfort index R comfort , Compliance Index R rule and completion index R success The weight of the safety index R safe The target vehicle is required to have no collision and maintain a certain safe distance from the front and rear vehicles; the efficiency index R fast The target vehicle is required to drive at a speed close to the expected speed when driving on the ramp, and is encouraged to drive at a fast speed when driving on the straight road; the comfort index R comfort The target vehicle is required to have a small lateral acceleration and longitudinal acceleration and to minimize lane changing behavior; the compliance index R rule The target vehicle is encouraged to drive in the rightmost lane and to drive along the center line of the lane; the completion index R success Encourage the target vehicle to complete the entire merge process smoothly and drive smoothly on the straight road;

[0038] S25: Given the structure of the strategy model and the evaluation model: Use a neural network to fit the strategy model and the evaluation model. The strategy model represents the mapping relationship between state and action. The evaluation model uses the weighted sum of maximizing reward and strategy entropy as the evaluation criterion to evaluate the actions taken by the maximum entropy deep reinforcement learning model.

[0039] S26: The experience replay pool stores the experience data of the interaction between the vehicle and the environment. During model training, a certain amount of data is randomly extracted from the experience replay pool. At the same time, new data of the continuous interaction between the vehicle and the environment will also be stored in the experience replay pool.

[0040] Further, step S3 constructs a trajectory planning algorithm based on a polynomial curve, specifically comprising the following steps:

[0041] S31: Constructing a lane-changing action correction model: The lane-changing action correction model is used to correct unreasonable lane-changing behaviors in the decision-making of the deep reinforcement learning algorithm. If the end point of the vehicle's planned trajectory is outside the lane range, or the vehicle has merged into the main road and changed lanes back to the ramp, the action output by the deep reinforcement learning algorithm is corrected to cancel the lane-changing behavior.

[0042] S32: Construct a lateral trajectory planning algorithm based on a quintic polynomial. The lateral position y(t) of the vehicle with respect to the current decision time is expressed as follows:

[0043] y(t)=a5·(t-t0) 5 +a4·(t-t0) 4 +a3·(t-t0) 3 +a2·(t-t0) 2 +a1·(t-t0)+a0

[0044] Among them, t0 represents the starting time point of the decision step, a0, a1, a2, a3, a4, a5 are all constant coefficients of the quintic polynomial; the coefficients of the lateral trajectory are determined by the initial state {y(t0), y′(t0), y″(t0)} at time t0 and t y The final state at time {y(t y ),y′(t y ),y″(t y )} to solve;

[0045] S33: Deriving the analytical expression for the lateral position, the analytical expressions for the lateral velocity y′(t) and lateral acceleration y″(t) of the vehicle are obtained as follows:

[0046] y′(t)=5a5·(t-t0) 4 +4a4·(t-t0) 3 +3a3·(t-t0) 2 +2a2·(t-t0)+a1

[0047] y″(t)=20a5·(t-t0) 3 +12a4·(t-t0) 2 +6a3·(t-t0)+2a2

[0048] S34: Construct a longitudinal trajectory planning algorithm based on a quartic polynomial. The analytical expression of the change of the longitudinal position x(t) of the vehicle with respect to the current decision time t is expressed as:

[0049] x(t)=b4·(t-t0) 4 +b3·(t-t0) 3 +b2·(t-t0) 2 +b1·(t-t0)+b0

[0050] Among them, b0, b1, b2, b3, b4 are all constant coefficients of quartic polynomials; the coefficients of the longitudinal trajectory are determined by the initial state {x(t0), x′(t0), x″(t0)} at time t0 and t x The final state at time {x′(t x ),x″(t x )} to solve;

[0051] S35: Deriving the analytical expression of the longitudinal position, the analytical expressions of the longitudinal velocity x′(t) and longitudinal acceleration x″(t) of the vehicle are:

[0052] x′(t)=4b4·(t-t0) 3 +3b3·(t-t0) 2 +2b2·(t-t0)+b1

[0053] x″(t)=12b4·(t-t0) 2 +6b3·(t-t0)+2b2

[0054] S36: Polynomial trajectory merging: concatenate the obtained lateral trajectory (y, t) and longitudinal trajectory (x, t) to obtain a three-dimensional trajectory (x, y, t).

[0055] Further, the step S4 of constructing the action mapping function and path tracking control model based on the ramp scenario specifically includes the following steps:

[0056] S41: Constructing an action mapping function based on ramp scenarios: Convert the vehicle trajectory planned by the polynomial curve into a Cartesian coordinate system. Different action mapping functions need to be designed for different ramp scenarios.

[0057] S42, constructing a path tracking control model based on the PID algorithm: calculating the lateral and longitudinal errors between the predetermined trajectory and the current state of the vehicle at each specific time node, using this as the input of the PID control model, and calculating the steering angle and acceleration values ​​that need to be adjusted to control the vehicle movement.

[0058] Further, the training of the decision model based on maximum entropy deep reinforcement learning in step S5 specifically includes the following steps:

[0059] S51: Initialize the decision model based on maximum entropy deep reinforcement learning, including the hyperparameters of the strategy model and the evaluation model;

[0060] S52: Add target vehicles to the training environment to generate interactive training data (s t ,a t ,r t ,s t+1 ), add the training data to the experience replay pool;

[0061] S53: Extract training data from the experience replay pool, and use the gradient descent method to update the two evaluation models of the evaluation function respectively. The descent gradient is:

[0062]

[0063] Among them, M is the number of samples, |M| represents the size of the sample set, and s t ,a t ,r t They are the state of the vehicle at time t, the action taken, and the reward obtained, Q i is the i-th evaluation model, θ i is the evaluation model Q i The parameters of the evaluation model, y(·) is the prediction of the evaluation model value, Q tar-j is the jth objective function, π(·|s t ) represents the policy function, is the state s at the next moment t+1 , the behavior at the next moment sampled according to the policy function; α is the temperature coefficient; γ is the discount factor;

[0064] S54: Update the policy function using the gradient descent method, and the descent gradient is:

[0065]

[0066] Among them, ψ is the parameter of the policy function, is in state s t Next, the behavior sampled according to the policy function;

[0067] S55: Update the temperature coefficient using the gradient descent method, and the descent gradient is:

[0068]

[0069] Among them, α is the temperature coefficient, H0 is the target entropy;

[0070] S56: Update two target evaluation models of the evaluation model respectively:

[0071] θtar,i =ρθ tar,i +(1-ρ)θ i ,for i=1,2

[0072] Among them, ρ is the soft update coefficient, θ tar,i is the objective function Q tar-i The parameter θ i is the evaluation model Q i Parameters;

[0073] S57: Iteratively update the maximum entropy deep reinforcement learning model: After the maximum entropy deep reinforcement learning model converges, test the maximum entropy deep reinforcement learning model. If it does not meet expectations, optimize the hyperparameters and reward function of the maximum entropy reinforcement learning model, and return to step S51.

[0074] The beneficial effects of the present invention are:

[0075] 1) The present invention designs a highway ramp merging training environment with interactive characteristics, so that the training data is more in line with the characteristics of real traffic scenes. The scenes have high randomness and can effectively cover various special situations occurring on the ramps, which is conducive to improving the decision-making and planning performance of the reinforcement learning model in real traffic scenes.

[0076] 2) The present invention designs a hierarchical decision-making control method for ramp merging scenarios, which effectively reduces the training difficulty of reinforcement learning and improves the flexibility and stability of autonomous driving vehicles in complex scenarios.

[0077] 3) The present invention designs an effective action space for ramp merging scenarios, which, combined with a polynomial-based trajectory planning algorithm, can ensure that vehicles have certain safety, efficiency and comfort when merging.

[0078] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0080] Figure 1 It is a logical structure diagram of the hierarchical decision control method for autonomous driving in ramp merging scenarios according to the present invention;

[0081] Figure 2 This is a schematic diagram of the structure of the reinforcement learning decision planning model;

[0082] Figure 3 A schematic diagram of the interactive highway ramp merging environment;

[0083] Figure 4 This is a schematic diagram of the highway ramp scene and observation mapping;

[0084] Figure 5 Update the training process diagram for the reinforcement learning decision planning model;

[0085] Figure 6 This is the overall flow chart of the hierarchical decision-making control method for autonomous driving for ramp merging scenarios. DETAILED DESCRIPTION

[0086] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0087] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0088] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0089] See also Figure 1 to Figure 6The present invention provides an autonomous driving hierarchical decision-making control method for ramp merging scenarios. Considering the interactive behaviors between vehicles in a real traffic environment, an interactive simulation training environment is provided to provide interactive training data. At the same time, considering the requirements of stability and driving efficiency for the ramp merging problem of autonomous driving vehicles, a hierarchical decision-making control method based on maximum entropy deep reinforcement learning with better stability, higher merging efficiency and higher sample utilization is proposed. The method specifically includes the following steps:

[0090] Step S1, construct an interactive highway parallel ramp merging scene, use a two-degree-of-freedom kinematic model to describe the movement of vehicles in the training scene, and use a rule-based intelligent decision-making control model to control other vehicles to make the environment interactive. Specifically, the following steps are included:

[0091] S11: Construct parallel ramp merging scenarios, such as Figure 4 As shown. This scene is mainly composed of the main road and ramps, which can be divided into ramp lanes, speed lanes and merging lanes according to behavior. At the end of the merging lane, there is a road end. The merging vehicle needs to change lanes and merge into the main road before reaching the road end. Therefore, a square obstacle is set at the end of the lane. Its speed, acceleration and other parameters are all 0, and it only has position information;

[0092] S12: In the specified environment, the movement of the vehicle is described by a two-degree-of-freedom kinematic model, namely:

[0093]

[0094] Among them, x, y, v are the longitudinal position, lateral position, yaw angle and velocity of the vehicle, respectively, x', y', v' is the first-order derivative of the longitudinal position, lateral position, yaw angle and velocity, a is the acceleration, l r ,l f is the distance from the center of gravity of the vehicle to the front and rear axles, β is the sideslip angle at the center of gravity of the vehicle, and δ is the front wheel steering angle;

[0095] S13: Building a rule-based environmental vehicle decision-making control model: In order to enable the environmental vehicle to actively respond to changes in the environment and make the environment interactive, a rule-based intelligent decision-making control model is given to the environmental vehicle. The longitudinal control of the environmental vehicle adopts the intelligent driving model (Intelligent Driver Model), and the lateral control adopts the Minimizing Overall Braking Decelerations Induced by LaneChanges model;

[0096] S14: Randomly initialize the initial position, initial speed and expected speed of the environment vehicle on the straight road in the environment, as well as the initial position and initial speed of the merging vehicle. It is necessary to ensure that the initial spacing of the environment vehicles is within a certain range, and the initial speed should satisfy the normal distribution of the optimal target speed, so as to ensure the randomness of the environment and avoid vehicle collisions;

[0097] Step S2, constructing a decision model based on maximum entropy deep reinforcement learning, specifically including the following steps:

[0098] S21: For ramp merging scenarios, such as Figure 4 In order to ensure the correct timing of vehicle merging and avoid the position offset caused by Frenet coordinate conversion, a state mapping based on the ramp scenario is constructed by combining Cartesian coordinates and Frenet coordinates:

[0099]

[0100] Among them, x e ,y e are the horizontal and vertical coordinates of the vehicle after observation transformation, s kb ,x kb are the arc length and section length of the kb section respectively, s r is the arc length from the merging vehicle to the starting point, d is the lane width, and Δy is the deviation value relative to the center line of the road in the Frenet coordinate system;

[0101] S22: Build state space S: In the ramp merging scenario, important environmental information includes the state of the vehicle, the state of surrounding vehicles, and road parameters. Although the road parameters are specific and complex, the added state dimension does not provide additional value because they do not change with the decision point. Therefore, the state space S consists of the state of the vehicle, the state of surrounding vehicles, and obstacle information. That is:

[0102] S=(S e ,S i ,S b )

[0103] Among them, S e Indicates the state of the incoming vehicle, S i Indicates the state of surrounding vehicles, S b Indicates obstacle information;

[0104] Import vehicle status S e The specific expressions are as follows:

[0105]

[0106] Among them, p e =1, t e The vehicle strategy type is set to 0.e ,y e They represent the horizontal and vertical positions of the merging vehicle after state mapping, v xe ,v ye , are the lateral and longitudinal speeds and heading angles of the incoming vehicle in the Frenet coordinate system;

[0107] Surrounding vehicle status S i The specific expressions are as follows:

[0108]

[0109] stp i ∈{0,1},t i ∈{1,2,3},i≤K

[0110] Among them, p i Indicates whether vehicle i is within the observation range of the merging vehicle. If the longitudinal position is within the observation range of the merging vehicle, then p i =1, otherwise, p i =0. t i is the strategy type of vehicle i. If it is an aggressive strategy, then t i =1, which is a normal strategy, then t i =2, which is a conservative strategy, then t i =3. x i ,y i Represent the lateral and longitudinal positions of vehicle i, v xi ,v yi are the lateral and longitudinal velocities of vehicle i, respectively. is the heading angle of vehicle i. Since the size of the state space needs to be a constant, the number of surrounding vehicles to be observed is set to K. During observation, the longitudinal distances from surrounding vehicles to the incoming vehicle are sorted, and the state information of the closest K vehicles is selected and added. If there are less than K surrounding vehicles, the insufficient part is filled with 0;

[0111] When the distance from the obstacle set at the end of the ramp to the merging vehicle is less than the distance to the farthest vehicle in the state space, the state information of the obstacle is added to the state space. The obstacle at the end of the ramp can be regarded as a stationary vehicle with only position information and the rest of the states are 0. That is:

[0112] S b =(p b ,0,-x e ,-y e ,-v xe ,-v ye ,0),p b ∈{0,1}

[0113] S23: Determine action space A: Simplify the action space, assume that the vehicle maintains a constant speed along the center line of the road after each decision, and use as few output actions as possible to determine the shape of the polynomial curve. Therefore, the action space is designed to consist of the lane change action lc, the target speed x′(t x ) and trajectory duration T y It consists of three parts, so as to make strategic decisions at the top level, namely:

[0114] A=(lc,x′(t x ),T y ),lc∈{left,middle,right}

[0115] Among them, lc represents the lane change behavior, left, middle, and right represent left lane change, straight driving, and right lane change, respectively;

[0116] S24: Construct reward function R: The reward function is the security indicator R safe , high efficiency index R fast , comfort index R comfort , Compliance Index R rule and completion index R success The weighted sum of the five indicators is:

[0117] R=w1R safe +w2R fast +w3R comfort +w4R rule +w5R success

[0118] Among them, w1, w2, w3, w4, w5 are the weights of each item, and the security index R safe The target vehicle is required to have no collision and maintain a certain safe distance from the front and rear vehicles. The efficiency index R fast The target vehicle is required to drive at a speed close to the expected speed when driving on the ramp, and is encouraged to drive at a faster speed when driving on the straight road. The comfort index R comfort The target vehicle is required to have a small lateral acceleration and longitudinal acceleration, and to minimize lane changing behavior. The compliance index R rule The target vehicle is encouraged to drive in the rightmost lane and to drive along the center line of the lane. The completion index R success Encourage the target vehicle to complete the entire merge process smoothly and drive smoothly on the straight road;

[0119] S241: In this embodiment, the safety index R safe The collision benefit function r collision and safety distance benefit function r distance composition:

[0120] R safe =W safe ·[r collision ,r distance ]

[0121] Among them, W safe is the weight matrix of the security index, and the profit value is obtained by multiplying it with the profit function matrix;

[0122] S242: In this embodiment, the efficiency index R safe By high speed profit function r speed and the congestion benefit function r congestion composition:

[0123] R fast =W fast ·[r speed ,r congestion ]

[0124] Among them, W fast is the weight matrix of high efficiency index;

[0125] S243: In this embodiment, the comfort index R comfort It is required that the lateral acceleration and longitudinal acceleration of the merging vehicle are kept small and the lane changing behavior is minimized. The expression is:

[0126]

[0127] Among them, W comfort is the weight matrix of comfort index, δ f is the front wheel steering angle, are the accelerations of the incoming vehicle at the current moment and the previous moment, They represent the maximum and minimum values ​​of acceleration respectively, and lane_change indicates whether the target vehicle has executed a lane change behavior;

[0128] S244: In this embodiment, the compliance indicator R rule The road maintenance benefit function r keep and the right-hand revenue function r right composition:

[0129] R rule =W rule ·[r keep ,r right ]

[0130] Among them, W rule is the weight matrix of compliance indicators;

[0131] S245: In this embodiment, the completion index R successIt can be expressed as:

[0132] R success =W success ·Ind(success)

[0133] Among them, W success is the weight value of the completion index. Success indicates whether the vehicle has reached the designated location on the main road without collision and completed the merging process.

[0134] S25: Given the structure of the strategy model and the evaluation model: Use a neural network to fit the strategy model and the evaluation model. The strategy model represents the mapping relationship between state and action. The evaluation model uses the weighted sum of maximizing reward and strategy entropy as the evaluation criterion to evaluate the actions taken by the maximum entropy deep reinforcement learning model.

[0135] S26: The experience replay pool stores the experience data of the interaction between the vehicle and the environment. During model training, a certain amount of data is randomly extracted from the experience replay pool, and new data of the continuous interaction between the vehicle and the environment is also stored in the experience replay pool.

[0136] Step S3, constructing a trajectory planning algorithm based on a polynomial curve, specifically comprising the following steps:

[0137] S31: Construct lane-changing action correction model: The lane-changing action correction model is mainly used to correct unreasonable lane-changing behaviors in the decision-making of the deep reinforcement learning algorithm. If the end point of the vehicle's planned trajectory is outside the lane range, or the vehicle has merged into the main road and changed lanes back to the ramp, the action output by the deep reinforcement learning algorithm is corrected to cancel the lane-changing behavior;

[0138] S32: Construct a lateral trajectory planning algorithm based on a quintic polynomial. The analytical expression of the vehicle's lateral position y(t) with respect to the current decision time can be expressed as:

[0139] y(t)=a5·(t-t0) 5 +a4·(t-t0) 4 +a3·(t-t0) 3 +a2·(t-t0) 2 +a1·(t-t0)+a0

[0140] Where t0 represents the starting time of the decision step, and a0, a1, a2, a3, a4, and a5 are all constant coefficients of a quintic polynomial. The coefficients of the lateral trajectory can be obtained by the initial state {y(t0), y′(t0), y″(t0)} at time t0 and t y The final state at time {y(t y ),y′(t y ),y″(t y )} to solve;

[0141] S33: By taking the derivative of the lateral position analytical expression, the analytical expressions of the vehicle's lateral velocity y′(t) and lateral acceleration y″(t) can be obtained as follows:

[0142] y′(t)=5a5·(t-t0) 4 +4a4·(t-t0) 3 +3a3·(t-t0) 2 +2a2·(t-t0)+a1

[0143] y″(t)=20a5·(t-t0) 3 +12a4·(t-t0) 2 +6a3·(t-t0)+2a2

[0144] S34: Construct a longitudinal trajectory planning algorithm based on a quartic polynomial. The analytical expression of the change of the longitudinal position x(t) of the vehicle with respect to the current decision time t can be expressed as:

[0145] x(t)=b4·(t-t0) 4 +b3·(t-t0) 3 +b2·(t-t0) 2 +b1·(t-t0)+b0

[0146] Where b0, b1, b2, b3, b4 are all constant coefficients of a quartic polynomial. The coefficients of the longitudinal trajectory can be obtained by the initial state {x(t0), x′(t0), x″(t0)} at time t0 and t x The final state at time {x′(t x ),x″(t x )} to solve;

[0147] S35: Deriving the analytical expression of the longitudinal position, the analytical expressions of the longitudinal velocity x′(t) and longitudinal acceleration x″(t) of the vehicle are:

[0148] x′(t)=4b4·(t-t0) 3 +3b3·(t-t0) 2 +2b2·(t-t0)+b1

[0149] x″(t)=12b4·(t-t0) 2 +6b3·(t-t0)+2b2

[0150] S36: merging polynomial trajectories, concatenating the obtained lateral trajectory (y, t) and longitudinal trajectory (x, t) to obtain a three-dimensional trajectory (x, y, t);

[0151] Step S4: constructing an action mapping function and a path tracking control model based on a ramp scenario, specifically including the following steps:

[0152] S41: Constructing action mapping functions based on ramp scenarios: To realize the merging process in the real ramp scenario, the vehicle trajectory planned by the polynomial curve needs to be converted into a Cartesian coordinate system. Different action mapping functions need to be designed for different ramp scenarios to meet actual needs.

[0153] S42, constructing a path tracking control model based on the PID algorithm: calculating the lateral and longitudinal errors between the predetermined trajectory and the current state of the vehicle at each specific time node, using this as the input of the PID control model, and calculating the steering angle and acceleration values ​​that need to be adjusted to control the vehicle movement;

[0154] Step S5, training a decision model based on maximum entropy deep reinforcement learning, specifically includes the following steps:

[0155] S51: Initialize the decision model based on maximum entropy deep reinforcement learning, including the hyperparameters of the strategy model and the evaluation model;

[0156] S52: Add target vehicles to the training environment to generate interactive training data (s t ,a t ,r t ,s t+1 ), add the training data to the experience replay pool;

[0157] S53: Extract training data from the experience replay pool, and use the gradient descent method to update the two evaluation models of the evaluation function respectively. The descent gradient is:

[0158]

[0159] Among them, M is the number of samples, |M| represents the size of the sample set, and s t ,a t ,r t They are the state of the vehicle at time t, the action taken, and the reward obtained, Q i is the i-th evaluation model, θ i is the evaluation model Q i The parameters of the evaluation model, y(·) is the prediction of the evaluation model value, Q tar-j is the jth objective function, π(·|s t ) represents the policy function, is the state s at the next moment t+1 , the behavior at the next moment sampled according to the policy function; α is the temperature coefficient; γ is the discount factor;

[0160] S54: Update the policy function using the gradient descent method, and the descent gradient is:

[0161]

[0162] Among them, ψ is the parameter of the policy function, is in state s t Next, the behavior sampled according to the policy function;

[0163] S55: Update the temperature coefficient using the gradient descent method, and the descent gradient is:

[0164]

[0165] Among them, α is the temperature coefficient, H0 is the target entropy;

[0166] S56: Update two target evaluation models of the evaluation model respectively:

[0167] θ tar,i =ρθ tar,i +(1-ρ)θ i ,for i=1,2

[0168] Among them, ρ is the soft update coefficient, θ tar,i is the objective function Q tar-i The parameter θ i is the evaluation model Q i Parameters;

[0169] S57: Iteratively update the maximum entropy deep reinforcement learning model: After the maximum entropy deep reinforcement learning model converges, test the maximum entropy deep reinforcement learning model. If it does not meet expectations, optimize the hyperparameters and reward function of the maximum entropy reinforcement learning model, and return to step S51.

[0170] A person skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium. When the program is executed, the steps of the method can be implemented. The storage medium, such as ROM / RAM, a disk, an optical disk, etc.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A hierarchical decision-making control method for autonomous driving in ramp merging scenarios, characterized by: The following steps are involved: S1: Construct an interactive highway parallel ramp merge scenario, use a two-degree-of-freedom kinematic model to describe the motion of vehicles in the training scenario, and use a rule-based intelligent decision-making control model to control other vehicles, making the environment interactive; S2: Construct a decision model based on maximum entropy deep reinforcement learning, including constructing a state mapping, state space, action space, reward function, strategy model, evaluation model and experience replay pool based on the ramp scenario; step S2 specifically includes the following steps: S21: Construct the state mapping based on the ramp scenario as follows: Among them, x e ,y e are the horizontal and vertical coordinates of the vehicle after observation transformation, s kb ,x kb are the arc length and section length of the kb section respectively, s r is the arc length from the merging vehicle to the starting point, d is the lane width, and Δy is the deviation value relative to the center line of the road in the Frenet coordinate system; S22: Build the state space S, which consists of the vehicle state, surrounding vehicle state and obstacle information, namely: S=(S e ,S i ,S b ) Among them, S e Indicates the state of the merging vehicle, S i Indicates the state of surrounding vehicles, S b Indicates obstacle information; The merged vehicle state S e It is expressed as follows: Among them, p e =1, t e The vehicle strategy type is set to 0. e ,y e They represent the horizontal and vertical positions of the merging vehicle after state mapping, v xe ,v ye , are the lateral and longitudinal speeds and heading angles of the incoming vehicle in the Frenet coordinate system; The surrounding vehicle state S i The specific expressions are as follows: s.t.p i ∈{0,1},t i ∈{1,2,3},i≤K Among them, p i represents whether vehicle i is within the observation range of the merging vehicle. If the longitudinal position is within the observation range of the merging vehicle, then p i =1, otherwise, p i =0; t i is the strategy type of vehicle i. If it is an aggressive strategy, then t i =1, which is a normal strategy, then t i =2, which is a conservative strategy, then t i =3;x i ,y i Represent the lateral and longitudinal positions of vehicle i, v xi ,v yi are the lateral and longitudinal velocities of vehicle i, respectively; is the heading angle of vehicle i; the number of surrounding vehicles to be observed is set to K. During observation, the longitudinal distances from surrounding vehicles to the incoming vehicle are sorted, and the status information of the K vehicles closest to them is selected and added; if there are less than K surrounding vehicles, the insufficient part is filled with 0; When the distance from the obstacle set at the end of the ramp to the merging vehicle is less than the distance to the farthest vehicle in the state space, the state information of the obstacle is added to the state space; the obstacle at the end of the ramp is regarded as a stationary vehicle with only position information and the rest of the states are 0, that is: S b =(p b ,0,-x e ,-y e ,-v xe ,-v ye ,0),p b ∈{0,1} S23: Determine the action space A, which is composed of the lane change action lc, the target speed x′(t x ) and trajectory duration T y Composition, namely: A=(lc,x′(t x ),T y ),lc∈{left,middle,right} Among them, lc represents the lane change behavior, left, middle, and right represent left lane change, straight driving, and right lane change, respectively; S24: Construct a reward function R, which is a security indicator R safe , high efficiency index R fast , comfort index R comfort , Compliance Index R rule and completion index R success The weighted sum of the five indicators is: R=w1R safe +w2R fast +w3R comfort +w4R rule +w5R success Among them, w1, w2, w3, w4, and w5 are the safety indicators R safe , high efficiency index R fast , comfort index R comfort , Compliance Index R rule and completion index R success The weight of the safety index R safe The target vehicle is required to have no collision and maintain a certain safe distance from the front and rear vehicles; the efficiency index R fast The target vehicle is required to drive at a speed close to the expected speed when driving on the ramp, and is encouraged to drive at a fast speed when driving on the straight road; the comfort index R comfort The target vehicle is required to have a small lateral acceleration and longitudinal acceleration and to minimize lane changing behavior; the compliance index R rule The target vehicle is encouraged to drive in the rightmost lane and to drive along the center line of the lane; the completion index R success Encourage the target vehicle to complete the entire merge process smoothly and drive smoothly on the straight road; S25: Given the structure of the strategy model and the evaluation model: Use a neural network to fit the strategy model and the evaluation model. The strategy model represents the mapping relationship between state and action. The evaluation model uses the weighted sum of maximizing reward and strategy entropy as the evaluation criterion to evaluate the actions taken by the maximum entropy deep reinforcement learning model. S26: The experience replay pool stores the experience data of the interaction between the vehicle and the environment. During model training, a certain amount of data is randomly extracted from the experience replay pool, and new data of the continuous interaction between the vehicle and the environment is also stored in the experience replay pool. S3: constructing a trajectory planning algorithm based on a polynomial curve, including constructing a lane-changing action correction model, calculating a lateral trajectory of the vehicle based on a quintic polynomial, calculating a longitudinal trajectory of the vehicle based on a quartic polynomial, and splicing the lateral trajectory and the longitudinal trajectory to obtain a three-dimensional trajectory of the vehicle; S4: Construct an action mapping function and path tracking control model based on the ramp scenario, convert the vehicle trajectory and action planned by the polynomial curve into a Cartesian coordinate system, and track the trajectory through the PID algorithm to control the vehicle motion; S5: Training the decision model based on maximum entropy deep reinforcement learning.

2. The hierarchical decision-making control method for autonomous driving in ramp merging scenarios according to claim 1, characterized in that: In the interactive highway parallel ramp merge scenario, a two-degree-of-freedom kinematic model is used to describe the motion of the vehicle in the training scenario, as follows: Among them, x, y, v are the longitudinal position, lateral position, yaw angle and velocity of the vehicle, respectively, x', y', v' is the first-order derivative of the longitudinal position, lateral position, yaw angle and velocity, a is the acceleration, l r ,l f is the distance from the center of gravity of the vehicle to the front axle and the rear axle, β is the sideslip angle at the center of gravity of the vehicle, and δ is the front wheel steering angle.

3. The autonomous driving hierarchical decision control method for ramp merging scenarios according to claim 1, characterized in that: The rule-based intelligent decision-making control model described in step S1 includes: the longitudinal control of the environmental vehicle adopts an intelligent driving model, and the lateral control adopts a braking minimization lane change model; randomly initializes the initial position, initial speed and expected speed of the environmental vehicle on the straight road in the environment, as well as the initial position and initial speed of the merging vehicle; ensures that the initial spacing of the environmental vehicles is within a certain range, and the initial speed satisfies the normal distribution about the optimal target vehicle speed.

4. The hierarchical decision-making control method for autonomous driving in ramp merging scenarios according to claim 1, characterized in that: Step S3 constructs a trajectory planning algorithm based on a polynomial curve, specifically comprising the following steps: S31: Constructing a lane-changing action correction model: The lane-changing action correction model is used to correct unreasonable lane-changing behaviors in the decision-making of the deep reinforcement learning algorithm. If the end point of the vehicle's planned trajectory is outside the lane range, or the vehicle has merged into the main road and changed lanes back to the ramp, the action output by the deep reinforcement learning algorithm is corrected to cancel the lane-changing behavior. S32: Construct a lateral trajectory planning algorithm based on a quintic polynomial. The lateral position y(t) of the vehicle with respect to the current decision time is expressed as follows: y(t)=a5·(t-t0) 5 +a4·(t-t0) 4 +a3·(t-t0) 3 +a2·(t-t0) 2 +a1·(t-t0)+a0 Among them, t0 represents the starting time point of the decision step, a0, a1, a2, a3, a4, a5 are all constant coefficients of the quintic polynomial; the coefficients of the lateral trajectory are determined by the initial state {y(t0), y′(t0), y″(t0)} at time t0 and t y The final state at time {y(t y ),y′(t y ),y″(t y )} to solve; S33: Deriving the analytical expression for the lateral position, the analytical expressions for the lateral velocity y′(t) and lateral acceleration y″(t) of the vehicle are obtained as follows: y′(t)=5a5·(t-t0) 4 +4a4·(t-t0) 3 +3a3·(t-t0) 2 +2a2·(t-t0)+a1 y″(t)=20a5·(t-t0) 3 +12a4·(t-t0) 2 +6a3·(t-t0)+2a2 S34: Construct a longitudinal trajectory planning algorithm based on a quartic polynomial. The analytical expression of the change of the longitudinal position x(t) of the vehicle with respect to the current decision time t is expressed as: x(t)=b4·(t-t0) 4 +b3·(t-t0) 3 +b2·(t-t0) 2 +b1·(t-t0)+b0 Among them, b0, b1, b2, b3, b4 are all constant coefficients of quartic polynomials; the coefficients of the longitudinal trajectory are determined by the initial state {x(t0), x′(t0), x″(t0)} at time t0 and t x The final state at time {x′(t x ),x″(t x )} to solve; S35: Deriving the analytical expression of the longitudinal position, the analytical expressions of the longitudinal velocity x′(t) and longitudinal acceleration x″(t) of the vehicle are: x′(t)=4b4·(t-t0) 3 +3b3·(t-t0) 2 +2b2·(t-t0)+b1 x″(t)=12b4·(t-t0) 2 +6b3·(t-t0)+2b2 S36: Polynomial trajectory merging: concatenate the obtained lateral trajectory (y, t) and longitudinal trajectory (x, t) to obtain a three-dimensional trajectory (x, y, t).

5. The hierarchical decision control method for autonomous driving in ramp merging scenarios according to claim 4, characterized in that: The step S4 of constructing the action mapping function and path tracking control model based on the ramp scenario specifically includes the following steps: S41: Constructing an action mapping function based on ramp scenarios: Convert the vehicle trajectory planned by the polynomial curve into a Cartesian coordinate system. Different action mapping functions need to be designed for different ramp scenarios. S42, constructing a path tracking control model based on the PID algorithm: calculating the lateral and longitudinal errors between the predetermined trajectory and the current state of the vehicle at each specific time node, using this as the input of the PID control model, and calculating the steering angle and acceleration values ​​that need to be adjusted to control the vehicle movement.

6. The hierarchical decision-making control method for autonomous driving in ramp merging scenarios according to claim 5, characterized in that: The training of the decision model based on maximum entropy deep reinforcement learning in step S5 specifically includes the following steps: S51: Initialize the decision model based on maximum entropy deep reinforcement learning, including the hyperparameters of the strategy model and the evaluation model; S52: Add target vehicles to the training environment to generate interactive training data (s t ,a t ,r t ,s t+1 ), add the training data to the experience replay pool; S53: Extract training data from the experience replay pool, and use the gradient descent method to update the two evaluation models of the evaluation function respectively. The descent gradient is: Among them, M is the number of samples, |M| represents the size of the sample set, and s t ,a t ,r t They are the state of the vehicle at time t, the action taken, and the reward obtained, Q i is the i-th evaluation model, θ i is the evaluation model Q i The parameters of the evaluation model, y(·) is the prediction of the evaluation model value, Q tar-j is the jth objective function, π(·|s t ) represents the policy function, is the state s at the next moment t+1 , the behavior at the next moment sampled according to the policy function; α is the temperature coefficient; γ is the discount factor; S54: Update the policy function using the gradient descent method, and the descent gradient is: Among them, ψ is the parameter of the policy function, is in state s t Next, the behavior sampled according to the policy function; S55: Update the temperature coefficient using the gradient descent method, and the descent gradient is: Among them, α is the temperature coefficient, H0 is the target entropy; S56: Update two target evaluation models of the evaluation model respectively: i tar,i =ρθ tar,i +(1-p)θ i ,for i=1,2 Among them, ρ is the soft update coefficient, θ tar,i is the objective function Q tar-i The parameter θ i is the evaluation model Q i Parameters; S57: Iteratively update the maximum entropy deep reinforcement learning model: After the maximum entropy deep reinforcement learning model converges, test the maximum entropy deep reinforcement learning model. If it does not meet expectations, optimize the hyperparameters and reward function of the maximum entropy reinforcement learning model, and return to step S51.

Images