Urban road overtaking decision-making method based on Level-k game algorithm

Abstract

The invention provides a vehicle overtaking decision-making method based on a Level-k game algorithm, and the method comprises the steps: firstly, considering the uncertainty of the actual behavior of a vehicle, and building a dynamic model containing the uncertainty of the mass center position of the vehicle, and a prediction model; secondly, designing a state set, an action set and a reward function of the vehicle to describe the overtaking situation of the vehicle on the urban road; then, the overtaking problem of the vehicles on the urban road is constructed based on a Level-k game framework between the vehicles; and finally, on the basis of a Level-k game algorithm framework, obtaining an optimal overtaking action sequence of the vehicle from the vehicle action set based on a Q-Learning algorithm in reinforcement learning, and ensuring that the main vehicle makes an overtaking decision in time.

Description

Technical Field

[0001] The present invention relates to a method for overtaking decision on urban roads based on a Level-k game algorithm, and belongs to the field of automobile driving control; Background Art

[0002] With the rapid development of autonomous driving technology, the progress of vehicle autonomous driving technology has had a profound impact on the transportation system, significantly improving safety and travel efficiency. Vehicle decision control is one of the core technologies of the autonomous driving system. Its main task is to ensure that the vehicle can accurately and stably execute driving decisions in the environment and move smoothly along the preset path by formulating reasonable control schemes. However, due to the increasingly complex vehicle interactions on urban roads, how to effectively make overtaking decisions to improve traffic safety and efficiency remains a research hotspot and difficulty in this field. Traditional overtaking behavior often relies on the driver's experience and judgment, which can easily lead to traffic accidents or congestion. Therefore, there is an urgent need for a vehicle overtaking decision method that can handle complex traffic interaction scenarios and adapt to the nonlinear characteristics of urban roads and dynamically changing traffic flows in real time to promote the intelligent development of urban transportation systems. Summary of the invention

[0003] The present invention proposes an urban road overtaking decision method based on the Level-k game algorithm. Considering that the vehicle interaction on urban roads is becoming increasingly complex, the overtaking problem has become more challenging. A vehicle overtaking decision controller based on the Level-k game algorithm on urban roads is designed.

[0004] The urban road overtaking decision-making method based on the Level-k game algorithm described in the present invention is implemented by adopting the following technical solutions:

[0005] The present invention proposes an urban road overtaking decision-making method based on a Level-k game algorithm. Firstly, the uncertainty of the actual driving behavior of the vehicle is considered, and a dynamic model and a prediction model containing the uncertainty of the center of mass position of the vehicle are established. Secondly, a vehicle state set, an action set and a reward function are designed to characterize the overtaking scene on the urban road. Then, a Level-k game framework is built to describe the overtaking problem on the urban road. Finally, on the basis of the Level-k game framework, the overtaking problem of the vehicle on the urban road is solved based on the Q-Learning algorithm in reinforcement learning, and the optimal overtaking action sequence of the vehicle is obtained from the action set, so that the vehicle makes an overtaking decision. The method is characterized in that the specific steps of the method are as follows:

[0006] Step 1: Establish a dynamic model and prediction model with uncertainty in the vehicle center of mass position

[0007] The vehicle dynamics modeling in this method is based on the following assumptions: the direction of the front wheel of the vehicle is consistent with the current speed direction of the vehicle; the vehicle moves in the horizontal direction, and the vertical movement of the vehicle is ignored; the mass of the vehicle is concentrated at the center of mass of the vehicle; based on the above assumptions, a two-degree-of-freedom dynamic model of the vehicle is constructed. The two-degree-of-freedom dynamic model of the vehicle is established in a system consisting of a main vehicle, surrounding vehicles and traffic environment, where the main vehicle refers to the vehicle as the analysis object on the urban road, and the surrounding vehicles refer to all vehicles perceived by the main vehicle through sensors; the specific description of the two-degree-of-freedom dynamic model of the vehicle is as follows:

[0008]

[0009] Wherein, t represents the current discrete moment of the system, in units of s; t+1 represents the next discrete moment of the system, in units of s; x(t) represents the lateral position of the center of mass of the vehicle at the current discrete moment t of the system, in units of m; x(t+1) represents the lateral position of the center of mass of the vehicle at the next discrete moment t+1 of the system, in units of m; y(t) represents the longitudinal position of the center of mass of the vehicle at the current discrete moment t of the system, in units of m; y(t+1) represents the longitudinal position of the center of mass of the vehicle at the next discrete moment t+1 of the system, in units of m; the speed of the vehicle at the current discrete moment t of the system is represented by v(t), in units of m / s; v(t+1) represents the speed of the vehicle at the next discrete moment t+1 of the system, in units of m / s; β(t) represents the sideslip angle of the center of mass of the vehicle at the current discrete moment t of the system, in units of rad; It represents the yaw angle of the vehicle at the current discrete time t of the system, in rad; It represents the yaw angle of the vehicle at the next discrete moment t+1 of the system, in rad; a(t) represents the acceleration of the vehicle at the current discrete moment t of the system, in m / s 2 ; Δt represents the sampling step of the system, in seconds; δ f (t) represents the front wheel turning angle of the vehicle at the current discrete time t of the system, in rad; l f is the vertical distance from the center of mass of the vehicle to the front axle of the vehicle, in meters; l r is the vertical distance from the center of mass of the vehicle to the rear axle of the vehicle, in meters. Since the mathematical model cannot predict the vehicle behavior with absolute accuracy, there is still the possibility of traffic accidents. Therefore, the κ in the two-degree-of-freedom dynamic model of the vehicle (1) is x (t) is used to represent the uncertainty of the lateral position of the vehicle center of mass at the current discrete time t of the system, in m; κ y (t) is used to represent the uncertainty of the longitudinal position of the vehicle's center of mass at the current discrete time t of the system, in meters;

[0010] Assuming that the rear wheels of the vehicle cannot steer and tire forces are not considered, the action input of the two-degree-of-freedom dynamic model of the vehicle is given by Γ(t) = (a(t),δ f (t)), where Γ(t) represents the action input set of the system at the current discrete time t, including the vehicle acceleration a(t) and the front wheel steering angle δ at the current discrete time t. f (t), controlled by the vehicle's throttle, brake and steering wheel; the two-degree-of-freedom dynamic model of the vehicle is extended to the prediction time domain N, and based on the rolling time domain strategy, the vehicle prediction model formula (2) is obtained:

[0011]

[0012] Among them, i represents the prediction step length of a positive integer from 0 to N-1, and x t+i represents the lateral position of the vehicle's center of mass when the system predicts the step size i; x t+i+1 represents the lateral position of the vehicle's center of mass when the system predicts the step size to be i+1; y t+i represents the longitudinal position of the vehicle's center of mass when the system predicts the step size is i; y t+i+1 represents the longitudinal position of the vehicle's center of mass when the system predicts the step size to be i+1; v t+i represents the vehicle speed when the system predicts the step length to be i; v t+i+1 Indicates the vehicle speed when the system predicts the step size is i+1; β t+i It represents the vehicle center of mass sideslip angle when the system predicts the step length is i; It represents the vehicle yaw angle when the system predicts the step length is i; Indicates the vehicle yaw angle when the system predicts the step length to be i+1; a t+i Indicates the vehicle acceleration when the system predicts the step length to be i; δ f,t+i represents the front wheel turning angle of the vehicle when the system predicts the step length is i; κ x,t+i represents the uncertainty of the lateral position of the vehicle center of mass when the system predicts the step size i; κ y,t+i represents the uncertainty of the longitudinal position of the vehicle center of mass when the system predicts the step length i; at this time, the action input of the vehicle prediction model (2) is Γ t+i =(a t+i ,δ f,t+i ), Γ t+i Represents the input set when the system predicts the step length is i, including the vehicle acceleration a when the system predicts the step length is i t+i and the vehicle front wheel steering angle δ f,t+i ;

[0013] Step 2: Set the state set, action set and penalty function in the vehicle overtaking decision process

[0014] Due to the frequent interaction between the host vehicle and surrounding vehicles, the selection of the vehicle state set is very important; the state set used in this method is formula (3), which runs through the entire overtaking decision:

[0015]

[0016] Among them, S(t) is the state set of the vehicle at the current discrete time t of the system, including the lateral position x(t) of the vehicle at the current discrete time t of the system, the longitudinal position y(t) of the vehicle, and the yaw angle of the vehicle and the speed of the vehicle v(t), T represents the transpose of the vector;

[0017] Based on the two-degree-of-freedom dynamics model action input Γ(t) of the vehicle obtained in step 1, seven actions of vehicle overtaking decision can be obtained:

[0018]

[0019] Among them, Μ(t) is the action set of the vehicle at the current discrete time t of the system; a(t) represents the acceleration of the vehicle at the current discrete time t of the system; -a(t) represents the deceleration of the vehicle at the current discrete time t of the system; δ f (t) represents the front wheel turning angle of the vehicle changing lanes to the left at the current discrete time t of the system; -δ f (t) represents the front wheel turning angle of the vehicle changing lanes to the right at the current discrete time t of the system; therefore, the actions represented by the action set Μ(t) from front to back are: driving straight at original speed, turning left slightly, turning right slightly, driving straight at an accelerated speed, driving straight at a decelerated speed, turning left significantly, and turning right significantly. The actions in the action set Μ(t) are sufficient to describe the movement of the vehicle;

[0020] To define the penalty function, we assume that the geometric outlines of both the autonomous vehicle and the human-driven vehicle are rectangular. We first model the uncertainty of the vehicle's center of mass position, assuming that these uncertainties come from the following set of perturbations:

[0021]

[0022] Where κ(t) is the lateral position of the vehicle center of mass at the current discrete time t of the system. x (t) and the longitudinal position κ of the vehicle center of mass at the current discrete time t y (t), κ(t) T is the transposed matrix of κ(t), is a set of real numbers, η is a set of positive even numbers, ξ and θ are perturbation parameters, ξ is a parameter matrix of η×2 dimensions, and θ is a parameter matrix of η×1 dimensions;

[0023] This method uses the outer contour of the vehicle body to enclose two rectangular areas: the inner area is the vehicle determination area C that is close to the vehicle body contour; the outer area is the vehicle uncertainty area U surrounding the vehicle determination area C, and the vehicle uncertainty area U takes into account the uncertainty of the vehicle's center of mass position; the vertical distance between the two rectangular areas of the vehicle determination area C and the vehicle uncertainty area U is represented by the disturbance set representing the uncertainty of the vehicle's center of mass position In order to include the uncertainty of all positions of the vehicle's center of mass as much as possible, the distance between the vehicle's determined area C and the vehicle's uncertain area U is in represents the maximum uncertainty of the lateral position of the vehicle's center of mass at the current discrete time t of the system, It represents the maximum uncertainty of the longitudinal position of the center of mass of the vehicle at the current discrete time t of the system. To facilitate the derivation of subsequent formulas, h is defined to represent the main vehicle and j to represent the surrounding vehicle.

[0024] Based on the two areas of vehicle determination area C and vehicle uncertainty area U, for the main vehicle h, the following penalty function is obtained:

[0025] (1) Collision penalty: The Greiner-Hormann algorithm can be used to detect whether there is an overlapping area between the determined area C of the main vehicle h and the determined area C of the surrounding vehicle j, and between the uncertain area U of the main vehicle h and the uncertain area U of the surrounding vehicle j. The Greiner-Hormann algorithm first calculates and sorts the edge intersections of the determined area C of the main vehicle h and the determined area C of the surrounding vehicle j, and between the uncertain area U of the main vehicle h and the uncertain area U of the surrounding vehicle j, to identify potential intersection locations. Then, the Greiner-Hormann algorithm generates a new polygonal area based on the state of the edge intersections, which represents the intersection area of ​​the determined area C of the main vehicle h and the determined area C of the surrounding vehicle j, and between the uncertain area U of the main vehicle h and the uncertain area U of the surrounding vehicle j. If the intersection area is non-empty, The polygonal area indicates that there is an overlap between the determined area C of the main vehicle h and the determined area C of the surrounding vehicle j, and between the uncertain area U of the main vehicle h and the uncertain area U of the surrounding vehicle j; if the intersection area is an empty polygonal area, it indicates that there is no overlap between the determined area C of the main vehicle h and the determined area C of the surrounding vehicle j, and between the uncertain area U of the main vehicle h and the uncertain area U of the surrounding vehicle j; the three-dimensional information of the surrounding environment is obtained through the on-board laser radar and camera, and its position is updated in real time using the global positioning system and the inertial navigation system. This information supports the detection of the overlap between the determined area C of the main vehicle h and the determined area C of the surrounding vehicle j, and between the uncertain area U of the main vehicle h and the uncertain area U of the surrounding vehicle j, and the results are input into the vehicle's overtaking decision module, so as to adjust the overtaking decision according to the real-time situation, thereby ensuring safe driving;

[0026] If there is an overlapping area between the determined area C of the main vehicle h and the determined area C of the surrounding vehicle j, it means that the main vehicle h and the surrounding vehicle j will collide. At this time, the collision penalty function ψ 1,t Assign a value of -10; if there is no overlapping area between the determined area C of the main vehicle h and the determined area C of the surrounding vehicle j detected by the Greiner-Hormann algorithm, it means that the main vehicle h and the surrounding vehicle j will not collide, ψ 1,t Assign a value of 10;

[0027]

[0028] Among them, C t (h) represents the determined area C of the main vehicle h at the current discrete time t of the system, C t (j) represents the determined area C of the peripheral vehicle j at the current discrete time t of the system; O represents a positive integer between 1 and n except h, and n is the total number of the main vehicle h and the peripheral vehicles j;

[0029] (2) Lane boundary penalty: Assume that the lane boundary is the boundary line of the vehicle driving area. If the determined area C detected by the Greiner-Hormann algorithm overlaps with the area outside the lane boundary, it means that the vehicle has left the vehicle driving area. The lane boundary penalty function ψ 2,t The value is -10. If the Greiner-Hormann algorithm detects that there is no overlapping area between the vehicle's determined area C and the area outside the lane boundary, it means that the vehicle is traveling within the vehicle driving area. 2,t Assign a value of 10;

[0030]

[0031] Among them, F t Represents the set of areas outside the lane boundary of the vehicle at the current discrete time t of the system;

[0032] (3) Uncertain area penalty: If the Greiner-Hormann algorithm detects that there is an overlapping area between the uncertainty area U of the main vehicle h and the uncertainty area U of the surrounding vehicle j, it means that there is a possibility of collision between the main vehicle h and the surrounding vehicle j. The penalty function of the uncertainty area U is ψ 3,t The value is -10. If the Greiner-Hormann algorithm detects that there is no overlapping area between the uncertainty area U of the main vehicle h and the uncertainty area U of the surrounding vehicle j, it means that there is no possibility of collision between the main vehicle h and the surrounding vehicle j. 3,t Assign a value of 10;

[0033]

[0034] Among them, U t(h) represents the uncertainty region U of the main vehicle h at the current discrete time t of the system, U t (j) represents the uncertainty region U of the vehicle j at the current discrete time t of the system;

[0035] (4) Smooth driving penalty: The comfort of the vehicle occupants is also an important part of the evaluation index. If the vehicle speed is too fast or too slow, it will affect the physical and mental comfort of the driver and passengers, and even lead to traffic accidents. The smooth driving penalty function is defined as ψ 4,t :

[0036] ψ 4,t =-10·|a t | (9)

[0037] Among them, a t is the acceleration of the vehicle at the current discrete time t of the system;

[0038] (5) Frequent lane-changing penalty: If the main vehicle h changes lanes frequently, it will cause great trouble to the surrounding vehicle j, causing the surrounding vehicle j to be unable to make evasive actions in time, thus colliding with the main vehicle h. The frequent lane-changing penalty function is defined as ψ 5,t :

[0039] ψ 5,t =-10·n cl (10)

[0040] Among them, n cl Indicates the number of lane changes of the main vehicle h in the total system time, n cl and the vehicle's front wheel turning angle δ f The following relationship exists:

[0041]

[0042] f n is the number of lane changes per unit time of the host vehicle h, that is, the lane change frequency, t total is the total system time; k is a proportional constant, and the value of k depends on the system environment and driver characteristics.

[0043] The above five penalty functions have different priorities and occupy different proportions in the overall penalty function; the collision of vehicles in the determined area C or the uncertain area U will directly lead to an accident, and the vehicle running out of the lane boundary and frequently changing lanes may also cause an accident; in contrast, smooth driving has a relatively low impact on the comfort of the driver and passengers in the vehicle, so its importance is relatively small; the weight factors of the five penalty functions are given as follows: α1, α2, α3, α4, α5, where each weight factor is a positive number, satisfying α1+α2+α3+α4+α5=1, and we get:

[0044] α1>α3>α2>α5>α4 (12)

[0045] Based on the prediction model of the vehicle in step 1, when the prediction time domain is N, the action input sequence Γ of the vehicle's two-degree-of-freedom dynamics model is given set ={Γ t ,Γ t+1 ,…,Γ t+N-1}, according to the rolling prediction strategy, the stage penalty function R when the system prediction step is i is obtained t+i :

[0046] R t+i =α T ψ t+i (13)

[0047] Among them, ψ t+i =[ψ 1,t+i ,ψ 2,t+i ,ψ 3,t+i ,ψ 4,t+i ,ψ 5,t+i ] T , is the vector composed of the above five penalty functions, t+i is the prediction step length of the system at step i, T is the transpose of the vector, α=[α1,α2,α3,α4,α5] T , is a vector of weight factors corresponding to the five penalty functions; then the stage penalty function of all prediction steps is obtained, and then the cumulative penalty function in the prediction time domain N is obtained

[0048]

[0049] Among them, λ i is the discount factor of the penalty function when the system prediction step is i, λ i ∈[0,1];

[0050] Step 3: Build the Level-k game framework between the main vehicle and the surrounding vehicles and solve the overtaking decision problem

[0051] For the action input sequence Γ of the two-degree-of-freedom dynamic model of the main vehicle h set [h] = {Γ t [h],Γ t+1 [h],…,Γ t+N-1 [h]}, if we want to predict the next overtaking decision action of the main vehicle h, we need to predict the action input sequence Γ of the two-degree-of-freedom dynamic model of the surrounding vehicle j set [j] = {Γ t [j],Γ t+1 [j],…,Γ t+N-1 [j]}, As well as the status of the main vehicle h and the surrounding vehicle j, when the system prediction step is i, through S t+i Represents the state of all vehicles when the system prediction step is i:

[0052]

[0053] Among them, S t+i is the vehicle state set when the system predicts step size i, x t+i [h] is the lateral position of the center of mass of vehicle h when the system predicts step size i, y t+i [h] is the longitudinal position of the center of mass of vehicle h when the system predicts step size i, is the yaw angle of the vehicle h when the system predicts the step length i, v t+i [h] is the speed of vehicle h when the system predicts step size i; similarly, x t+i [j] is the lateral position of the center of mass of the vehicle j when the system predicts step size i, y t+i [j] is the longitudinal position of the center of mass of vehicle j when the system predicts step size i, is the yaw angle of vehicle j when the system predicts step size i, v t+i [j] is the speed of the vehicle j when the system predicts the step size is i, and T is the transpose of the vector;

[0054] This method first considers all vehicles as Level-0, obtains the state set of surrounding vehicle j and the action input sequence of the two-degree-of-freedom dynamic model, then considers the main vehicle h as Level-1, and then considers the surrounding vehicle j as Level-0. Based on the state set of surrounding vehicle j and the action input sequence of the two-degree-of-freedom dynamic model, the state set of the main vehicle h and the action input sequence of the two-degree-of-freedom dynamic model can be obtained, and then the optimal overtaking decision action sequence of the main vehicle h can be obtained; this method does not consider levels above Level-1 to avoid increasing the computational complexity; the stage penalty function obtained by the main vehicle h when the system prediction step is i Depends on the initial state set S of all vehicles t , the action input sequence of the two-degree-of-freedom dynamic model of the main vehicle h The action input sequence of the two-degree-of-freedom dynamic model of Zhou Chej in, It represents the action input of the two-degree-of-freedom dynamic model of the Level-1 main vehicle h when the system predicts the step length to be N-1. It represents the action input of the two-degree-of-freedom dynamic model of Level-0 vehicle j when the system prediction step is N-1, and N is the system prediction time domain; therefore, the stage penalty function obtained by the main vehicle h when the system prediction step is i is:

[0055]

[0056] The cumulative penalty function of the main vehicle h is:

[0057]

[0058] This method uses the Q-Learning algorithm in reinforcement learning to solve the optimal action of the main vehicle h to overtake the vehicle at the current discrete moment of the system; the core idea of ​​the Q-Learning algorithm is to continuously learn and predict the action input sequence of the two-degree-of-freedom dynamic model of the main vehicle h and the action input sequence of the two-degree-of-freedom dynamic model of the surrounding vehicle j in the time domain N, and update the cumulative penalty function of the main vehicle h Finally, the optimal overtaking decision action sequence of the main vehicle h in the prediction time domain N is found; based on the cumulative penalty function of the main vehicle h in the above Level-k game framework By comparing the action sequences of different main vehicles h and surrounding vehicles j at the current discrete moment of the system Size, get the cumulative penalty function that makes the system main vehicle h at the current discrete moment The optimal overtaking decision action sequence for the largest host vehicle h is:

[0059]

[0060] in, It represents the optimal action sequence of the Level-1 main vehicle h in the current discrete moment of the system and the prediction time domain N. Thus, the overtaking decision problem of the vehicle is solved.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] The present invention provides an urban road overtaking decision-making method based on the Level-k game algorithm. Firstly, a dynamic model and a prediction model containing the uncertainty of the vehicle's center of mass position are established to describe the uncertainty of the vehicle's actual behavior. Secondly, a vehicle state set, an action set and a reward function are designed to describe the overtaking scene on the urban road. On this basis, the overtaking problem of the vehicle on the urban road is constructed based on the Level-k game framework between vehicles. Finally, the overtaking problem of the vehicle on the urban road is solved based on the Q-Learning algorithm in reinforcement learning, and the vehicle overtaking decision that adapts to the nonlinear characteristics of the urban road and the dynamically changing traffic flow in real time is obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The present invention will be further described below in conjunction with the accompanying drawings:

[0064] Figure 1 A simplified flow chart of a method for making decisions on overtaking on urban roads based on a Level-k game algorithm provided by the present invention.

[0065] Figure 2A schematic diagram of a two-degree-of-freedom dynamic model of a vehicle in step one of an urban road overtaking decision-making method based on a Level-k game algorithm provided by the present invention.

[0066] Figure 3 A schematic diagram of the division of the area outside the vehicle body contour in step 2 of an urban road overtaking decision-making method based on a Level-k game algorithm provided by the present invention. DETAILED DESCRIPTION

[0067] The present invention is described in detail below in conjunction with the accompanying drawings:

[0068] The present invention proposes a method for overtaking decision on urban roads based on the Level-k game algorithm. The design process is as follows: Figure 1 As shown, the specific implementation steps are as follows:

[0069] Step 1: Establish a dynamic model and prediction model with uncertainty in the vehicle center of mass position

[0070] The vehicle dynamics modeling in this method is based on the following assumptions: the direction of the vehicle's front wheels is consistent with the vehicle's current speed; the vehicle moves in the horizontal direction, and the vertical movement of the vehicle is ignored; the mass of the vehicle is concentrated at the center of mass of the vehicle. Based on the above assumptions, a two-degree-of-freedom dynamic model of the vehicle is constructed, such as Figure 2 As shown in the figure, the two-degree-of-freedom dynamic model of the vehicle is established in a system consisting of the main vehicle, surrounding vehicles and traffic environment, where the main vehicle refers to the vehicle as the analysis object on the urban road, and the surrounding vehicles refer to all vehicles sensed by the main vehicle through sensors; the specific description of the two-degree-of-freedom dynamic model of the vehicle is as follows:

[0071]

[0072] Wherein, t represents the current discrete moment of the system, in units of s; t+1 represents the next discrete moment of the system, in units of s; x(t) represents the lateral position of the center of mass of the vehicle at the current discrete moment t of the system, in units of m; x(t+1) represents the lateral position of the center of mass of the vehicle at the next discrete moment t+1 of the system, in units of m; y(t) represents the longitudinal position of the center of mass of the vehicle at the current discrete moment t of the system, in units of m; y(t+1) represents the longitudinal position of the center of mass of the vehicle at the next discrete moment t+1 of the system, in units of m; the speed of the vehicle at the current discrete moment t of the system is represented by v(t), in units of m / s; v(t+1) represents the speed of the vehicle at the next discrete moment t+1 of the system, in units of m / s; β(t) represents the sideslip angle of the center of mass of the vehicle at the current discrete moment t of the system, in units of rad; It represents the yaw angle of the vehicle at the current discrete time t of the system, in rad; It represents the yaw angle of the vehicle at the next discrete moment t+1 of the system, in rad; a(t) represents the acceleration of the vehicle at the current discrete moment t of the system, in m / s 2 ; Δt represents the sampling step of the system, in seconds; δ f (t) represents the front wheel turning angle of the vehicle at the current discrete time t of the system, in rad; l f is the vertical distance from the center of mass of the vehicle to the front axle of the vehicle, in meters; l r is the vertical distance from the center of mass of the vehicle to the rear axle of the vehicle, in meters. Since the mathematical model cannot predict the vehicle behavior with absolute accuracy, there is still the possibility of traffic accidents. Therefore, the κ in the two-degree-of-freedom dynamic model of the vehicle (1) is x (t) is used to represent the uncertainty of the lateral position of the vehicle center of mass at the current discrete time t of the system, in m; κ y (t) is used to represent the uncertainty of the longitudinal position of the vehicle's center of mass at the current discrete time t of the system, in meters;

[0073] Assuming that the rear wheels of the vehicle cannot steer and tire forces are not considered, the action input of the two-degree-of-freedom dynamic model of the vehicle is given by Γ(t) = (a(t),δ f (t)), where Γ(t) represents the action input set of the system at the current discrete time t, including the vehicle acceleration a(t) and the front wheel steering angle δ at the current discrete time t. f (t), controlled by the vehicle's throttle, brake and steering wheel; the two-degree-of-freedom dynamic model of the vehicle is extended to the prediction time domain N, and based on the rolling time domain strategy, the vehicle prediction model formula (2) is obtained:

[0074]

[0075] Among them, i represents the prediction step length of a positive integer from 0 to N-1, and x t+i represents the lateral position of the vehicle's center of mass when the system predicts the step size i; x t+i+1 represents the lateral position of the vehicle's center of mass when the system predicts the step size to be i+1; y t+i represents the longitudinal position of the vehicle's center of mass when the system predicts the step size is i; y t+i+1 represents the longitudinal position of the vehicle's center of mass when the system predicts the step size to be i+1; v t+i represents the vehicle speed when the system predicts the step length to be i; v t+i+1 Indicates the vehicle speed when the system predicts the step size is i+1; β t+i It represents the vehicle center of mass sideslip angle when the system predicts the step length is i; It represents the vehicle yaw angle when the system predicts the step length is i; Indicates the vehicle yaw angle when the system predicts the step length to be i+1; a t+iIndicates the vehicle acceleration when the system predicts the step length to be i; δ f,t+i represents the front wheel turning angle of the vehicle when the system predicts the step length is i; κ x,t+i represents the uncertainty of the lateral position of the vehicle center of mass when the system predicts the step size i; κ y,t+i represents the uncertainty of the longitudinal position of the vehicle center of mass when the system predicts the step length i; at this time, the action input of the vehicle prediction model (2) is Γ t+i =(a t+i ,δ f,t+i ), Γ t+i Represents the input set when the system predicts the step length is i, including the vehicle acceleration a when the system predicts the step length is i t+i and the vehicle front wheel steering angle δ f,t+i ;

[0076] Step 2: Set the state set, action set and penalty function in the vehicle overtaking decision process

[0077] Due to the frequent interaction between the host vehicle and surrounding vehicles, the selection of the vehicle state set is very important; the state set used in this method is formula (3), which runs through the entire overtaking decision:

[0078]

[0079] Among them, S(t) is the state set of the vehicle at the current discrete time t of the system, including the lateral position x(t) of the vehicle at the current discrete time t of the system, the longitudinal position y(t) of the vehicle, and the yaw angle of the vehicle and the speed of the vehicle v(t), T represents the transpose of the vector;

[0080] Based on the two-degree-of-freedom dynamics model action input Γ(t) of the vehicle obtained in step 1, seven actions of vehicle overtaking decision can be obtained:

[0081]

[0082] Among them, Μ(t) is the action set of the vehicle at the current discrete time t of the system; a(t) represents the acceleration of the vehicle at the current discrete time t of the system; -a(t) represents the deceleration of the vehicle at the current discrete time t of the system; δ f (t) represents the front wheel turning angle of the vehicle changing lanes to the left at the current discrete time t of the system; -δ f (t) represents the front wheel turning angle of the vehicle changing lanes to the right at the current discrete time t of the system; therefore, the actions represented by the action set Μ(t) from front to back are: driving straight at original speed, turning left slightly, turning right slightly, driving straight at an accelerated speed, driving straight at a decelerated speed, turning left significantly, and turning right significantly. The actions in the action set Μ(t) are sufficient to describe the movement of the vehicle;

[0083] To define the penalty function, we assume that the geometric outlines of both the autonomous vehicle and the human-driven vehicle are rectangular. We first model the uncertainty of the vehicle's center of mass position, assuming that these uncertainties come from the following set of perturbations:

[0084]

[0085] Where κ(t) is the lateral position of the vehicle center of mass at the current discrete time t of the system. x (t) and the longitudinal position κ of the vehicle center of mass at the current discrete time t y (t), κ(t) T is the transposed matrix of κ(t), is a set of real numbers, η is a set of positive even numbers, ξ and θ are perturbation parameters, ξ is a parameter matrix of η×2 dimensions, and θ is a parameter matrix of η×1 dimensions;

[0086] This method encloses the outer contour of the vehicle body into two rectangular areas, such as Figure 3 As shown: Figure 3 The area surrounded by the dashed line is the vehicle determination area C; the area surrounded by the solid line is the vehicle uncertainty area U, which takes into account the uncertainty of the vehicle's center of mass position; the vertical distance between the two rectangular areas of vehicle determination area C and vehicle uncertainty area U is represented by the disturbance set representing the uncertainty of the vehicle's center of mass position In order to include the uncertainty of all positions of the vehicle's center of mass as much as possible, the distance between the vehicle's determined area C and the vehicle's uncertain area U is in represents the maximum uncertainty of the lateral position of the vehicle's center of mass at the current discrete time t of the system, It represents the maximum uncertainty of the longitudinal position of the center of mass of the vehicle at the current discrete time t of the system. To facilitate the derivation of subsequent formulas, h is defined to represent the main vehicle and j to represent the surrounding vehicle.

[0087] Based on the two areas of vehicle determination area C and vehicle uncertainty area U, for the main vehicle h, the following penalty function is obtained:

[0088] (1) Collision penalty: The Greiner-Hormann algorithm can be used to detect whether there is an overlapping area between the determined area C of the main vehicle h and the determined area C of the surrounding vehicle j, and between the uncertain area U of the main vehicle h and the uncertain area U of the surrounding vehicle j. The Greiner-Hormann algorithm first calculates and sorts the edge intersections of the determined area C of the main vehicle h and the determined area C of the surrounding vehicle j, and between the uncertain area U of the main vehicle h and the uncertain area U of the surrounding vehicle j, to identify potential intersection locations. Then, the Greiner-Hormann algorithm generates a new polygonal area based on the state of the edge intersections, which represents the intersection area of ​​the determined area C of the main vehicle h and the determined area C of the surrounding vehicle j, and between the uncertain area U of the main vehicle h and the uncertain area U of the surrounding vehicle j. If the intersection area is non-empty, The polygonal area indicates that there is an overlap between the determined area C of the main vehicle h and the determined area C of the surrounding vehicle j, and between the uncertain area U of the main vehicle h and the uncertain area U of the surrounding vehicle j; if the intersection area is an empty polygonal area, it indicates that there is no overlap between the determined area C of the main vehicle h and the determined area C of the surrounding vehicle j, and between the uncertain area U of the main vehicle h and the uncertain area U of the surrounding vehicle j; the three-dimensional information of the surrounding environment is obtained through the on-board laser radar and camera, and its position is updated in real time using the global positioning system and the inertial navigation system. This information supports the detection of the overlap between the determined area C of the main vehicle h and the determined area C of the surrounding vehicle j, and between the uncertain area U of the main vehicle h and the uncertain area U of the surrounding vehicle j, and the results are input into the vehicle's overtaking decision module, so as to adjust the overtaking decision according to the real-time situation, thereby ensuring safe driving;

[0089] If there is an overlapping area between the determined area C of the main vehicle h and the determined area C of the surrounding vehicle j, it means that the main vehicle h and the surrounding vehicle j will collide. At this time, the collision penalty function ψ 1,t Assign a value of -10; if there is no overlapping area between the determined area C of the main vehicle h and the determined area C of the surrounding vehicle j detected by the Greiner-Hormann algorithm, it means that the main vehicle h and the surrounding vehicle j will not collide, ψ 1,t Assign a value of 10;

[0090]

[0091] Among them, C t (h) represents the determined area C of the main vehicle h at the current discrete time t of the system, C t (j) represents the determined area C of the peripheral vehicle j at the current discrete time t of the system; O represents a positive integer between 1 and n except h, and n is the total number of the main vehicle h and the peripheral vehicles j;

[0092] (2) Lane boundary penalty: Assume that the lane boundary is the boundary line of the vehicle driving area. If the determined area C detected by the Greiner-Hormann algorithm overlaps with the area outside the lane boundary, it means that the vehicle has left the vehicle driving area. The lane boundary penalty function ψ 2,t The value is -10. If the Greiner-Hormann algorithm detects that there is no overlapping area between the vehicle's determined area C and the area outside the lane boundary, it means that the vehicle is traveling within the vehicle driving area. 2,t Assign a value of 10;

[0093]

[0094] Among them, F t Represents the set of areas outside the lane boundary of the vehicle at the current discrete time t of the system;

[0095] (3) Uncertain area penalty: If the Greiner-Hormann algorithm detects that there is an overlapping area between the uncertainty area U of the main vehicle h and the uncertainty area U of the surrounding vehicle j, it means that there is a possibility of collision between the main vehicle h and the surrounding vehicle j. The penalty function of the uncertainty area U is ψ 3,t The value is -10. If the Greiner-Hormann algorithm detects that there is no overlapping area between the uncertainty area U of the main vehicle h and the uncertainty area U of the surrounding vehicle j, it means that there is no possibility of collision between the main vehicle h and the surrounding vehicle j. 3,t Assign a value of 10;

[0096]

[0097] Among them, U t (h) represents the uncertainty region U of the main vehicle h at the current discrete time t of the system, U t (j) represents the uncertainty region U of the vehicle j at the current discrete time t of the system;

[0098] (4) Smooth driving penalty: The comfort of the vehicle occupants is also an important part of the evaluation index. If the vehicle speed is too fast or too slow, it will affect the physical and mental comfort of the driver and passengers, and even lead to traffic accidents. The smooth driving penalty function is defined as ψ 4,t :

[0099] ψ 4,t =-10·|a t | (9)

[0100] Among them, a t is the acceleration of the vehicle at the current discrete time t of the system;

[0101] (5) Frequent lane-changing penalty: If the main vehicle h changes lanes frequently, it will cause great trouble to the surrounding vehicle j, causing the surrounding vehicle j to be unable to make evasive actions in time, thus colliding with the main vehicle h. The frequent lane-changing penalty function is defined as ψ 5,t :

[0102] ψ 5,t =-10·n cl (10)

[0103] Among them, n cl Indicates the number of lane changes of the main vehicle h in the total system time, n cl and the vehicle's front wheel turning angle δ f The following relationship exists:

[0104]

[0105] f n is the number of lane changes per unit time of the host vehicle h, that is, the lane change frequency, t total is the total system time; k is a proportional constant, and the value of k depends on the system environment and driver characteristics.

[0106] The above five penalty functions have different priorities and occupy different proportions in the overall penalty function; the collision of vehicles in the determined area C or the uncertain area U will directly lead to an accident, and the vehicle running out of the lane boundary and frequently changing lanes may also cause an accident; in contrast, smooth driving has a relatively low impact on the comfort of the driver and passengers in the vehicle, so its importance is relatively small; the weight factors of the five penalty functions are given as follows: α1, α2, α3, α4, α5, where each weight factor is a positive number, satisfying α1+α2+α3+α4+α5=1, and we get:

[0107] α1>α3>α2>α5>α4 (12)

[0108] Based on the prediction model of the vehicle in step 1, when the prediction time domain is N, the action input sequence Γ of the vehicle's two-degree-of-freedom dynamics model is given set ={Γ t ,Γ t+1 ,…,Γ t+N-1}, according to the rolling prediction strategy, the stage penalty function R when the system prediction step is i is obtained t+i :

[0109] R t+i =α T ψ t+i (13)

[0110] Among them, ψ t+i =[ψ 1,t+i ,ψ2,t+i ,ψ 3,t+i ,ψ 4,t+i ,ψ 5,t+i ] T , is the vector composed of the above five penalty functions, t+i is the prediction step length of the system at step i, T is the transpose of the vector, α=[α1,α2,α3,α4,α5] T , is a vector of weight factors corresponding to the five penalty functions; then the stage penalty function of all prediction steps is obtained, and then the cumulative penalty function in the prediction time domain N is obtained

[0111]

[0112] Among them, λ i is the discount factor of the penalty function when the system prediction step is i, λ i ∈[0,1];

[0113] Step 3: Build the Level-k game framework between the main vehicle and the surrounding vehicles and solve the overtaking decision problem

[0114] For the action input sequence Γ of the two-degree-of-freedom dynamic model of the main vehicle h set [h] = {Γ t [h],Γ t+1 [h],…,Γ t+N-1 [h]}, if we want to predict the next overtaking decision action of the main vehicle h, we need to predict the action input sequence Γ of the two-degree-of-freedom dynamic model of the surrounding vehicle j set [j] = {Γ t [j],Γ t+1 [j],…,Γ t+N-1 [j]}, As well as the status of the main vehicle h and the surrounding vehicle j, when the system prediction step is i, through S t+i Represents the state of all vehicles when the system prediction step is i:

[0115]

[0116] Among them, S t+i is the vehicle state set when the system predicts step size i, x t+i [h] is the lateral position of the center of mass of vehicle h when the system predicts step size i, y t+i [h] is the longitudinal position of the center of mass of vehicle h when the system predicts step size i, is the yaw angle of the vehicle h when the system predicts the step length i, v t+i [h] is the speed of vehicle h when the system predicts step size i; similarly, x t+i [j] is the lateral position of the center of mass of the vehicle j when the system predicts step size i, yt+i [j] is the longitudinal position of the center of mass of vehicle j when the system predicts step size i, is the yaw angle of vehicle j when the system predicts step size i, v t+i [j] is the speed of the vehicle j when the system predicts the step size is i, and T is the transpose of the vector;

[0117] The Level-k game algorithm is a theory that can analyze vehicle interactive decision-making. It attempts to explain the process of vehicle decision-making under uncertainty. The algorithm assumes that each vehicle has a different level. Level-0 vehicles regard surrounding vehicles in the traffic environment as obstacles. Therefore, the driver of the Level-0 vehicle will think that the surrounding vehicle has given up the right of way, and the Level-0 vehicle will be regarded as an aggressive vehicle. The driver of the Level-1 vehicle thinks that the surrounding vehicle is a Level-0 vehicle and regards the driving style of the driver of the Level-0 vehicle as aggressive. Therefore, the driver of the Level-1 vehicle will adopt a cautious driving style. Similarly, the Level-2 driver will actively assume that the surrounding vehicle is Level-1 and regard the driving style of the driver of the Level-1 vehicle as a cautious driving style, and then make decisions. Level-3 and above vehicles will assume that the surrounding vehicle is its next level vehicle. In this method, the main vehicle is regarded as Level-1, and the surrounding vehicles are regarded as Level-0. Levels above Level-1 are no longer considered to avoid increasing the computational complexity.

[0118] This method first considers all vehicles as Level-0, obtains the state set of surrounding vehicle j and the action input sequence of the two-degree-of-freedom dynamic model, then considers the main vehicle h as Level-1, and then considers the surrounding vehicle j as Level-0. Based on the state set of surrounding vehicle j and the action input sequence of the two-degree-of-freedom dynamic model, the state set of the main vehicle h and the action input sequence of the two-degree-of-freedom dynamic model can be obtained, and then the optimal overtaking decision action sequence of the main vehicle h can be obtained; this method does not consider levels above Level-1 to avoid increasing the computational complexity; the stage penalty function obtained by the main vehicle h when the system prediction step is i Depends on the initial state set S of all vehicles t , the action input sequence of the two-degree-of-freedom dynamic model of the main vehicle h The action input sequence of the two-degree-of-freedom dynamic model of Zhou Chej in, It represents the action input of the two-degree-of-freedom dynamic model of the Level-1 main vehicle h when the system predicts the step length to be N-1. It represents the action input of the two-degree-of-freedom dynamic model of Level-0 vehicle j when the system prediction step is N-1, and N is the system prediction time domain; therefore, the stage penalty function obtained by the main vehicle h when the system prediction step is i is:

[0119]

[0120] The cumulative penalty function of the main vehicle h is:

[0121]

[0122] This method uses the Q-Learning algorithm in reinforcement learning to solve the optimal action of the main vehicle h for overtaking at the current discrete moment of the system; Q-Learning is a model-free reinforcement learning algorithm that does not require modeling of the environment and can directly learn the optimal strategy through interaction with the environment, and is suitable for complex and dynamic environments; in addition, under appropriate conditions, Q-Learning can ensure convergence to the optimal strategy and provide reliability for long-term decision-making; for the urban road overtaking condition of this method, the Q-Learning algorithm is more suitable;

[0123] The core idea of ​​the Q-Learning algorithm is to continuously learn and predict the action input sequence of the two-degree-of-freedom dynamic model of the main vehicle h and the action input sequence of the two-degree-of-freedom dynamic model of the surrounding vehicle j in the time domain N, and update the cumulative penalty function of the main vehicle h. Finally, the optimal overtaking decision action sequence of the main vehicle h in the prediction time domain N is found; based on the cumulative penalty function of the main vehicle h in the above Level-k game framework By comparing the action sequences of different main vehicles h and surrounding vehicles j at the current discrete moment of the system Size, get the cumulative penalty function that makes the system main vehicle h at the current discrete moment The optimal overtaking decision action sequence for the largest host vehicle h is:

[0124]

[0125] in, It represents the optimal action sequence of the Level-1 main vehicle h in the current discrete moment of the system and the prediction time domain N; thus, the vehicle overtaking decision problem is solved.

Claims

1. A method for overtaking decision on urban roads based on the Level-k game algorithm. First, the uncertainty of the actual driving behavior of the vehicle is considered, and a dynamic model and a prediction model containing the uncertainty of the vehicle's center of mass position are established. Secondly, the vehicle's state set, action set and reward function are designed to characterize the overtaking scene on the urban road. Then, a Level-k game framework is built to describe the overtaking problem on the urban road. Finally, based on the Level-k game framework, the overtaking problem of the vehicle on the urban road is solved based on the Q-Learning algorithm in reinforcement learning. The optimal overtaking action sequence of the vehicle is obtained from the action set, so that the vehicle makes an overtaking decision. It is characterized in that The specific steps of this method are as follows: Step 1: Establish a dynamic model and prediction model with uncertainty in the vehicle center of mass position The vehicle dynamics modeling in this method is based on the following assumptions: the direction of the front wheel of the vehicle is consistent with the current speed direction of the vehicle; the vehicle moves in the horizontal direction, and the vertical movement of the vehicle is ignored; the mass of the vehicle is concentrated at the center of mass of the vehicle; based on the above assumptions, a two-degree-of-freedom dynamic model of the vehicle is constructed. The two-degree-of-freedom dynamic model of the vehicle is established in a system consisting of a main vehicle, surrounding vehicles and traffic environment, where the main vehicle refers to the vehicle as the analysis object on the urban road, and the surrounding vehicles refer to all vehicles perceived by the main vehicle through sensors; the specific description of the two-degree-of-freedom dynamic model of the vehicle is as follows: Wherein, t represents the current discrete moment of the system, in units of s; t+1 represents the next discrete moment of the system, in units of s; x(t) represents the lateral position of the center of mass of the vehicle at the current discrete moment t of the system, in units of m; x(t+1) represents the lateral position of the center of mass of the vehicle at the next discrete moment t+1 of the system, in units of m; y(t) represents the longitudinal position of the center of mass of the vehicle at the current discrete moment t of the system, in units of m; y(t+1) represents the longitudinal position of the center of mass of the vehicle at the next discrete moment t+1 of the system, in units of m; the speed of the vehicle at the current discrete moment t of the system is represented by v(t), in units of m / s; v(t+1) represents the speed of the vehicle at the next discrete moment t+1 of the system, in units of m / s; β(t) represents the sideslip angle of the center of mass of the vehicle at the current discrete moment t of the system, in units of rad; It represents the yaw angle of the vehicle at the current discrete time t of the system, in rad; It represents the yaw angle of the vehicle at the next discrete moment t+1 of the system, in rad; a(t) represents the acceleration of the vehicle at the current discrete moment t of the system, in m / s 2 ; Δt represents the sampling step of the system, in seconds; δ f (t) represents the front wheel turning angle of the vehicle at the current discrete time t of the system, in rad; l f is the vertical distance from the center of mass of the vehicle to the front axle of the vehicle, in meters; l r is the vertical distance from the center of mass of the vehicle to the rear axle of the vehicle, in meters. Since the mathematical model cannot predict the vehicle behavior with absolute accuracy, there is still the possibility of traffic accidents. Therefore, the κ in the two-degree-of-freedom dynamic model of the vehicle (1) is x (t) is used to represent the uncertainty of the lateral position of the vehicle center of mass at the current discrete time t of the system, in m; κ y (t) is used to represent the uncertainty of the longitudinal position of the vehicle's center of mass at the current discrete time t of the system, in meters; Assuming that the rear wheels of the vehicle cannot steer and tire forces are not considered, the action input of the two-degree-of-freedom dynamic model of the vehicle is given by Γ(t) = (a(t),δ f (t)), where Γ(t) represents the action input set of the system at the current discrete time t, including the vehicle acceleration a(t) and the front wheel steering angle δ at the current discrete time t. f (t), controlled by the vehicle's throttle, brake and steering wheel; the two-degree-of-freedom dynamic model of the vehicle is extended to the prediction time domain N, and based on the rolling time domain strategy, the vehicle prediction model formula (2) is obtained: Among them, i represents the prediction step length of a positive integer from 0 to N-1, and x t+i represents the lateral position of the vehicle's center of mass when the system predicts the step size i; x t+i+1 represents the lateral position of the vehicle's center of mass when the system predicts the step size to be i+1; y t+i represents the longitudinal position of the vehicle's center of mass when the system predicts the step size is i; y t+i+1 represents the longitudinal position of the vehicle's center of mass when the system predicts the step size to be i+1; v t+i represents the vehicle speed when the system predicts the step length to be i; v t+i+1 Indicates the vehicle speed when the system predicts the step size is i+1; β t+i It represents the vehicle center of mass sideslip angle when the system predicts the step length is i; It represents the vehicle yaw angle when the system predicts the step length is i; Indicates the vehicle yaw angle when the system predicts the step length to be i+1; a t+i Indicates the vehicle acceleration when the system predicts the step length to be i; δ f,t+i represents the front wheel turning angle of the vehicle when the system predicts the step length is i; κ x,t+i represents the uncertainty of the lateral position of the vehicle center of mass when the system predicts the step size i; κ y,t+i represents the uncertainty of the longitudinal position of the vehicle center of mass when the system predicts the step length i; at this time, the action input of the vehicle prediction model (2) is Γ t+i =(a t+i ,δ f,t+i ), Γ t+i Represents the input set when the system predicts the step length is i, including the vehicle acceleration a when the system predicts the step length is i t+i and the vehicle front wheel steering angle δ f,t+i ; Step 2: Set the state set, action set and penalty function in the vehicle overtaking decision process Due to the frequent interaction between the host vehicle and surrounding vehicles, the selection of the vehicle state set is very important; the state set used in this method is formula (3), which runs through the entire overtaking decision: Among them, S(t) is the state set of the vehicle at the current discrete time t of the system, including the lateral position x(t) of the vehicle at the current discrete time t of the system, the longitudinal position y(t) of the vehicle, and the yaw angle of the vehicle and the speed of the vehicle v(t), T represents the transpose of the vector; Based on the two-degree-of-freedom dynamics model action input Γ(t) of the vehicle obtained in step 1, seven actions of vehicle overtaking decision can be obtained: Among them, Μ(t) is the action set of the vehicle at the current discrete time t of the system; a(t) represents the acceleration of the vehicle at the current discrete time t of the system; -a(t) represents the deceleration of the vehicle at the current discrete time t of the system; δ f (t) represents the front wheel turning angle of the vehicle changing lanes to the left at the current discrete time t of the system; -δ f (t) represents the front wheel turning angle of the vehicle changing lanes to the right at the current discrete time t of the system; therefore, the actions represented by the action set Μ(t) from front to back are: driving straight at original speed, turning left slightly, turning right slightly, driving straight at an accelerated speed, driving straight at a decelerated speed, turning left significantly, and turning right significantly. The actions in the action set Μ(t) are sufficient to describe the movement of the vehicle; To define the penalty function, we assume that the geometric outlines of both the autonomous vehicle and the human-driven vehicle are rectangular. We first model the uncertainty of the vehicle's center of mass position, assuming that these uncertainties come from the following set of perturbations: Where κ(t) is the lateral position of the vehicle center of mass at the current discrete time t. x (t) and the longitudinal position κ of the vehicle center of mass at the current discrete time t y (t), κ(t) T is the transposed matrix of κ(t), is a set of real numbers, η is a set of positive even numbers, ξ and θ are perturbation parameters, ξ is a parameter matrix of η×2 dimensions, and θ is a parameter matrix of η×1 dimensions; This method uses the outer contour of the vehicle body to enclose two rectangular areas: the inner area is the vehicle determination area C that is close to the vehicle body contour; the outer area is the vehicle uncertainty area U surrounding the vehicle determination area C, and the vehicle uncertainty area U takes into account the uncertainty of the vehicle's center of mass position; the vertical distance between the two rectangular areas of the vehicle determination area C and the vehicle uncertainty area U is represented by the disturbance set representing the uncertainty of the vehicle's center of mass position In order to include the uncertainty of all positions of the vehicle's center of mass as much as possible, the distance between the vehicle's determined area C and the vehicle's uncertain area U is in represents the maximum uncertainty of the lateral position of the vehicle's center of mass at the current discrete time t of the system, It represents the maximum uncertainty of the longitudinal position of the center of mass of the vehicle at the current discrete time t of the system. To facilitate the derivation of subsequent formulas, h is defined to represent the main vehicle and j to represent the surrounding vehicle. Based on the two areas of vehicle determination area C and vehicle uncertainty area U, for the main vehicle h, the following penalty function is obtained: (1) Collision penalty: The Greiner-Hormann algorithm can detect whether there is an overlapping area between the determined area C of the main vehicle h and the determined area C of the surrounding vehicle j, and between the uncertain area U of the main vehicle h and the uncertain area U of the surrounding vehicle j. The Greiner-Hormann algorithm first calculates and sorts the edge intersections of the determined area C of the main vehicle h and the determined area C of the surrounding vehicle j, and between the uncertain area U of the main vehicle h and the uncertain area U of the surrounding vehicle j, to identify potential intersection locations. Then, the Greiner-Hormann algorithm generates a new polygonal area based on the state of the edge intersections, which represents the intersection area of ​​the determined area C of the main vehicle h and the determined area C of the surrounding vehicle j, and between the uncertain area U of the main vehicle h and the uncertain area U of the surrounding vehicle j. If the intersection area is a non-empty polygon, If the intersection area is an empty polygonal area, it means that there is an overlap between the determined area C of the main vehicle h and the determined area C of the surrounding vehicle j, and the uncertain area U of the main vehicle h and the uncertain area U of the surrounding vehicle j; if the intersection area is an empty polygonal area, it means that there is no overlap between the determined area C of the main vehicle h and the determined area C of the surrounding vehicle j, and the uncertain area U of the main vehicle h and the uncertain area U of the surrounding vehicle j; the three-dimensional information of the surrounding environment is obtained through the on-board laser radar and camera, and its position is updated in real time using the global positioning system and the inertial navigation system. This information supports the detection of the overlapping area between the determined area C of the main vehicle h and the determined area C of the surrounding vehicle j, and the uncertain area U of the main vehicle h and the uncertain area U of the surrounding vehicle j, and the results are input into the vehicle's overtaking decision module, so as to adjust the overtaking decision according to the real-time situation, thereby ensuring safe driving; If there is an overlapping area between the determined area C of the main vehicle h and the determined area C of the surrounding vehicle j, it means that the main vehicle h and the surrounding vehicle j will collide. At this time, the collision penalty function ψ 1,t Assign a value of -10; if there is no overlapping area between the determined area C of the main vehicle h and the determined area C of the surrounding vehicle j detected by the Greiner-Hormann algorithm, it means that the main vehicle h and the surrounding vehicle j will not collide, ψ 1,t Assign a value of 10; Among them, C t (h) represents the determined area C of the main vehicle h at the current discrete time t of the system, C t (j) represents the determined area C of the peripheral vehicle j at the current discrete time t of the system; O represents a positive integer between 1 and n except h, and n is the total number of the main vehicle h and the peripheral vehicles j; (2) Lane boundary penalty: Assume that the lane boundary is the boundary line of the vehicle driving area. If the determined area C detected by the Greiner-Hormann algorithm overlaps with the area outside the lane boundary, it means that the vehicle has left the vehicle driving area. The lane boundary penalty function ψ 2,t The value is -10. If the Greiner-Hormann algorithm detects that there is no overlapping area between the vehicle's determined area C and the area outside the lane boundary, it means that the vehicle is traveling within the vehicle driving area. 2,t Assign a value of 10; Among them, F t Represents the set of areas outside the lane boundary of the vehicle at the current discrete time t of the system; (3) Uncertain area penalty: If the Greiner-Hormann algorithm detects that there is an overlapping area between the uncertainty area U of the main vehicle h and the uncertainty area U of the surrounding vehicle j, it means that there is a possibility of collision between the main vehicle h and the surrounding vehicle j. The penalty function of the uncertainty area U is ψ 3,t The value is -10. If the Greiner-Hormann algorithm detects that there is no overlapping area between the uncertainty area U of the main vehicle h and the uncertainty area U of the surrounding vehicle j, it means that there is no possibility of collision between the main vehicle h and the surrounding vehicle j. 3,t Assign a value of 10; Among them, U t (h) represents the uncertainty region U of the main vehicle h at the current discrete time t of the system, U t (j) represents the uncertainty region U of the vehicle j at the current discrete time t of the system; (4) Smooth driving penalty: The comfort of the vehicle occupants is also an important part of the evaluation index. If the vehicle speed is too fast or too slow, it will affect the physical and mental comfort of the driver and passengers, and even lead to traffic accidents. The smooth driving penalty function is defined as ψ 4,t : ψ 4,t =-10·|a t | (9) Among them, a t is the acceleration of the vehicle at the current discrete time t of the system; (5) Frequent lane-changing penalty: If the main vehicle h changes lanes frequently, it will cause great trouble to the surrounding vehicle j, causing the surrounding vehicle j to be unable to make evasive actions in time, thus colliding with the main vehicle h. The frequent lane-changing penalty function is defined as ψ 5,t : ψ 5,t =-10·n cl (10) Among them, n cl Indicates the number of lane changes of the main vehicle h in the total system time, n cl and the vehicle's front wheel turning angle δ f The following relationship exists: f n is the number of lane changes per unit time of the host vehicle h, that is, the lane change frequency, t total is the total system time; k is a proportional constant, and the value of k depends on the system environment and driver characteristics. The above five penalty functions have different priorities and occupy different proportions in the overall penalty function; the collision of vehicles in the determined area C or the uncertain area U will directly lead to an accident, and the vehicle running out of the lane boundary and frequently changing lanes may also cause an accident; in contrast, smooth driving has a relatively low impact on the comfort of the driver and passengers in the vehicle, so its importance is relatively small; the weight factors of the five penalty functions are given as follows: α1, α2, α3, α4, α5, where each weight factor is a positive number, satisfying α1+α2+α3+α4+α5=1, and we get: α1＞α3＞α2＞α5＞α4 (12) Based on the prediction model of the vehicle in step 1, when the prediction time domain is N, the action input sequence Γ of the vehicle's two-degree-of-freedom dynamics model is given set ={Γ t ,Γ t+1 ,…,Γ t+N-1 }, according to the rolling prediction strategy, the stage penalty function R when the system prediction step is i is obtained t+i : R t+i =a T ψ t+i (13) Among them, ψ t+i =[ψ 1,t+i ,ψ 2,t+i ,ψ 3,t+i ,ψ 4,t+i ,ψ 5,t+i ] T , is the vector composed of the above five penalty functions, t+i is the prediction step length of the system in the i-th step, T is the transpose of the vector, α=[α1,α2,α3,α4,α5] T , is a vector of weight factors corresponding to the five penalty functions; then the stage penalty function of all prediction steps is obtained, and then the cumulative penalty function in the prediction time domain N is obtained Among them, λ i is the discount factor of the penalty function when the system prediction step is i, λ i ∈[0,1]; Step 3: Build the Level-k game framework between the main vehicle and the surrounding vehicles and solve the overtaking decision problem For the action input sequence Γ of the two-degree-of-freedom dynamic model of the main vehicle h set [h] = {Γ t [h],Γ t+1 [h],…,Γ t+N-1 [h]}, if we want to predict the next overtaking decision action of the main vehicle h, we need to predict the action input sequence Γ of the two-degree-of-freedom dynamic model of the surrounding vehicle j set [j] = {Γ t [j],Γ t+1 [j],…,Γ t+N-1 [j]}, As well as the status of the main vehicle h and the surrounding vehicle j, when the system prediction step is i, through S t+i Represents the state of all vehicles when the system prediction step is i: Among them, S t+i is the vehicle state set when the system predicts step size i, x t+i [h] is the lateral position of the center of mass of vehicle h when the system predicts step size i, y t+i [h] is the longitudinal position of the center of mass of vehicle h when the system predicts step size i, is the yaw angle of the vehicle h when the system predicts the step length i, v t+i [h] is the speed of vehicle h when the system predicts step size i; similarly, x t+i [j] is the lateral position of the center of mass of the vehicle j when the system predicts step size i, y t+i [j] is the longitudinal position of the center of mass of vehicle j when the system predicts step size i, is the yaw angle of vehicle j when the system predicts step size i, v t+i [j] is the speed of the vehicle j when the system predicts the step size is i, and T is the transpose of the vector; This method first considers all vehicles as Level-0, obtains the state set of surrounding vehicle j and the action input sequence of the two-degree-of-freedom dynamic model, then considers the main vehicle h as Level-1, and then considers the surrounding vehicle j as Level-0. Based on the state set of surrounding vehicle j and the action input sequence of the two-degree-of-freedom dynamic model, the state set of the main vehicle h and the action input sequence of the two-degree-of-freedom dynamic model can be obtained, and then the optimal overtaking decision action sequence of the main vehicle h can be obtained; this method does not consider levels above Level-1 to avoid increasing the computational complexity; the stage penalty function obtained by the main vehicle h when the system prediction step is i Depends on the initial state set S of all vehicles t , the action input sequence of the two-degree-of-freedom dynamic model of the main vehicle h The action input sequence of the two-degree-of-freedom dynamic model of Zhou Chej in, It represents the action input of the two-degree-of-freedom dynamic model of the Level-1 main vehicle h when the system predicts the step length to be N-1. It represents the action input of the two-degree-of-freedom dynamic model of Level-0 vehicle j when the system prediction step is N-1, and N is the system prediction time domain; therefore, the stage penalty function obtained by the main vehicle h when the system prediction step is i is: The cumulative penalty function of the main vehicle h is: This method uses the Q-Learning algorithm in reinforcement learning to solve the optimal action of the main vehicle h to overtake the vehicle at the current discrete moment of the system; the core idea of ​​the Q-Learning algorithm is to continuously learn and predict the action input sequence of the two-degree-of-freedom dynamic model of the main vehicle h and the action input sequence of the two-degree-of-freedom dynamic model of the surrounding vehicle j in the time domain N, and update the cumulative penalty function of the main vehicle h Finally, the optimal overtaking decision action sequence of the main vehicle h in the prediction time domain N is found; based on the cumulative penalty function of the main vehicle h in the above Level-k game framework By comparing the action sequences of different main vehicles h and surrounding vehicles j at the current discrete moment of the system Size, get the cumulative penalty function that makes the system main vehicle h at the current discrete moment The optimal overtaking decision action sequence for the largest host vehicle h is: in, It represents the optimal action sequence of the Level-1 main vehicle h in the current discrete moment of the system and the prediction time domain N; thus, the vehicle overtaking decision problem is solved.

Images