A motion planning method for autonomous vehicles based on multimodal prediction

Abstract

The present invention relates to a motion planning method for an autonomous driving vehicle based on multimodal prediction, and belongs to the technical field of autonomous driving vehicles. The method comprises: S1: based on the trajectory history of the vehicle and the trajectory history of surrounding vehicles, an LSTM model is used to predict the future multimodal behavior of surrounding vehicles, that is, the probability distribution of the future positions of surrounding vehicles under different modes; S2: based on the multimodal behavior predicted by the LSTM model in step S1, corresponding behavior branches of other vehicles and trajectory branches of the vehicle are constructed, and the corresponding branch probabilities are determined by multimodal probabilities and safety constraints; and the MPC algorithm is used to solve the feedback strategy in the form of trajectory branches. The present invention adopts a multimodal prediction model in the motion planning process, improves the prediction accuracy, optimizes the traditional MPC, and considers the mutual influence between the vehicle and other vehicles and the overall risk optimization through the feedback strategy.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving vehicles and relates to a motion planning method for an autonomous driving vehicle based on multimodal prediction. Background Art

[0002] The emergence of autonomous driving technology provides a new solution to reduce traffic accident rates, improves travel efficiency, and reduces the burden on drivers. As a key module connecting the perception layer and the control execution layer, the motion planning layer can plan reasonable trajectories for different traffic scenarios, thereby reducing the probability of accidents.

[0003] Traditional motion planning methods for autonomous vehicles mainly include direct construction, sampling, and machine learning. These methods have the disadvantages of being easily trapped in local minima during optimization calculations, losing algorithm completeness, and endangering road safety during data collection. Model predictive control (MPC) can formulate vehicle dynamics and safety constraints as finite range constraints, and has the advantages of predicting future trajectories and handling multi-objective constraints. It has become the main method for solving vehicle safety trajectory planning.

[0004] The complex traffic environment brings uncertainty to the vehicle's motion planning. Prediction can make motion planning safer and more reliable. Traditional motion planning methods are generally based on static information of the environment, which cannot effectively improve the safety of the vehicle during driving. They focus on vehicle trajectory prediction or trajectory planning, but pay insufficient attention to prediction-based motion planning. Therefore, there is an urgent need for a motion planning method for autonomous vehicles that can effectively combine prediction. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a motion planning method for an autonomous driving vehicle based on multimodal prediction. A multimodal prediction model is adopted in the motion planning process to improve the prediction accuracy, optimize the traditional MPC and consider the mutual influence between the vehicle and other vehicles and the overall risk optimization through a feedback strategy.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A motion planning method for autonomous vehicles based on multimodal prediction uses a deep learning algorithm long short-term memory (LSTM) model to collect road vehicle information for prediction, obtain the predicted vehicle state at future moments and the probability of taking different modes, build a backup trajectory based on the multimodal behavior of the uncontrolled body, and use the branch MPC algorithm to solve a feedback strategy in the form of a trajectory tree with the same scene tree topology, with the goal of minimizing the expected total cost. The method specifically includes the following steps:

[0008] S1: Based on the trajectory history of the ego vehicle and the trajectory history of surrounding vehicles, the LSTM model is used to predict the future multimodal behavior of surrounding vehicles, that is, the probability distribution of the future positions of surrounding vehicles under different modes;

[0009] S2: Based on the multimodal behavior predicted by the LSTM model in step S1, the corresponding behavior branches of other vehicles and the trajectory branches of the vehicle are constructed, and the corresponding branch probabilities are determined by the multimodal probabilities and safety constraints; and the feedback strategy is solved in the form of trajectory branches using the Model Predictive Control (MPC) algorithm.

[0010] Further, step S1 specifically includes the following steps:

[0011] S11: Construct the coordinates of the vehicle, with the origin fixed on the predicted vehicle at time t, the y-axis being the lane direction, and the x-axis being perpendicular to the y-axis;

[0012] S12: The encoder is an LSTM with shared weights, whose input is the predicted vehicle and other vehicles (tt h ) to the historical trajectory X of frame t:

[0013]

[0014]

[0015] Among them, (x0, y0) and (x1, y1) are the coordinates of the predicted vehicle and other vehicles at time t respectively;

[0016] For each moment, (tt h ) to t frame through the LSTM encoder to the predicted vehicle and surrounding vehicles; the LSTM state of each vehicle in the past t h The final LSTM state of each vehicle encodes the motion state of the vehicle.

[0017] S13: Define a lane-based grid to build a social tensor to improve the accuracy of future motion prediction; and use the social tensor as one of the inputs to the model; define a grid of a certain size around the predicted vehicle; form a social tensor by filling the grid with the positions of surrounding vehicles, and use convolution and pooling layers to obtain the social context encoding; the LSTM state of the predicted vehicle passes through a fully connected layer to obtain the vehicle dynamics encoding; concatenate the two encodings together to form a trajectory encoding, which is passed to the decoder;

[0018] S14: Use the LSTM-based decoder to generate the predicted vehicle’s future states from (t+1) to (t+tf )Frame coordinate y (t) And the modal probability P(Y|X):

[0019]

[0020]

[0021] P(Y|X)=∑P(Y|m i ,X)P(m i |X)

[0022] Among them, m i is the driving behavior under the i-th mode, Indicates the predicted vehicle's future modes from (t+1) to (t+t f ) frame y coordinate set, P(m i |X) represents the probability of each mode, P(Y|m i ,X) represents the position distribution of the predicted vehicle under each mode.

[0023] Further, step S2 specifically includes the following steps:

[0024] S21: Construct the behavior branches of other vehicles based on the multimodal behavior of the uncontrolled body, simplify the different modes into a set of strategies, each node of the behavior branch contains the state of the surrounding vehicles, and the nodes following the same strategy form a branch; the behavior branches correspond to different strategies π i The number of nodes in each branch is the same as the number of prediction frames of LSTM, and the probability of the strategy is determined by the modal probability P(m i |X) is given;

[0025] S22, construct the same trajectory branch according to the structure of the behavior branch, and the node of the trajectory branch represents the future state of the vehicle; construct the safe state set S according to the constraints of the lane boundary and the collision avoidance conditions, and control the obstacle function h S (x), the weight w corresponding to the i-th branch is obtained from the collision probability P i ;

[0026] x(t)∈S={x|h S (x)≥0}

[0027]

[0028] w i =w Pre(i) P(b i ∣x Pre(i) ,z Pre(i) )

[0029] Among them, x(t) is the set of allowed states, P(πi |x,z) is the collision probability, h(x,z) is the safe state set, η is the saturation parameter, b i represents the i-th branch, w Pre(i) represents the predicted branch probability, x Pre(i) ,z Pre(i) are the future states of the ego vehicle and surrounding vehicles respectively.

[0030] S23: Construct a cost function, which minimizes the weighted expected total cost:

[0031]

[0032]

[0033] w0=1,w i =P(m i |X)P(b i ∣x Pre(i) ,z Pre(i) ) (2)

[0034]

[0035] Wherein, formula (1) represents the dynamics of the ego vehicle and other vehicles in each branch; formula (2) represents w i The weight corresponding to the i-th branch is obtained from the collision probability P; Formula (3) represents the constraint on the initial conditions. i ,z i are the states of the vehicle and surrounding vehicles, u i is the control input sequence of the ego vehicle, is the state at time t, is the control input of t-1, The future and t f frame; are the states of the vehicle and surrounding vehicles at the initial moment, x t ,z t are the states of the vehicle and surrounding vehicles at time t respectively.

[0036] The beneficial effects of the present invention are:

[0037] 1) The present invention adopts a multimodal prediction model in the motion planning process, decouples the future multimodal behavior of the vehicle, improves the prediction accuracy, and adds a social pooling layer to consider the interaction between vehicles.

[0038] 2) The present invention optimizes the traditional MPC and makes full use of the prediction results of LSTM by using a branch structure, and considers the mutual influence between the vehicle and other vehicles and the overall risk optimization through a feedback strategy.

[0039] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0041] Figure 1 This is a diagram of a motion planning method for an autonomous driving vehicle based on multimodal prediction according to the present invention;

[0042] Figure 2 Schematic diagram of the LSTM prediction model. DETAILED DESCRIPTION

[0043] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0044] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0045] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0046] See also Figure 1-2 The present invention provides a motion planning method for an autonomous vehicle based on multimodal prediction, which uses the LSTM algorithm to predict other vehicles in the environment, obtain the state of surrounding vehicles at future moments and the probability of taking different behaviors, construct behavior branches according to the multimodal behavior of uncontrolled bodies, and use branch MPC to solve the feedback strategy of trajectory branches. Figure 1 As shown, the control method specifically includes the following steps:

[0047] S1: Use the LSTM model to predict the movement of surrounding vehicles and obtain the multimodal prediction distribution of future trajectories.

[0048] like Figure 1 As shown in the figure, a social tensor is established based on the trajectory history of the vehicle and the trajectory history of surrounding vehicles, and the future coordinates and modal probabilities of the vehicle under different modes are predicted through social pooling.

[0049] S11: Construct the coordinates of the vehicle, with the origin fixed on the predicted vehicle at time t, the y-axis being the lane direction, and the x-axis being perpendicular to it.

[0050] S12: The encoder is an LSTM with shared weights, whose input is the predicted vehicle and other vehicles (tt h ) to the historical trajectory of frame t:

[0051]

[0052]

[0053] Among them, (x0, y0) and (x1, y1) are the coordinates of the predicted vehicle and other vehicles at time t respectively.

[0054] For each moment, (tt h ) to t frame through the LSTM encoder to the predicted vehicle and surrounding vehicles. The LSTM state of each vehicle in the past t h The final LSTM state of each vehicle encodes the motion state of that vehicle.

[0055] S13: Define a lane-based grid to build a social tensor to improve the accuracy of future motion prediction. Use the social tensor as one of the inputs to the model. Define a 13×3 grid with a side length of 4.5m around the predicted vehicle. Form a social tensor by filling the grid with the positions of surrounding vehicles, and use convolution and pooling layers to get the social context encoding. The LSTM state of the predicted vehicle passes through a fully connected layer to get the vehicle dynamics encoding. Concatenate the two encodings together to form a trajectory encoding, which is passed to the decoder.

[0056] S14: Use the LSTM-based decoder to generate predictions of the vehicle’s future states from (t+1) to (t+t f )Frame coordinate y (t) And the modal probability:

[0057]

[0058]

[0059] P(Y|X)=∑P(Y|m i ,X)P(m i |X)

[0060] Among them, m i is the driving behavior under the i-th mode, Indicates the predicted vehicle's future modes from (t+1) to (t+t f ) frame y coordinate set, P(m i |X) represents the probability of each mode, P(Y|m i ,X) represents the position distribution of the predicted vehicle under each mode.

[0061] S2: According to the multimodal behavior in the LSTM model, the corresponding behavior branches of other vehicles and the trajectory branches of the vehicle are constructed, the corresponding branch probabilities are determined by the multimodal probability and safety constraints, and the feedback strategy is solved in the form of trajectory branches using the MPC algorithm.

[0062] like Figure 2 As shown, in step S2, according to the multimodal behavior in the LSTM model, the strategy corresponding to the behavior branch of the other vehicle is constructed. According to the behavior branch and the trajectory branch of the vehicle, the corresponding branch probability is determined by the multimodal probability and safety constraints, and then solved.

[0063] S21: Construct the behavior branch of other vehicles based on the multimodal behavior of the uncontrolled body, simplify the different modes into a set of strategies, each node of the behavior branch contains the state of the surrounding vehicles, and the nodes following the same strategy form a branch. The behavior branch corresponds to different strategies π i The number of nodes in each branch is the same as the number of prediction frames of LSTM, and the probability of the strategy is determined by the modal probability P(m i |X) is given.

[0064] S22: Construct the same trajectory branch according to the structure of the behavior branch. The nodes of the trajectory branch represent the future state of the vehicle. According to the constraints of the lane boundary and the collision avoidance conditions, construct the safe state set S and control the obstacle function h S (x), the weight w corresponding to the i-th branch is obtained from the collision probability P i .

[0065] x(t)∈S={x|h S (x)≥0}

[0066]

[0067] w i =w Pre(i) P(b i ∣x Pre(i) ,z Pre(i) )

[0068] Among them, x(t) is the set of allowed states, P(π i |x,z) is the collision probability, h(x,z) is the safe state set, η is the saturation parameter, b i represents the i-th branch, w Pre(i) represents the predicted branch probability, x Pre(i) ,z Pre(i) is the future state of the vehicle and surrounding vehicles.

[0069] S23: Construct a cost function, which minimizes the weighted expected total cost:

[0070]

[0071]

[0072] w0=1,w i =P(m i |X)P(b i ∣x Pre(i) ,z Pre(i) ) (2)

[0073]

[0074] Wherein, formula (1) represents the dynamics of the ego vehicle and other vehicles in each branch; formula (2) represents w i The weight corresponding to the i-th branch is obtained from the collision probability P; Formula (3) represents the constraint on the initial conditions; x i ,z i are the states of the vehicle and surrounding vehicles, u i is the control input sequence of the ego vehicle, is the state at time t, is the control input of t-1, The future and t f frame; are the states of the vehicle and surrounding vehicles at the initial moment, x t ,z t They are the states of the vehicle and surrounding vehicles at moment 1.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A motion planning method for an autonomous driving vehicle based on multimodal prediction, characterized in that: The method comprises the following steps: S1: Based on the trajectory history of the ego vehicle and surrounding vehicles, the LSTM model is used to predict the future multimodal behavior of surrounding vehicles, that is, the probability distribution of the future positions of surrounding vehicles under different modes. Specifically, the following steps are included: S11: Construct the coordinates of the vehicle, with the origin fixed on the predicted vehicle at time t, the y-axis being the lane direction, and the x-axis being perpendicular to the y-axis; S12: The encoder is an LSTM with shared weights, whose input is the ego vehicle and surrounding vehicles (tt h ) to the historical trajectory X of frame t: Among them, (x0, y0) and (x1, y1) are the coordinates of the vehicle and surrounding vehicles at time t respectively; For each moment, (tt h The historical trajectory fragments from frame t to frame t are passed to the predicted vehicle and surrounding vehicles through the LSTM encoder; the LSTM state of each vehicle is calculated in the past t h The final LSTM state of each vehicle encodes the motion state of the vehicle. S13: Define a lane-based grid to build a social tensor and use the social tensor as one of the inputs to the model; define a grid of a certain size around the predicted vehicle; form a social tensor by filling the grid with the positions of surrounding vehicles, and use convolution and pooling layers to obtain the social context encoding; pass the LSTM state of the predicted vehicle through a fully connected layer to obtain the vehicle dynamics encoding; concatenate the two encodings together to form a trajectory encoding and pass it to the decoder; S14: Use the LSTM-based decoder to generate the frame coordinates y from (t+1) to (t+tf) of the predicted vehicle in different future modes (t) And the modal probability P(Y|X): P(Y|X)=∑P(Y|m i ,X)P(m i |X) Among them, m i is the driving behavior under the i-th mode, represents the y coordinate set of the predicted vehicle in each future mode from (t+1) to (t+tf) frames, P(m i |X) represents the probability of each mode, P(Y|m i ,X) represents the position distribution of the predicted vehicle under each mode; S2: According to the multimodal behavior predicted by the LSTM model in step S1, the corresponding behavior branches of other vehicles are constructed, and the trajectory branches of the vehicle are planned based on the behavior branches. The corresponding branch probabilities are determined by the multimodal probabilities and safety constraints; and the feedback strategy is solved in the form of trajectory branches using the model predictive control MPC algorithm, which specifically includes the following steps: S21: Construct the behavior branches of other vehicles based on the multimodal behavior of the uncontrolled body, simplify the different modes into a set of strategies, each node of the behavior branch contains the state of the surrounding vehicles, and the nodes following the same strategy form a branch; the behavior branches correspond to different strategies π i The number of nodes in each branch is the same as the number of prediction frames of LSTM, and the probability of the strategy is determined by the modal probability P(m i |X) is given; S22, construct the same trajectory branch according to the structure of the behavior branch, and the node of the trajectory branch represents the future state of the vehicle; construct the safe state set S according to the constraints of the lane boundary and the collision avoidance conditions, and control the obstacle function h S (x), the weight w corresponding to the i-th branch is obtained from the collision probability P i ; x(t)∈S={x∣h S (x)≥0} w i =w Pre(i) P(b i ∣x Pre(i) ,z Pre(i) ) Among them, x(t) is the set of allowed states, P(π i |x,z) is the collision probability, h(x,z) is the safe state set, η is the saturation parameter, b i represents the i-th branch, w Pre(i) represents the predicted branch probability, x Pre(i) ,z Pre(i) are the future states of the vehicle and surrounding vehicles respectively; S23: Construct a cost function, which minimizes the weighted expected total cost: w0=1,w i =P(m i |X)P(b i ∣x Pre(i) ,z Pre(i) ) (2) Wherein, formula (1) represents the dynamics of the ego vehicle and other vehicles in each branch; formula (2) represents w i The weight corresponding to the i-th branch is obtained from the collision probability P; Formula (3) represents the constraint on the initial conditions; x i ,z i are the states of the vehicle and surrounding vehicles, u i is the control input sequence of the ego vehicle, is the state at time t, is the control input of t-1, The future and t f frame; are the states of the vehicle and surrounding vehicles at the initial moment, x t ,z t are the states of the vehicle and surrounding vehicles at time t respectively.

Images