Vehicle multi-modal trajectory prediction method and system considering space-time interaction information

Abstract

The invention provides a vehicle multi-mode trajectory prediction method and system considering space-time interaction information, and relates to the field of automatic driving vehicle trajectory prediction. The method comprises the following steps: collecting scene information when a vehicle runs; a space-time attention model is selected as a base model, the scene information is imported into the base model for coding, and coding information fusing a space-time attention mechanism and social grid information of surrounding vehicles are obtained; selecting an automatic driving intention prediction model, fusing the coded information and the social grid information, and inputting the fused information and the social grid information into the automatic driving intention prediction model to obtain an intention prediction probability of the vehicle in a future time domain; constructing a decoder; and inputting the spliced intention prediction probability and coding information into a decoder, and decoding to obtain a prediction track under the corresponding intention. According to the method, multi-modal trajectory prediction including transverse and longitudinal motives of the intelligent driving vehicle at future moments can be realized, and support is provided for subsequent path planning and decision making of the vehicle.

Description

Technical Field

[0001] The present invention relates to the field of autonomous vehicle trajectory prediction, and particularly to a vehicle multi-modal trajectory prediction method and system considering spatio-temporal interaction information. Background Art

[0002] In the fields of transportation and autonomous driving, trajectory prediction is a key task, aiming to infer the potential positions and movement trajectories of traffic participants (such as vehicles, pedestrians, cyclists, etc.) at future moments based on historical observations and environmental information. High-precision trajectory prediction is an important prerequisite for achieving safe and efficient decision-making (decision-making planning, collision avoidance, speed control, etc.).

[0003] Traditional prediction methods based on kinematic or rule-based models achieve short-term accurate modeling by using common uniform motion, acceleration models, etc., but in complex scenarios and long-term prediction tasks, it is difficult to guarantee the accuracy.

[0004] Deep learning-driven prediction methods can capture complex non-linear relationships in data, and achieve prediction in multi-dimensional feature scenarios by building deep networks and training with massive data. Sequence models based on RNN (such as LSTM, GRU) can better capture the dynamic changes of time series and make multi-step predictions by combining scene information. In recent years, Transformer, spatio-temporal graph convolutional network (Graph Neural Networks, GNN), etc. have also been widely used to more flexibly model scene interactions and temporal features.

[0005] However, the same historical information in the real world may lead to multiple future trajectories (such as going straight or merging), and different prediction branches need to be explicitly expressed. Due to the uncertainty of traffic scenarios in real life, trajectory prediction emphasizes more on the prediction distribution rather than the prediction result of a single trajectory.

[0006] In order to make up for the deficiencies of existing trajectory prediction methods, which often lack in multi-modality and uncertainty modeling and are difficult to fully express the possible multiple future motion patterns of traffic participants, taking deep neural networks as the main means, combined with the modeling of scene encoding, intention prediction, social interaction, and multi-modal uncertainty, has become the mainstream route for high-precision trajectory prediction. It provides safer and more efficient support for intelligent decision-making in autonomous driving and complex traffic scenarios. Summary of the Invention

[0007] Objective of the Invention: To propose a multi-modal trajectory prediction method for vehicles considering spatio-temporal interaction information, and to propose a system that can automatically execute the above prediction method, aiming to achieve multi-modal trajectory prediction including lateral and longitudinal motivations of intelligent vehicles at future moments based on the historical information of the traffic scene where the intelligent vehicle is located, and to provide support for subsequent path planning and decision-making of intelligent vehicles.

[0008] In the first aspect of the present invention, a multi-modal trajectory prediction method for vehicles considering spatio-temporal interaction information is proposed, and the steps are as follows:

[0009] Collect the scene information when the vehicle is driving; the scene information includes the trajectory information of the vehicle itself and the trajectory information of surrounding vehicles within a predetermined range before and after in adjacent lanes in the same direction of travel.

[0010] Select an open-source spatio-temporal attention model as the base model, import the scene information into the base model for encoding, and obtain the encoded information H incorporating the spatio-temporal attention mechanism enc and the social grid information h of surrounding vehicles nbrs ;

[0011] Select an open-source autonomous driving intention prediction model, fuse the encoded information H enc and the social grid information h nbrs and input them into the autonomous driving intention prediction model to obtain the intention prediction probability P of the vehicle itself within the future time domain.

[0012] Construct a decoder; input the concatenated intention prediction probability P and the encoded information H enc into the decoder to decode and obtain the predicted trajectory corresponding to the intention.

[0013] In a further embodiment of the first aspect, the scene information I includes the trajectory information I of the vehicle itself ego and the trajectory information I of surrounding vehicles within a predetermined range before and after in adjacent lanes in the same direction of travel nbrs :

[0014] I = [I ego , I nbrs

[0015] The trajectory information I of the vehicle itself ego = {x, y, v x , v y , a x , a y , lane};

[0016] Among them, x represents the abscissa information of the vehicle, y represents the ordinate information of the vehicle, v x , v y are respectively the velocity components of the vehicle in the horizontal and vertical directions, ax , a y are the acceleration components of the vehicle in the transverse and longitudinal directions, respectively, and lane represents the lane number.

[0017] In a further embodiment of the first aspect, the intention prediction probability P of the host vehicle within the future time domain is P = {P lon , P lat};

[0018] where P lon represents the longitudinal driving intention probability, including the probabilities of accelerating, maintaining speed, and decelerating; P lat represents the lateral driving intention probability, including the probabilities of changing lanes to the left, changing lanes to the right, and going straight.

[0019] In a further embodiment of the first aspect, importing the scenario information into the base model for encoding specifically includes:

[0020] Embedding the trajectory information I of the host vehicle ego through an encoder to obtain the embedding state e of the host vehicle ego :

[0021] e ego = MLP(I ego , W emb )

[0022] Embedding the trajectory information I of the surrounding vehicles nbrs through an encoder to obtain the embedding state e of the surrounding vehicles nbrs :

[0023] e nbrs = MLP(I nbrs , W emb )

[0024] In the formula, MLP(*) is a multi-layer perceptron, and W emb is the weight coefficient of the embedding layer;

[0025] Inputting the embedding state e of the host vehicle ego and the embedding state e of the surrounding vehicles nbrs into the LSTM layer respectively for trajectory feature encoding to obtain their respective hidden layer feature vectors θ:

[0026] θ ego = LSTM(e ego , W emb )

[0027] θ nbrs = LSTM(e nbrs , W emb )

[0028] wherein, LSTM(*) represents encoding through the LSTM layer; θ egi represents the hidden layer feature vector of the host vehicle after LSTM encoding; θ nbrs represents the hidden layer feature vector of surrounding vehicles after LSTM encoding.

[0029] In a further embodiment of the first aspect, the scenario information is imported into the base model for encoding to obtain the encoded information H integrating the spatio-temporal attention mechanism enc :

[0030]

[0031] wherein, h is the output of the temporal attention mechanism, is the output of the spatial attention mechanism, and α, β, γ are learnable weight parameters for controlling the contribution degree of each part.

[0032] In a further embodiment of the first aspect, the scenario information is imported into the base model for encoding to obtain the social grid information h of surrounding vehicles nbrs , specifically including:

[0033] Using the mask matrix mask to sparsely extract the normalized hidden layer feature vector θ of surrounding vehicles layernorm to obtain the social grid information h of surrounding vehicles nbrs :

[0034] h nbrs = θ layernorm ⊙ mask

[0035] wherein, the normalized hidden layer feature vector of surrounding vehicles is the hidden layer feature vector of surrounding vehicles at the i-th time step, μ is the mean of the hidden vector, σ 2 represents the variance, and ∈ is a constant.

[0036] In a further embodiment of the first aspect, obtaining the intention prediction probability P of the host vehicle in the future time domain specifically includes:

[0037] Fusing the encoded information H enc and the social grid information h nbrs through the attention mechanism:

[0038] H fused = Attention(H enc , h nbrs )

[0039] wherein, H fused represents the fused feature, and Attention(*) is the attention mechanism;

[0040] Extract features from the fused feature H fused by means of a fully connected layer:

[0041] H trans = ReLU(H fused W trans + b trans )

[0042] where H trans represents the extracted feature, W trans is the weight matrix of the fully connected layer, b trans is the bias vector, and ReLU(*) is the activation function;

[0043] Predict the high-level intention category P high of the traffic participant:

[0044] P high = Softmax(H trans W high + b high )

[0045] W high is the weight matrix for high-level intention classification, and b high is the bias vector;

[0046] Based on the high-level intention category P hig , refine to obtain the longitudinal driving intention probability P lon , the lateral driving intention probability P lat :

[0047] P lat = {p lc , p rc , p kl} = Softmax(H lat W lat + b kat )

[0048] P lon = {p acc , p keep , p dec} = Softmax(H lon W lon + b lon )

[0049] where W lat , W lon are the weight matrices for low-level intention classification, and b lat , b lon are the bias vectors.

[0050] In a further embodiment of the first aspect, the decoder has six branches, each branch corresponding to an intention;

[0051] After receiving the shared features, the six branches independently decode to generate corresponding future trajectories. The structure of each branch is as follows:

[0052]

[0053] wherein, are the hidden state and memory state of the k-th branch at time step t, respectively, and LSTM k represents the LSTM network of the k-th branch, is the predicted output coordinate of the k-th branch at time step t. is the mapping parameter of the output;

[0054] Through recursive generation over T time steps, the multi-modal trajectory prediction results under 6 intentions are integrated and obtained

[0055]

[0056] wherein, are the left lane change intention, right lane change intention, left accelerating lane change intention, right accelerating lane change intention, straight driving at a constant speed intention, and executing deceleration intention, respectively.

[0057] In the second aspect of the present invention, a vehicle multi-modal trajectory prediction system is proposed. The prediction system includes at least one processing module and a memory storing computer program instructions; when the at least one processing module executes the computer program instructions, the vehicle multi-modal trajectory prediction method considering spatio-temporal interaction information as described in the first aspect is implemented.

[0058] In the third aspect of the present invention, a computer-readable storage medium is proposed. At least one executable instruction is stored in the storage medium. When the executable instruction runs on an electronic device, the electronic device executes the vehicle multi-modal trajectory prediction method considering spatio-temporal interaction information as described in the first aspect.

[0059] Compared with the prior art, the present invention has at least the following beneficial effects:

[0060] (1) Separating the base model from the sub-model, the overall trajectory prediction task is divided into two independent modules: the base model focuses on encoding historical data in the scene, extracting spatio-temporal interaction information, and performing an abstract representation of the global situation; while the sub-model generates future candidate trajectories based on the output of the base model. This modular design facilitates independent optimization of the system, reduces the complexity of model operation, enables the system to have better scalability and flexibility, and the independent design between modules facilitates debugging and optimization of each link, and is convenient for deployment in large-scale real-time scenarios.

[0061] (2) Introduce multi-level coarse and fine-grained intention classification and recognition: make a coarse-grained determination through high-level intention classification (maintaining lane / lanekeeping), and then achieve joint prediction of lateral and longitudinal intentions to realize a finer-grained intention classification, which helps to guide the discretized branches of future trajectories. In addition, a model training loss function for multi-level classification is proposed to improve the semantic interpretability and practical usability of the intention prediction results. Brief Description of the Drawings

[0062] Figure 1 is the overall architecture diagram of the vehicle multi-modal trajectory prediction method in the embodiment of the present invention. Detailed Embodiments

[0063] In the following description, a large number of specific details are given to provide a more thorough understanding of the present invention. However, it is obvious to those skilled in the art that the present invention can be implemented without one or more of these details. In other instances, in order to avoid confusion with the present invention, some technical features well known to those skilled in the art are not described.

[0064] This embodiment discloses a vehicle multi-modal trajectory prediction method considering spatio-temporal interaction information, and the steps are as follows:

[0065] S1. Collect the scene information when the vehicle is driving, including the trajectory information of its own vehicle and the trajectory information of the surrounding vehicles within 100 meters before and after in the adjacent lanes in the same driving direction, that is:

[0066] I = [I ego , I nbrs

[0067] I ego = {x, y, v x , v y , a x , a y , lane}

[0068] In the formula, I ego , I nbrs are the trajectory information of the target vehicle and the surrounding vehicles respectively. Taking I ego as an example, x represents the abscissa information of the vehicle, y represents the ordinate information of the vehicle, v x , v y are the velocity components of the vehicle in the horizontal and vertical directions respectively, a x , a y are the acceleration components of the vehicle in the horizontal and vertical directions respectively, and lane represents the lane number.

[0069] ​S2. Based on the data information collected in S1, use the base model to encode the collected scene information to obtain the encoded information H incorporating spatio-temporal attention mechanism enc and the social grid information h representing surrounding vehicles nbrs . Here, the base model adopts an open-source spatio-temporal attention model, which can be STAR (Spatio-Temporal Attention Network), ST-GCN (Spatial-Temporal Graph Convolutional Networks), or Trajectory Transformer

[0070] S3. Based on the encoded information H obtained by encoding in S2 enc and the social grid information h nbrs , input them into the autonomous driving intention prediction model for pooling operation to obtain the predicted driving probability P = {P lon , P lat} of the ego vehicle within the future time domain, where the longitudinal driving intention probability P lon includes p acc , p keep , p dec , and the lateral driving intention probability P lat includes p lc , p rc , p kl . Among them, p acc , p keep , p dec respectively represent the probabilities of accelerating, maintaining vehicle speed, and decelerating, and p lc , p rc , p kl respectively represent the probabilities of left lane change, right lane change, and going straight. The autonomous driving intention prediction model adopts an open-source model, which can be CAR-Net (Context-Aware RoadNetwork), IntentNet, PRECOG (PREdiction Conditioned On Goals), or MultiPath

[0071] S4. Concatenate the intention prediction probability obtained in S3 with the scene encoded information H obtained in S2 enc and input them into each sub-model respectively for decoding to obtain the predicted trajectories under the corresponding intentions

[0072] As a preferred embodiment, in S1, information about the host vehicle and the surrounding environment is collected by installing a positioning module, a speed sensor, an acceleration sensor, a lidar, a camera, and a radar sensor on the vehicle, including vehicle driving information, lane information, the state information of vehicles within 100 meters before and after the lane where the host vehicle is located and the adjacent lanes, and the relative position to the host vehicle.

[0073] As a preferred embodiment, in S2, the base model is a spatio-temporal attention mechanism model, and the encoding process of the base model includes:

[0074] S2.1. Feature encoding of the input data, including:

[0075] S2.1.1. Embedded representation of the host vehicle and adjacent vehicle information is obtained through their respective encoders to get e ego ,e nbrs :

[0076] e = MLP(I, W emb )

[0077] In the formula, e represents the embedded state, MLP is a multi-layer perceptron, and W emb is the weight coefficient of the embedding layer.

[0078] S2.1.2. The embedded states of different types of traffic participants obtained in S2.1.1 are input into the LSTM layer for trajectory feature encoding and layer normalization. Taking the host vehicle as an example, the process is as follows:

[0079] θ ego = LSTM(e ego , W emb )

[0080]

[0081] In the formula, LSTM(·) represents encoding through the LSTM layer, W emb is the weight coefficient, θ ego represents the hidden layer feature vector of the host vehicle after LSTM encoding, is the hidden layer feature vector at the i-th time step, μ is the mean of the hidden vectors, σ 2 represents the variance, and ∈ is a very small constant used to prevent division by zero errors. h i is the vector information at the i-th time step after layer normalization.

[0082] S2.1.3. Sparse extraction of the adjacent vehicle coding information in the social grid. The present invention defines a social grid centered on the vehicle itself, with a size of 3 * 41 and a single length of 5 meters, to represent the spatial feature information. Considering the sparsity of the social grid, the vehicle state information that is non-empty within the grid position is extracted. The expression is as follows:

[0083]

[0084] h nbrs = θ layernorm ⊙ mask

[0085] In the formula, mask is a mask matrix, used to represent the information of the existence of vehicles in the social grid. 1 indicates that there is a vehicle at this position at the current moment, and 0 indicates that there is no vehicle at the current moment. θ layernorm is the adjacent vehicle information after layer normalization, and h nbrs is the sparse adjacent vehicle coding information.

[0086] S2.2. Input the vehicle itself and the adjacent vehicle coding information into the spatial attention mechanism, focus on the spatial interaction between the vehicle itself and the adjacent vehicles at each time step, and dynamically adjust the attention to different adjacent vehicles to quantify the behavioral interaction between vehicles;

[0087] S2.3. Input the fused spatial distribution feature information into the temporal attention mechanism, integrate these spatio-temporal features over the entire time series, capture the temporal dependencies and global behavior patterns, and form a global feature representation;

[0088] The aforementioned vehicle multi-modal trajectory prediction method considering spatio-temporal interaction information, in S2.2, includes:

[0089] S2.2.1. Normalize the feature coding of traffic participants in the social grid to ensure that each participant obtains a consistent weight scale in the subsequent spatial attention mechanism.

[0090]

[0091] Among them, h i is the coding information of the i-th grid in the adjacent vehicles obtained in S2.1.3, and μ and σ respectively represent the mean and standard deviation of the social grid coding information.

[0092] S2.2.2. The vehicle itself, the normalized adjacent vehicle coding information h ego h nbrs,norm are projected to obtain the query, key, and value Q, K, V, and the attention H is calculated through the multi-head attention mechanism. The formula is as follows:

[0093] Q = h nbrs,norm W Q

[0094] K = h ego W K

[0095] V = h ego W V

[0096] H = MultiHead(Q, K, V) = {head 1 , head 2 , … head n}W O

[0097]

[0098] where W Q , W K , W V represent the trainable weight matrices of the projection respectively, head i represents the attention calculation result of each head, d k represents the dimension size of the vector of each head, and W O is the trainable weight matrix for linearly transforming the outputs of multiple heads.

[0099] S2.2.3. Input the spatial feature information H extracted by the spatial attention mechanism into the dual gated linear unit (D_GLU):

[0100]

[0101] where W 1 , W 2 are trainable weight matrices, and b 1 , b 2 are bias vectors. σ is the sigmoid activation function for the first gating, and φ is the tanh activation function for the second gating operation. The first gating function converts the input feature H into a gating vector between 0 and 1 through the sigmoid function, determining which feature information needs to be retained or suppressed; the second gating function further transforms the features through non - linear activation, increasing the non - linear expression ability of the features to capture the complex patterns of the model; then, the outputs of the two gating functions are multiplied element - by - element with the original feature H, and a residual connection is made.

[0102] As a preferred embodiment, in S2.3, it includes:

[0103] S2.3.1. Input all the spatial feature information in the historical time domain into the multi - head self - attention mechanism to capture the contribution relationship at each time step, and obtain the variable h containing time - feature information.

[0104]

[0105] h = MultiHead(Q t , K t , V t )

[0106] where represent the trainable weight matrices of the projection respectively.

[0107] S2.3.2. Input the temporal feature information h extracted by the temporal attention mechanism into the dual gated linear unit (D_GLU):

[0108]

[0109] where W 3 , W 4 are trainable weight matrices, and b 3 , b 4 are bias vectors.

[0110] S2.3.3. Perform weighted residual connection on the temporal feature information and the spatial feature information to obtain H enc :

[0111]

[0112] where h is the output of the temporal attention mechanism, is the output of the spatial attention mechanism, and α, β, γ are learnable weight parameters used to control the contribution degrees of each part.

[0113] As a preferred embodiment, in S3, it specifically includes:

[0114] S3.1. Perform feature fusion on the encoded information H enc obtained in S2 and the social grid information h nbrs through the attention mechanism:

[0115] H fused = Attention(H enc , h nbrs )

[0116] S3.2. Perform feature extraction on the fused feature H fused through the fully connected layer:

[0117] H trans = ReLU(H fused W trans + b trans )

[0118] where Wtrans is the weight matrix of the fully connected layer, b trans is the bias vector. ReLU is the activation function.

[0119] S3.3. The extracted feature H trans Construct a hierarchical multi-task intent classification module to achieve refined intent prediction layer by layer:

[0120] S3.3.1. First, predict the high-level intent category P high (Keep lane / Change lane):

[0121] P high = Softmax(H trans W high + b high )

[0122] where W high is the weight matrix for high-level intent classification, b high is the bias vector.

[0123] S3.3.2. Based on the high-level intent, further refine and predict the specific lateral and longitudinal intents. Introduce the idea of multi-task learning to achieve the joint prediction of lateral and longitudinal intents through shared and exclusive feature extraction layers:

[0124] H share = ReLU(H trans W share + b share )

[0125] H lat = ReLU(H share W lat + b lat )

[0126] H lon = ReLU(H share W lon + b lon )

[0127] where, W share , W lat , W lo are the weight matrices of the shared feature extraction layer, the lateral / longitudinal feature extraction layer respectively, b share , b lat , b lo are the corresponding bias vectors.

[0128] S3.3.3. Classify the lateral and longitudinal intents, i.e., the low-level intents, respectively to obtain probability values:

[0129] P lat = {plc , p rc , p kl} = Softmax(H lat W lat + b lat )

[0130] P lon = {p acc , p keep , p dec} = Softmax(H lon W lon + b lon )

[0131] Among them, W lat , W lon are the weight matrices for low-level intent classification, and b lat , b lon are the bias vectors.

[0132] S3.4. Calculate the intent prediction loss During the model training process, introduce a hierarchical loss function to not only optimize the classification accuracy of high-level intents but also optimize the fine-grained classification accuracy of low-level intents; introduce a feature orthogonality loss to avoid feature interference between tasks, ensure the orthogonality of shared features and task-specific features in the feature space, and improve the effect of multi-task learning:

[0133]

[0134] Among them, corresponds to the cross-entropy losses of high-level intents, horizontal intents, and vertical intents respectively. The calculation formula of the cross-entropy loss is shown as follows; λ high , λ low , λ ortho correspond to the weight coefficients of high-level intents, low-level intents, and orthogonality losses respectively.

[0135]

[0136] Among them, C represents the total number of categories, y = (y 1 , y 2 , …, y c ) is the one-hot encoding of the true label, is the predicted probability distribution of the output.

[0137] As a preferred embodiment, in S4, it specifically includes:

[0138] S4.1. Fuse the encoded information H of S2 encAnd the intention probability P obtained by S3 is used to obtain the initial feature H of the decoder cond

[0139] S4.2. Feed H cond Into the decoder. The decoder part is designed with 6 branches corresponding to 6 intentions, and the future trajectories under 6 intentions are generated correspondingly

[0140] The aforementioned vehicle trajectory fusion prediction method based on deep learning and kinematic models includes, in S4.2:

[0141] S4.2.1. The initial feature enters the shared layer for feature extraction:

[0142] H share_out = MLP(H cond )

[0143] Where H share_out Is the shared feature of the shared layer, and MLP is a multi-layer perceptron

[0144] S4.2.2. After receiving the shared feature, 6 branches independently decode to generate the corresponding future trajectories. The structure of each branch is as follows:

[0145]

[0146] Wherein, Are the hidden state and memory state of the k-th branch at time step t respectively, and LSTM k Represents the LSTM network of the k-th branch Is the predicted output coordinate of the k-th branch at time step t Is the mapping parameter of the output

[0147] S4.2.3. Through recursive generation of T time steps, the multi-modal trajectory prediction results under 6 intentions are integrated

[0148] Using the logic and process of the vehicle multi-modal trajectory prediction method disclosed in the above embodiments, a vehicle multi-modal trajectory prediction system can be formed. The prediction system includes at least one processing module and a memory storing computer program instructions; when at least one processing module executes the computer program instructions, the vehicle multi-modal trajectory prediction method considering spatio-temporal interaction information disclosed in the above embodiments is implemented, which will not be elaborated here

[0149] In addition, for the vehicle multi-modal trajectory prediction method disclosed in the above embodiments, during actual application, its operation logic can be written as a program product, which is written into a storage medium and runs on an electronic device. More specific examples of the computer-readable storage medium mentioned in this embodiment may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. The computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries the readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than the readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.

[0150] As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as a limitation of the present invention itself. Various changes may be made in its form and details without departing from the spirit and scope of the present invention defined by the appended claims.

Claims

1. A vehicle multimodal trajectory prediction method considering spatiotemporal interactive information, characterized in that: The steps include: Collecting scene information when the vehicle is traveling; the scene information includes the trajectory information of the own vehicle and the trajectory information of surrounding vehicles within a predetermined range before and after the adjacent lane when traveling in the same direction; Select the open source spatiotemporal attention model as the base model, import the scene information into the base model for encoding, and obtain the encoding information H that integrates the spatiotemporal attention mechanism enc And the social grid information of surrounding vehicles nbrs ; Select the open source autonomous driving intention prediction model and integrate the encoding information H enc and social grid information nbrs And input it into the autonomous driving intention prediction model to obtain the intention prediction probability P of the own vehicle in the future time domain; Build a decoder; The concatenated intention prediction probability P and encoded information H enc The data is input to the decoder and decoded to obtain the predicted trajectory under the corresponding intention.

2. The vehicle multimodal trajectory prediction method considering spatiotemporal interaction information according to claim 1 is characterized in that: The scene information I includes the trajectory information I of the vehicle itself ego And the trajectory information of surrounding vehicles within a predetermined range before and after the adjacent lane when traveling in the same direction I nbrs : I=[I ego ,I nbrs ] The trajectory information of the own vehicle I ego ={x,y,v x ,v y ,a x ,a y ,lane}; Among them, x represents the vehicle's horizontal coordinate information, y represents the vehicle's vertical coordinate information, and v x ,v y are the velocity components of the vehicle in the horizontal and vertical directions, a x ,a y They are the acceleration components of the vehicle in the horizontal and vertical directions respectively, and lane represents the lane number.

3. The vehicle multimodal trajectory prediction method considering spatiotemporal interaction information according to claim 1, characterized in that: The predicted probability of the vehicle's intention in the future time domain P = {P lon ,P lat }; Among them, P lon represents the longitudinal driving intention probability, including the probability of acceleration, maintaining speed, and deceleration; P lat Represents the probability of lateral driving intention, including the probability of changing lanes left, changing lanes right, and going straight.

4. The vehicle multimodal trajectory prediction method considering spatiotemporal interaction information according to claim 1, characterized in that: Importing the scene information into the base model for encoding specifically includes: The trajectory information of the vehicle I ego The embedded state of the vehicle is obtained by using the encoder to perform embedded representation ego : and ego =MLP(I ego ,IN emb ) The trajectory information of surrounding vehicles I nbrs The encoder is used to perform embedded representation to obtain the embedded state of the surrounding vehicles. nbrs : and nbrs =MLP(I nbrs ,IN emb ) Where MLP(*) is a multi-layer perceptron, W emb is the weight coefficient of the embedding layer; The embedded state of the vehicle is e ego and the embedded state of the surrounding vehicles e nbrs Input the LSTM layer respectively for trajectory feature encoding to obtain their respective hidden layer feature vectors θ: θ ego =LSTM(e ego ,W emb ) θ nbrs =LSTM(e nbrs ,W emb ) Where, LSTM(*) indicates encoding through the LSTM layer; θ ego Represents the hidden layer feature vector of the vehicle after LSTM encoding; θ nbrs Represents the hidden layer feature vector of the surrounding vehicles after LSTM encoding.

5. The vehicle multimodal trajectory prediction method considering spatiotemporal interaction information according to claim 4 is characterized in that: The scene information is imported into the base model for encoding to obtain the encoding information H fused with the spatiotemporal attention mechanism enc : Where h is the output of the temporal attention mechanism, is the output of the spatial attention mechanism, and α, β, and γ are the weight parameters learned in class, which are used to control the contribution of each part.

6. The vehicle multimodal trajectory prediction method considering spatiotemporal interaction information according to claim 4, characterized in that: The scene information is imported into the base model for encoding to obtain the social grid information h of the surrounding vehicles nbrs , specifically including: Use the mask matrix mask to normalize the hidden layer feature vector θ of the surrounding vehicles after layer normalization layernorm Perform sparse extraction to obtain the social grid information h of the surrounding vehicles nbrs : h nbrs =θ layernorm ⊙mask Among them, the normalized hidden layer feature vector of surrounding vehicles is is the hidden layer feature vector of the surrounding vehicles at the i-th time step, μ is the hidden vector mean, σ 2 represents variance, ∈ is a constant.

7. The vehicle multimodal trajectory prediction method considering spatiotemporal interaction information according to claim 3, 5 or 6, characterized in that: The intention prediction probability P of the vehicle in the future time domain is obtained, specifically including: Fusion of encoded information H through attention mechanism enc and social grid information nbrs : H fused =Attention(H enc ,h nbrs ) In the formula, H fused represents the fused features, Attention(*) is the attention mechanism; Through the fully connected layer, the fused feature H fused Perform feature extraction: A trans =ReLU(H fused W trans +b trans ) In the formula, H trans represents the extracted features, W trans is the weight matrix of the fully connected layer, b trans is the bias vector, ReLU(*) is the activation function; Predict the high-level intention category P of traffic participants high : P high =Softmax(H trans W high +b high ) W high is the weight matrix for high-level intent classification, b high is the bias vector; Based on the high-level intent category P high , and the longitudinal driving intention probability P is obtained by refining lon , lateral driving intention probability P lat : P lat ={p lc 、p rc 、p kl }=Softmax(H lat W lat +b lat ) P lon ={p acc 、p keep 、p dec }=Softmax(H lon W lon +b lon ) Among them, W lat , W lon is the weight matrix for low-level intent classification, b lat 、b lon is the bias vector.

8. The vehicle multimodal trajectory prediction method considering spatiotemporal interaction information according to claim 1, characterized in that: The decoder has six branches, each branch corresponding to an intent; After receiving the shared features, the six branches independently decode and generate the corresponding future trajectories. The structure of each branch is as follows: In the formula, are the hidden state and memory state of the k branch at time step t, LSTM k represents the LSTM network of the kth branch, is the predicted output coordinate of branch k at time step t; is the output mapping parameter; After T time steps of recursive generation, the multimodal trajectory prediction results under 6 intentions are integrated. In the formula, They are left lane change intention, right lane change intention, left lane change intention with acceleration, right lane change intention with acceleration, straight-ahead constant speed intention, and execution deceleration intention.

9. A vehicle multimodal trajectory prediction system, characterized in that: include: at least one processing module, and a memory storing computer program instructions; When at least one processing module executes the computer program instructions, the vehicle multimodal trajectory prediction method considering spatiotemporal interaction information as described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one executable instruction. When the executable instruction is executed on the electronic device, the electronic device executes the vehicle multimodal trajectory prediction method considering spatiotemporal interaction information as described in any one of claims 1 to 8.

Images