A multimodal trajectory prediction method, medium and system

Abstract

The application discloses a multi-modal trajectory prediction method, medium and system, comprising: establishing a trajectory prediction network, wherein the trajectory prediction network comprises a past trajectory encoder module, a feature storage library module storing trajectory features, a decoder module and a Transformer model; training the trajectory prediction network using sample trajectories, wherein the sample trajectories comprise continuous frame position coordinates of multiple traffic participants; inputting past trajectories of multiple traffic participants to be predicted into the trained trajectory prediction network, and outputting predicted future trajectories of the multiple traffic participants to be predicted. The application is based on a discrete vector feature storage library, and can convert continuous features into discrete features using corresponding structures; through the Transformer model, the internal correlation and regularity structure of the features can be learned; and more key information can be retained while stronger multi-modal trajectory prediction performance is realized.

Description

Technical Field

[0001] This invention relates to the field of trajectory prediction technology, and in particular to a multimodal trajectory prediction method, medium, and system. Background Technology

[0002] Trajectory prediction is a core module in autonomous driving, typically located behind the perception module and before the control module, serving as a crucial link between them. Its purpose is to predict the future movement of traffic participants based on observed past states. In recent years, significant efforts have been made to improve the performance of trajectory prediction, making a substantial contribution to the development of autonomous vehicles. Generally, trajectory prediction employs two main methods: parameter-based methods and repository-based methods.

[0003] To predict the future movements of traffic participants, models need to learn useful experiences from a large number of past-future trajectories in the dataset, and then use these experiences to predict the future trajectories of current traffic participants. Parametric methods use training data to optimize model parameters. In this way, during the optimization process, all the experience is implicitly summarized and stored in a single model as a whole. Parametric methods have been extensively studied and have formed a large number of mature theories. For example, using encoder-decoder architectures, considering generator-discriminator architectures to regress future trajectory prediction; using conditional variational autoencoders to sample multiple future trajectory embeddings from the latent distribution; relying on bivariate Gaussian mixture models to output position distributions; and repository-based methods are inspired by the neuropsychological concept of human retrospective memory, i.e., humans learn future actions by recalling previously learned information; and utilize memory mechanisms to achieve single-agent trajectory prediction, where the instances stored in the repository focus more on intent and can more effectively remember possible movement patterns.

[0004] Current trajectory prediction methods based on repositories store representative instances to provide a clearer connection between the current traffic participant's situation and the instances in the repository. However, due to factors such as search efficiency, the size of the repository is greatly limited, resulting in an incomplete record of instances. Specifically, each instance contains several features; two similar instances contain a large number of identical features and a small number of different features, while two dissimilar instances contain a large number of different features and a small number of identical features. Therefore, the representative instances in the repository contain a large number of identical features while losing many crucial features. Summary of the Invention

[0005] This invention provides a multimodal trajectory prediction method, medium, and system to address the problem of inaccurate predictions caused by limitations in the size of the feature repository and the number of stored features in existing technologies.

[0006] Firstly, a multimodal trajectory prediction method is provided, including:

[0007] A trajectory prediction network is established, wherein the trajectory prediction network includes: a past trajectory encoder module, a feature repository module storing trajectory features, a decoder module, and a Transformer model;

[0008] The trajectory prediction network is trained using sample trajectories, wherein the sample trajectories include the position coordinates of multiple traffic participants in consecutive frames;

[0009] The past trajectories of multiple traffic participants to be predicted are input into the trained trajectory prediction network, which outputs the predicted future trajectories of the multiple traffic participants to be predicted.

[0010] In a second aspect, a computer-readable storage medium is provided, wherein computer program instructions are stored on the computer-readable storage medium; when executed by a processor, the computer program instructions implement the multimodal trajectory prediction method as described in the first aspect embodiment.

[0011] Thirdly, a multimodal trajectory prediction system is provided, comprising: a computer-readable storage medium as described in the second aspect embodiment.

[0012] Thus, in this embodiment of the invention, based on a feature repository of discrete vectors, continuous features can be transformed into discrete features using a corresponding structure; through the Transformer model, the inherent correlation and regular structure between features can be learned; and stronger multimodal trajectory prediction performance can be achieved while retaining more key information. Attached Figure Description

[0013] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart of the multimodal trajectory prediction method according to an embodiment of the present invention;

[0015] Figure 2 This is a schematic diagram of the multimodal trajectory prediction method according to an embodiment of the present invention;

[0016] Figure 3 This is a schematic diagram of the encoder module according to an embodiment of the present invention;

[0017] Figure 4This is a schematic diagram of the decoder module according to an embodiment of the present invention;

[0018] Figure 5 This is a schematic diagram of the Transformer model according to an embodiment of the present invention;

[0019] Figure 6 This is a schematic diagram of the training encoder and decoder according to an embodiment of the present invention;

[0020] Figure 7 This is a schematic diagram illustrating the principle of training a Transformer model according to an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Trajectory prediction is the process of predicting the future trajectory of a traffic participant based on their past trajectories (observed trajectories). Mathematically, for a traffic participant to be predicted, let their spatial coordinates at time t be... The spatial coordinates of all trajectories within the past time interval Tp are Simultaneously, let the spatial coordinates of the trajectories of the N adjacent traffic participants be... in These are the spatial coordinates of the trajectory of the i-th adjacent traffic participant. The future trajectory of the traffic participant to be predicted is determined using... To represent, where, It is its spatial coordinates at a future time t. The ultimate goal is to train a prediction model g(·) that can predict the future trajectory. It should be as similar as possible to the actual future trajectory Y.

[0023] Based on this, embodiments of the present invention disclose a multimodal trajectory prediction method. For example... Figure 1 As shown, the method of this embodiment of the invention includes the following steps:

[0024] Step S101: Establish a trajectory prediction network.

[0025] Specifically, such as Figure 2 As shown, the trajectory prediction network includes: a past trajectory encoder module, a feature repository module storing trajectory features, a decoder module, and a Transformer model.

[0026] Specifically, such as Figure 3As shown, the past trajectory encoder module includes: a first convolutional layer, a GRU layer, an MLP layer, a summation layer, and a second convolutional layer. The first convolutional layer and the GRU layer are connected sequentially. The outputs of the GRU layer and the MLP layer are both connected to the input of the summation layer, and the output of the summation layer is connected to the input of the second convolutional layer.

[0027] Specifically, the feature repository module includes: VQ (Vector Quantization) layer.

[0028] Each instance (i.e., the trajectory of a traffic participant) can be encoded using a continuous representation. In this embodiment of the invention, this large continuous representation is decomposed and mapped to several smaller discrete vectors. Each vector represents a feature. By storing the features decomposed and mapped from all observed instances, a feature repository is obtained. During prediction, only appropriate features need to be selected from the feature repository based on past trajectories to decode a reasonable and accurate future trajectory. Mathematically, this is achieved using... Let z represent the feature repository, where z i The feature at position i in the feature repository has dimension d and records representative features extracted from the observed scene. n represents the number of features in the repository.

[0029] Within the same size repository, a feature-based repository can store more critical information. Furthermore, by composing features from the repository, more trajectories can be generated, resulting in stronger multimodal performance for trajectory prediction.

[0030] Specifically, such as Figure 4 As shown, the decoder module includes two MLP layers connected in sequence.

[0031] Specifically, such as Figure 5 As shown, the Transformer model is an existing model, specifically comprising an encoder and a decoder connected in sequence. The encoder includes, in sequence, a multi-head attention layer, a residual layer, a normalization layer, a feedforward network layer, a residual layer, and a normalization layer; the decoder includes, in sequence, a multi-head attention layer, a residual layer, a normalization layer, a multi-head attention layer, a residual layer, a normalization layer, a feedforward network layer, a residual layer, a normalization layer, and a fully connected layer.

[0032] Step S102: Train the trajectory prediction network using sample trajectories.

[0033] The sample trajectory includes the continuous frame location coordinates of multiple traffic participants. These traffic participants may or may not influence each other; when two traffic participants influence each other, they are each other's neighboring traffic participants.

[0034] Specifically, this step includes the following process:

[0035] 1. Establish a future trajectory encoder module.

[0036] The structure of the future trajectory encoder is the same as that of the past trajectory encoder, so it will not be described again here.

[0037] 2. The position coordinates of the first preset number of frames in the sample trajectory of each traffic participant are taken as the past trajectory of the sample, and the position coordinates of the remaining frames are taken as the future trajectory of the sample.

[0038] The preset number can be set according to the actual prediction task requirements. For example, if a total of 20 frames of location coordinates are collected, the first 8 frames can be used as the past trajectory of the sample, and the last 12 frames can be used as the future trajectory of the sample.

[0039] 3. Based on the future trajectory encoder module, the past trajectory and future trajectory of traffic participants are used to train the past trajectory encoder module, feature storage module, decoder module and future trajectory encoder module.

[0040] Specifically, such as Figure 6 As shown, the process is as follows:

[0041] (1) Input the sample past trajectory of each traffic participant into the past trajectory encoder module, so that the past trajectory encoder module outputs the continuous representation of the sample past trajectory of each traffic participant. At the same time, input the sample future trajectory of each traffic participant into the future trajectory encoder module, so that the future trajectory encoder module outputs the continuous representation of the sample future trajectory of each traffic participant.

[0042] Specifically, based on the aforementioned encoder module structure, the processing procedure is as follows:

[0043] The trajectories corresponding to different encoder modules are input into the first convolutional layer and the MLP layer respectively. The first convolutional layer performs convolution processing on the trajectory to obtain the first feature vector. The first feature vector is input into the GRU layer. The GRU layer processes the first feature vector to obtain the second feature vector. The MLP layer processes the trajectory to obtain the third feature vector. The summation layer sums the second and third feature vectors to obtain the fourth feature vector. The fourth feature vector is input into the second convolutional layer. The second convolutional layer performs convolution processing on the fourth feature vector to obtain a continuous representation of the corresponding trajectory.

[0044] The past trajectory encoder and the future trajectory encoder extract continuous representations e of the past trajectory of the sample with interactive influence from the sample past trajectory and the sample future trajectory, respectively. p and continuous representation of the future trajectory of the sample e f .

[0045] (2) Input the continuous representation of the past trajectory and the continuous representation of the future trajectory of each traffic participant into the feature repository module, so that the continuous representation of the past trajectory and the continuous representation of the future trajectory of each traffic participant are matched in the feature repository module respectively, and output the discrete representation combination and the index of the discrete representation combination of the past trajectory and the index of the discrete representation combination of the future trajectory of each traffic participant.

[0046] The matching method is the nearest neighbor algorithm. Specifically, using a shared feature space Z, the nearest neighbor algorithm is used to find and compute discrete representations z of the discrete past trajectories and future trajectories of the samples. p and z f e p and e f The formula for mapping discretization to the nearest element in the feature repository is as follows:

[0047]

[0048] Where e represents the continuous representation extracted by the encoder, and e p e f ∈e, z represents a vector in the feature repository module, and z p , z f ∈z.

[0049] In the feature repository, the continuous representations obtained by the encoder module are mapped to indices in the feature repository module to represent the trajectories of traffic participants. Specifically, the index combination of trajectory x is... This is certain and equivalent to the sequence s∈{0,…,|Z|-1} b×w Index encoding is obtained by replacing each consecutive representation with its index in the feature repository module Z, i.e.:

[0050] s i =k, and (z q ) i =z k .

[0051] By matching continuous representations using the nearest neighbor algorithm described above, the corresponding discrete representation combinations and their indices are determined.

[0052] (3) Input the discrete representation combination of the past trajectory of each traffic participant and the discrete representation combination of the future trajectory of each traffic participant into the decoder module, and output the reconstructed trajectory of each traffic participant.

[0053] The indexes in sequence s are encoded and mapped back to the corresponding continuous representations. It can then be decoded and reconstructed into a trajectory by the decoder.

[0054] The reconstructed trajectory includes: the reconstructed past trajectory of the corresponding sample and the reconstructed future trajectory of the corresponding sample.

[0055] In mathematics, let E past (·) and E future (·) As the past trajectory encoder module and the future trajectory encoder module, D joint (·) represents the decoder module, which, given the past trajectory X of a traffic participant, calculates the past trajectories X of its neighboring traffic participants. N And its future trajectory Y, and the future trajectories Y of its neighboring traffic participants. N The joint reconstruction process can be denoted as:

[0056] e p =E past (X, X) N ), e f =E future (Y, Y) N ), z p , z f =q(e p e f ),

[0057] Through the above process, past and future features are connected and input into the decoder to obtain the jointly reconstructed past and future trajectories.

[0058] (4) Based on the reconstructed trajectory and sample trajectory of each traffic participant, calculate the mean of the first loss function of each traffic participant, and backpropagate the mean to optimize the past trajectory encoder module, decoder module and future encoder module, and store the feature vectors in the sample trajectory in the feature storage module to complete the training.

[0059] Since the operation in equation (1) is nondifferentiable, backpropagation is achieved through a pass-through gradient estimator, that is, the gradient is directly copied from the decoder module to the encoder module. In this way, the model and feature storage module can be trained end-to-end through the loss function.

[0060] Specifically, the first loss function for each traffic participant includes:

[0061]

[0062] Where L1 is the first loss function for each traffic participant. and Let X and Y be the reconstructed past and future trajectories of the traffic participant output by the encoder module, respectively, and let sg[·] be the stopping gradient operation. For direct gradient propagation, This is the commitment loss for the weighting factor β.

[0063] Once the above training is completed, the past trajectory encoder and decoder can be fixed, and all features extracted from the training data can be mapped to the feature repository. The encoder, decoder, and feature repository will not participate in gradient propagation in subsequent training.

[0064] 4. Based on the trained past trajectory encoder module, feature repository module, decoder module, and future trajectory encoder module, the Transformer model is trained using sample trajectories of traffic participants.

[0065] Specifically, such as Figure 7 As shown, the process is as follows:

[0066] (1) Input the sample past trajectory of each traffic participant into the past trajectory encoder module, so that the past trajectory encoder module outputs the continuous representation of the sample past trajectory of each traffic participant. At the same time, input the sample future trajectory of each traffic participant into the future trajectory encoder module, so that the future trajectory encoder module outputs the continuous representation of the sample future trajectory of each traffic participant.

[0067] (2) Input the continuous representation of the past trajectory and the continuous representation of the future trajectory of each traffic participant into the feature repository module, so that the continuous representation of the past trajectory and the continuous representation of the future trajectory of each traffic participant are matched in the feature repository module respectively, and output the discrete representation combination and the index of the discrete representation combination of the past trajectory and the index of the discrete representation combination of the future trajectory of each traffic participant.

[0068] (3) First, concatenate the index of the discrete representation combination of the past trajectory of each traffic participant and the index of the discrete representation combination of the future trajectory of each traffic participant, then perform random masking, and input them into the Transformer model so that the Transformer model outputs the probability distribution of the index of the discrete representation combination of the past trajectory of each traffic participant and the probability distribution of the index of the discrete representation combination of the future trajectory of each traffic participant.

[0069] After obtaining the index encoding sort of the past trajectories of samples in sequence s, the trajectory prediction task can be represented as the prediction of the feature index of the autoregressive future trajectory. Therefore, a Transformer model was designed to learn the inherent correlation and regularity between the index encodings of the constituent sequences s. From the inherent spatial-temporal relationship between the original trajectories, it can be inferred that the features extracted from the original trajectories also have a certain spatial-temporal relationship. To train the Transformer model, a portion of the vectors are randomly replaced with randomly generated vectors of the same dimension. This process can be understood as adding strong noise to the features. Then, during the denoising process of the Transformer model, the spatial-temporal characteristics between the feature vectors can be learned. The training process is as follows: Figure 7 As shown, it is denoted as:

[0070]

[0071] (4) Based on the probability distribution of the index of the discrete representation combination of the past trajectory of each traffic participant and the probability distribution of the index of the discrete representation combination of the future trajectory of the sample, as well as the discrete representation combination of the past trajectory of the sample and the discrete representation combination of the future trajectory of the sample, calculate the value of the second loss function until the value of the second loss function is less than the preset threshold, and complete the training.

[0072] Specifically, the second loss function includes:

[0073] L2=CrossEntropy(Y′,X′) (3)

[0074] Where L2 is the value of the second loss function for each traffic participant, Y′ is the probability distribution of the index of the discrete representation combination of the sample past trajectory and the index of the discrete representation combination of the sample future trajectory for each traffic participant, and X′ is the discrete representation combination of the sample past trajectory and the discrete representation combination of the sample future trajectory for each traffic participant.

[0075] The above process completes the training of the Transformer model.

[0076] Through the above training process, the inherent correlations and regularities among the features that make up the trajectory are learned in order to predict the feature composition of current traffic participants.

[0077] Step S103: Input the past trajectories of multiple traffic participants to be predicted into the trained trajectory prediction network, and output the predicted future trajectories of multiple traffic participants to be predicted.

[0078] Specifically, such as Figure 2 As shown, this step includes the following process:

[0079] 1. Input the past trajectories of multiple traffic participants to be predicted into the past trajectory encoder module, so that the past trajectory encoder module outputs a continuous representation of the past trajectories of multiple traffic participants to be predicted.

[0080] 2. Input the continuous representations of the past trajectories of multiple traffic participants to be predicted into the feature repository module, and match the continuous representations of the past trajectories of multiple traffic participants to be predicted in the feature repository module, and output the discrete representation combination of the past trajectories of multiple traffic participants to be predicted and the index of the discrete representation combination.

[0081] 3. Input the index of the discrete representation combination of the past trajectories of multiple traffic participants to be predicted into the Transformer model, so that the Transformer model outputs the set of indices with the highest probability from the probability distribution of the discrete representation combination indexes corresponding to the predicted future trajectories of multiple traffic participants.

[0082] 4. Input the set of indices with the highest probability from the probability distribution of the discrete representation combination indices corresponding to the predicted future trajectories of multiple traffic participants into the feature storage module. Then, in the feature storage module, match the set of indices with the highest probability from the probability distribution of the discrete representation combination indices corresponding to the predicted future trajectories of multiple traffic participants, and output the discrete representation combination of the predicted future trajectories of multiple traffic participants.

[0083] 5. Combine the discrete representations of the predicted future trajectories of multiple traffic participants to be predicted and input them into the decoder module to output the predicted future trajectories of multiple traffic participants to be predicted.

[0084] It should be understood that the prediction of future trajectories consists of location coordinates.

[0085] Furthermore, embodiments of the present invention also provide a computer-readable storage medium storing computer program instructions; when the computer program instructions are executed by a processor, they implement the multimodal trajectory prediction method as described in the above embodiments.

[0086] Furthermore, embodiments of the present invention also provide a multimodal trajectory prediction system, comprising: a computer-readable storage medium as described in the above embodiments.

[0087] In summary, the embodiments of the present invention, based on a feature repository of discrete vectors, can transform continuous features into discrete features using a corresponding structure; through the Transformer model, the inherent correlation and regular structure between features can be learned; and stronger multimodal trajectory prediction performance can be achieved while retaining more key information.

[0088] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multimodal trajectory prediction method, characterized in that, include: A trajectory prediction network is established, wherein the trajectory prediction network includes: a past trajectory encoder module, a feature repository module storing trajectory features, a decoder module, and a Transformer model; The trajectory prediction network is trained using sample trajectories, wherein the sample trajectories include the position coordinates of multiple traffic participants in consecutive frames; The past trajectories of multiple traffic participants to be predicted are input into the trained trajectory prediction network, which outputs the predicted future trajectories of multiple traffic participants to be predicted. The step of training the trajectory prediction network using sample trajectories includes: Establish a future trajectory encoder module; The position coordinates of the first preset number of frames in the sample trajectory of each traffic participant are taken as the past trajectory of the sample, and the position coordinates of the remaining frames are taken as the future trajectory of the sample. Based on the future trajectory encoder module, the past trajectory encoder module, the feature repository module, the decoder module, and the future trajectory encoder module are trained using sample past trajectories and sample future trajectories of traffic participants; Based on the trained past trajectory encoder module, feature repository module, decoder module, and future trajectory encoder module, the Transformer model is trained using sample trajectories of traffic participants; The steps of training the past trajectory encoder module, the feature repository module, and the decoder module include: The sample past trajectory of each traffic participant is input into the past trajectory encoder module, so that the past trajectory encoder module outputs a continuous representation of the sample past trajectory of each traffic participant. At the same time, the sample future trajectory of each traffic participant is input into the future trajectory encoder module, so that the future trajectory encoder module outputs a continuous representation of the sample future trajectory of each traffic participant. The continuous representation of the past trajectory and the continuous representation of the future trajectory of each traffic participant are input into the feature repository module, so that the continuous representation of the past trajectory and the continuous representation of the future trajectory of each traffic participant are matched in the feature repository module respectively, and the discrete representation combination and the index of the discrete representation combination of the past trajectory and the index of the discrete representation combination of the future trajectory are output. The discrete representation combination of the sample past trajectory and the discrete representation combination of the sample future trajectory of each traffic participant are input into the decoder module, and the reconstructed trajectory of each traffic participant is output. The reconstructed trajectory includes: the reconstructed past trajectory corresponding to the sample past trajectory and the reconstructed future trajectory corresponding to the sample future trajectory. Based on the reconstructed trajectory and sample trajectory of each traffic participant, the mean of the first loss function of each traffic participant is calculated, and the mean is backpropagated through backpropagation to optimize the past trajectory encoder module, the decoder module and the future trajectory encoder module. The feature vectors in the sample trajectory are stored in the feature repository module to complete the training.

2. The multimodal trajectory prediction method according to claim 1, characterized in that, in, The first loss function for each traffic participant includes: ； in, The value of the first loss function for each traffic participant. and These are the reconstructed past trajectories and reconstructed future trajectories of traffic participants, respectively, output by the decoder module. X and Y These are the past and future trajectories of traffic participants, respectively. To stop the gradient operation, For direct gradient propagation, Weighting factors β The loss of commitment.

3. The multimodal trajectory prediction method according to claim 1, characterized in that, The steps for training the Transformer model include: The sample past trajectory of each traffic participant is input into the past trajectory encoder module, so that the past trajectory encoder module outputs a continuous representation of the sample past trajectory of each traffic participant. At the same time, the sample future trajectory of each traffic participant is input into the future trajectory encoder module, so that the future trajectory encoder module outputs a continuous representation of the sample future trajectory of each traffic participant. The continuous representation of the past trajectory and the continuous representation of the future trajectory of each traffic participant are input into the feature repository module, so that the continuous representation of the past trajectory and the continuous representation of the future trajectory of each traffic participant are matched in the feature repository module respectively, and the discrete representation combination and the index of the discrete representation combination of the past trajectory and the index of the discrete representation combination of the future trajectory are output. The index of the discrete representation combination of the sample past trajectory and the index of the discrete representation combination of the sample future trajectory of each traffic participant are first concatenated and then randomly masked, and then input into the Transformer model, so that the Transformer model outputs the probability distribution of the index of the discrete representation combination of the sample past trajectory and the probability distribution of the index of the discrete representation combination of the sample future trajectory of each traffic participant. Based on the probability distribution of the index of the discrete representation combination of the sample past trajectory and the probability distribution of the index of the discrete representation combination of the sample future trajectory for each traffic participant, as well as the discrete representation combination of the sample past trajectory and the discrete representation combination of the sample future trajectory, the value of the second loss function is calculated until the value of the second loss function is less than a preset threshold, and the training is completed. The second loss function includes: ； in, The value of the second loss function for each traffic participant. The probability distribution of the index of the discrete representation combination of the sample's past trajectory for each traffic participant and the probability distribution of the index of the discrete representation combination of the sample's future trajectory. For each traffic participant, there is a discrete combination of past trajectory representations and a discrete combination of future trajectory representations.

4. The multimodal trajectory prediction method according to claim 1, characterized in that, The step of outputting the predicted future trajectories of multiple traffic participants to be predicted includes: The past trajectories of multiple traffic participants to be predicted are input into the past trajectory encoder module, so that the past trajectory encoder module outputs a continuous representation of the past trajectories of multiple traffic participants to be predicted. The continuous representations of the past trajectories of multiple traffic participants to be predicted are input into the feature repository module, so that the continuous representations of the past trajectories of multiple traffic participants to be predicted are matched in the feature repository module, and the discrete representation combination of the past trajectories of multiple traffic participants to be predicted and the index of the discrete representation combination are output. The index of the discrete representation combination of the past trajectories of multiple traffic participants to be predicted is input into the Transformer model, so that the Transformer model outputs the set of indices with the highest probability in the probability distribution of the discrete representation combination indexes corresponding to the predicted future trajectories of multiple traffic participants to be predicted. The set of indices with the highest probability in the probability distribution of the discrete representation combination indexes corresponding to the predicted future trajectories of multiple traffic participants to be predicted is input into the feature repository module, so that the set of indices with the highest probability in the probability distribution of the discrete representation combination indexes corresponding to the predicted future trajectories of multiple traffic participants to be predicted is matched in the feature repository module, and the discrete representation combination of the predicted future trajectories of multiple traffic participants to be predicted is output. The discrete representations of the predicted future trajectories of multiple traffic participants to be predicted are combined and input into the decoder module, which outputs the predicted future trajectories of the multiple traffic participants to be predicted.

5. The multimodal trajectory prediction method according to claim 1, characterized in that, Both the future trajectory encoder module and the past trajectory encoder module include: a first convolutional layer, a GRU layer, an MLP layer, an summation layer, and a second convolutional layer; Trajectories corresponding to different encoder modules are input into the first convolutional layer and the MLP layer, respectively. The first convolutional layer performs convolution processing on the trajectory to obtain a first feature vector. The first feature vector is input into the GRU layer, which processes the first feature vector to obtain a second feature vector. The MLP layer processes the trajectory to obtain a third feature vector. The summation layer sums the second feature vector and the third feature vector to obtain a fourth feature vector. The fourth feature vector is input into the second convolutional layer, which performs convolution processing on the fourth feature vector to obtain a continuous representation of the corresponding trajectory.

6. The multimodal trajectory prediction method according to claim 1, characterized in that, The decoder module includes two MLP layers connected in sequence.

7. The multimodal trajectory prediction method according to claim 1, characterized in that, The feature repository module includes: VQ layer.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, they implement the multimodal trajectory prediction method as described in any one of claims 1 to 7.

9. A multimodal trajectory prediction system, characterized in that, include: The computer-readable storage medium as described in claim 8.

Images