Transform-based multi-agent trajectory prediction device and method

Abstract

The invention belongs to the technical field of automatic driving, and particularly relates to a Transform-based multi-agent trajectory prediction device and method, and the device comprises a trajectory sequence representation unit, a time encoder unit, a Transform encoder-decoder unit, a CVAE prior and posterior unit, and a future trajectory decoder unit. According to the method, a joint modeling framework is adopted, and dependence of a time dimension and interaction of a social dimension are captured at the same time.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of automatic driving, and particularly relates to a multi-agent trajectory prediction device and method based on a Transformer. BACKGROUND

[0002] Multi-agent trajectory prediction is one of the core technologies in the fields of automatic driving and intelligent monitoring, and its core goal is to accurately predict the future motion trajectory of multiple pedestrians, vehicles and other agents based on historical trajectory data.

[0003] Existing multi-agent trajectory prediction methods are based on recurrent neural networks (RNN), represented by Social LSTM. Such methods model the trajectory of each agent in the time dimension through an LST long short-term memory network to capture its historical motion rules. Meanwhile, a "social pooling layer" is introduced to aggregate the state information of surrounding agents to achieve simple social interaction modeling. The core modules include a time sequence feature extraction module, a social interaction module and a trajectory prediction module. The time sequence feature extraction module encodes the historical position and speed sequence of a single agent using LSTM to output a time sequence feature vector. The social interaction module aggregates the time sequence features of other agents within a certain range around the target agent through pooling operations (such as max pooling and average pooling) to generate social features. The trajectory prediction module fuses the time sequence features and social features and outputs the future trajectory through a decoder.

[0004] Existing multi-agent trajectory prediction methods can also be based on traditional Transformer methods. Such methods apply Transformer to trajectory prediction and use self-attention mechanisms to capture long-time dependencies, but are not optimized for multi-agent scenarios. The core modules include a sequence conversion module, a self-attention module and an independent prediction module. The sequence conversion module converts the historical trajectory of each agent into an independent sequence and marks the time information through position encoding. The self-attention module uses a standard self-attention mechanism to calculate the correlation between elements in the sequence to capture the dependency relationship in the time dimension. The independent prediction module independently decodes the features of each agent to output the future trajectory without considering the interaction between agents.

[0005] The existing multi-agent trajectory prediction method can also be based on a graph neural network (GNN, for example, GraphAttention Network), represented by Social-BiGAT. This kind of method first extracts time sequence features through an RNN, and then uses a GNN to model the social interaction between agents. The core modules include a time sequence encoding module, a social graph construction module, a graph convolution module, and a joint prediction module. The time sequence encoding module processes the historical trajectory of each agent using an RNN to obtain time sequence features. The social graph construction module constructs a social graph with agents as nodes and distance or interaction strength as edges. The graph convolution module propagates node features through a GNN to capture social interaction. The joint prediction module fuses time sequence features and social features to predict future trajectories.

[0006] The existing multi-agent trajectory prediction method can also be based on a generative model, for example, the Trajectron++ method. This kind of method uses a conditional variational autoencoder (CVAE) or a generative adversarial network (GAN) to model the multi-modal distribution of trajectories, trying to capture the uncertainty of future trajectories. The core modules include a feature encoding module, a latent distribution modeling module, and a multi-modal sampling module. The feature encoding module fuses historical trajectories and scene information (such as semantic maps) to generate latent features. The latent distribution modeling module outputs the distribution parameters of the latent variables through the encoder of the CVAE, and the decoder generates future trajectories based on the latent variables. The multi-modal sampling module generates diversified trajectory candidates by sampling the latent variables.

[0007] The existing method models the time dimension and the social dimension separately, resulting in serious information loss. Specifically, the existing method first models the time sequence features and then introduces social interaction, or first calculates the social features and then processes the time sequence dependency. This separate modeling causes the state of an agent at a certain time to directly affect the future state of other agents, and the long-time dependency and cross-agent interaction are not fully captured. For example, when vehicle A suddenly slows down, the braking decision of vehicle B depends on the real-time state of vehicle A, but the separate method needs to first encode the time sequence features of vehicle A and then pass them to vehicle B through a social module, resulting in information delay or loss. SUMMARY

[0008] Therefore, the purpose of the present application is to provide a Transformer-based multi-agent trajectory prediction device and method to fundamentally solve the problem of separate processing of time and social dimensions in the existing method, resulting in insufficient capture of cross-agent and cross-time dependency and large long-time sequence prediction error.

[0009] The embodiment of the present application is implemented in the following manner. A Transformer-based multi-agent trajectory prediction device is provided, which comprises: a trajectory sequence representation unit, configured to convert the received historical trajectories of a plurality of agents into a flattened historical trajectory sequence; a time encoder unit configured to generate timestamp features and add the timestamp features to the historical trajectory sequence; a Transformer encoder-decoder unit configured to encode the historical trajectory sequence to which the timestamp features are added to jointly model time dimension and social dimension, and output an encoded historical feature sequence, and generate a future feature sequence based on the encoded historical feature sequence and a known future trajectory sequence; a CVAE prior and posterior unit configured to model a prior distribution of all agent latent intents according to the historical feature sequence through a prior network, and model an approximate posterior distribution of all agent latent intents according to the historical feature sequence and the future feature sequence through a posterior network; a future trajectory decoder unit configured to generate a future trajectory of each agent based on the historical feature sequence and the latent intent sampled by the CVAE prior and posterior unit.

[0010] In some embodiments, the trajectory sequence representation unit comprises: a historical trajectory sequence representation module configured to assume that there are N agents, and a historical trajectory contains H+1 time points, and a state of each agent n at time point t is convert the historical trajectory into a flattened historical trajectory sequence through the following formula: , and the length is: .

[0011] In some embodiments, the trajectory sequence representation unit further comprises: a dynamic scenario adaptation module configured to remove features corresponding to missing agents from the historical trajectory sequence or add features corresponding to increased agents when the number of agents changes over time.

[0012] In some embodiments, the Transformer encoder-decoder unit comprises: an encoder module configured to output an encoded historical feature sequence after inputting the historical trajectory sequence to which the timestamp features are added; an attention mechanism module configured to generate two groups of key-value pairs of the plurality of agents through a double projection matrix using an attention mechanism, the two groups of key-value pairs respectively corresponding to queries, keys and values of the same agent at different time points, and queries, keys and values of different agents at the same time point; a decoder module configured to generate a future feature sequence based on the encoded historical feature sequence and a known future trajectory sequence.

[0013] In some embodiments, the encoder module comprises: a sequence receiving submodule configured to receive the historical trajectory sequence to which the timestamp feature is added; an encoding submodule configured to encode using a 2-layer Transformer encoder, wherein each layer uses 8-headed agent-aware attention and a feed-forward network; a sequence output submodule configured to output the encoded historical feature sequence, wherein the historical feature sequence is represented as , which contains a time dimension and a social dimension.

[0014] In some embodiments, the encoder module and the decoder module use the same design structure.

[0015] In some embodiments, the Transformer encoder-decoder unit further comprises: a mask module configured to dynamically generate a mask matrix according to the agent identity to which the query-key belongs, to selectively activate the projection matrix.

[0016] In some embodiments, the mask module comprises: a mask submodule configured to control the activation of the projection matrix by , where L represents the sequence length; when the query and the key belong to the same agent, , the projection matrix matching the same agent at different time points is activated; otherwise , the projection matrix matching different agents at the same time point is activated.

[0017] In some embodiments, the multi-agent trajectory prediction device further comprises: a semantic map encoding unit configured to process semantic map information to generate semantic map features, and the future trajectory decoder unit generates the future trajectory of each agent based on the historical feature sequence, the latent intention sampled by the CVAE prior and posterior unit, and the semantic map features; a trajectory sampler unit configured to generate a plurality of diversified latent intention sampling values by optimizing a strategy, so as to drive the future trajectory decoder unit to generate a plurality of diversified future trajectory prediction samples.

[0018] The embodiment of the present application also provides a multi-agent trajectory prediction method based on a Transformer, comprising the following steps: converting the received historical trajectories of a plurality of agents into a flattened historical trajectory sequence; generating a timestamp feature and adding the timestamp feature to the historical trajectory sequence; encode the history trajectory sequence to which the timestamp feature is added to jointly model the time dimension and the social dimension, and output an encoded history feature sequence, and generate a future feature sequence based on the encoded history feature sequence and a known future trajectory sequence; model a prior distribution of potential intentions of all agents according to the history feature sequence through a prior network, and model an approximate posterior distribution of potential intentions of all agents according to the history feature sequence and the future feature sequence through a posterior network; generate a future trajectory of each agent based on the history feature sequence and the potential intention sampled.

[0019] The embodiment of the present application provides a multi-agent trajectory prediction device and method based on a Transformer. A trajectory sequence conversion unit converts a plurality of received historical trajectories of agents into a flattened history trajectory sequence. A time encoder unit generates a timestamp feature and adds the timestamp feature to the history trajectory sequence. A Transformer encoder-decoder unit encodes the history trajectory sequence to which the timestamp feature is added to jointly model the time dimension and the social dimension, and outputs an encoded history feature sequence. A CVAE prior and posterior unit models a prior distribution of potential intentions of all agents according to the history feature sequence through a prior network, and models an approximate posterior distribution of potential intentions of all agents according to the history feature sequence and the future feature sequence through a posterior network. A future trajectory decoder unit generates a future trajectory of each agent based on the history feature sequence and the potential intention sampled by the CVAE prior and posterior unit. The present application adopts a joint modeling framework to simultaneously capture the dependence of the time dimension and the interaction of the social dimension. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 is a structural block diagram of a multi-agent trajectory prediction device provided by the embodiment of the present application; Figure 2 is a flow block diagram of a multi-agent trajectory prediction method provided by the embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to facilitate the understanding of the present application, the present application will be described more fully below with reference to the related drawings. The drawings show several embodiments of the present application. However, the present application can be realized in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0022] It is to be understood that where an element such as a layer, region or substrate is described as being "on" another element, it can be directly on the other element or intervening elements can also be present. Where an element such as a layer, region or substrate is described as being "connected" to another element, it can be directly connected to the other element or intervening elements can also be present. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0024] With reference to Figure 1 An embodiment of the present application provides a multi-agent trajectory prediction device, comprising: a trajectory sequence representation unit, configured to convert a plurality of historical trajectories of a plurality of agents into a flattened historical trajectory sequence; a time encoder unit, configured to generate timestamp features and add the timestamp features to the historical trajectory sequence; a Transformer encoder-decoder unit, configured to encode the historical trajectory sequence added with the timestamp features to jointly model a time dimension and a social dimension, and output an encoded historical feature sequence, and generate a future feature sequence based on the encoded historical feature sequence and a known future trajectory sequence; a CVAE prior and posterior unit, configured to model a prior distribution of all agent potential intentions according to the historical feature sequence through a prior network, and model an approximate posterior distribution of all agent potential intentions according to the historical feature sequence and the future feature sequence through a posterior network; a semantic map encoding unit, configured to process semantic map information to generate semantic map features; a future trajectory decoder unit, configured to generate a future trajectory of each agent based on the historical feature sequence, the potential intention sampled by the CVAE prior and posterior unit and the semantic map features; a trajectory sampler unit, configured to generate a plurality of diversified potential intention sampling values through an optimization strategy, so as to drive the future trajectory decoder unit to generate a plurality of diversified future trajectory prediction samples.

[0025] In the present application, through the Transformer and CVAE framework, the joint modeling of time dimension and social dimension, agent identity perception, latent intention modeling and diversified trajectory generation are realized. Specifically, an external device such as a computer device transmits the historical trajectories of multiple agents to a trajectory sequence representation unit; the trajectory sequence representation unit converts the received historical trajectories of multiple agents into a flattened historical trajectory sequence to adapt to the input format of the subsequent Transformer encoder-decoder unit, and transmits the historical trajectory sequence to a time encoder unit; the time encoder unit generates timestamp features and adds the timestamp features to the received historical trajectory sequence, retains the time dimension information, and transmits the historical trajectory sequence added with the timestamp features to the Transformer encoder-decoder unit; the Transformer encoder-decoder unit encodes the received historical trajectory sequence added with the timestamp features to jointly model the time dimension and the social dimension, and outputs the encoded historical feature sequence, and generates a future feature sequence based on the encoded historical feature sequence and the known future trajectory sequence, and transmits the historical feature sequence and the future feature sequence to a CVAE prior and posterior unit, and transmits the historical feature sequence to a future trajectory decoder unit; the CVAE prior and posterior unit models the prior distribution of all agent latent intentions according to the historical feature sequence through a prior network, and models the approximate posterior distribution of all agent latent intentions according to the historical feature sequence and the future feature sequence through a posterior network, models the distribution of latent intentions, realizes the generation of multi-modal trajectories, and transmits the sampled latent intentions to the future trajectory decoder unit; a semantic map encoding unit processes semantic map information to generate semantic map features, fuses scene information (such as road, sidewalk, etc. Scene information), enhances the scene adaptability of prediction, and transmits the semantic map features to the future trajectory decoder unit; the future trajectory decoder unit generates the future trajectory of each agent based on the received historical feature sequence, the latent intention sampled by the CVAE prior and posterior unit and the semantic map features, generates the future trajectory in a self-recurrent manner to ensure the temporal consistency; the trajectory sampler unit generates multiple sets of diversified latent intention sampling values through an optimization strategy, thereby driving the future trajectory decoder unit to generate multiple diversified future trajectory prediction samples, optimizing the distribution of latent intentions and improving the coverage. Finally, the multi-agent trajectory prediction device outputs the optimal future trajectory prediction sample or outputs all generated future trajectory prediction samples for use by the downstream decision system.

[0026] In the present application, the following advantages are achieved: ①, a joint modeling framework is adopted to simultaneously capture the dependence of the time dimension and the interaction of the social dimension; ②, an agent identity perception mechanism is designed to retain agent identity information and accurately distinguish between time series dependence and social interaction; ③, a latent intention model is constructed to enable the mutual influence of the latent intentions of agents, thereby improving the social rationality of the trajectory; ④, a diversified sampling strategy is proposed to generate trajectory samples with higher coverage and enhance the adaptability of dynamic scenarios. The multi-agent trajectory prediction device provided by the present application can provide more accurate, reasonable and diverse trajectory prediction results for intelligent mobile devices in the fields of autonomous driving, intelligent monitoring and the like, thereby improving the safety decision-making capability of intelligent mobile devices.

[0027] In the prior art, the time dimension (e.g., the historical trajectory of a single agent) and the social dimension (e.g., the interaction between multiple agents) are often processed in stages, for example, the historical trajectory of each agent is independently encoded by an LSTM or the like time series model, and then the interaction information is aggregated by a social pooling layer. This kind of "time series encoding → social aggregation" mode has the following two defects: ①, information fragmentation: the time series dependence (e.g., the movement trend of the agent itself) and the social interaction (e.g., the avoidance of adjacent vehicles) are split into independent processing steps, resulting in the loss of key associated information across time and across agents. For example, the deceleration behavior of vehicle A at t=2 cannot be directly associated with the avoidance action of vehicle B at t=5. ②, poor dynamic adaptability: when the number of agents changes over time (e.g., pedestrians enter or leave the scene), the traditional method needs to fill in invalid data or reinitialize the model, resulting in prediction bias.

[0028] To this end, the present application provides a trajectory sequence representation unit, which converts the historical trajectory of multiple agents into a flattened historical trajectory sequence suitable for subsequent processing by a Transformer encoder-decoder unit, while retaining the association information of agents and time, thereby achieving deep integration of the time dimension and the social dimension, by means of a flattened sequence joint representation method.

[0029] In some embodiments of the present application, the trajectory sequence representation unit comprises: a historical trajectory sequence representation module, configured to assume that there are N agents, the historical trajectory contains H+1 time points (t=-H,..., 0), and the state of each agent n at time t is which includes position, velocity, etc., and the historical trajectory is converted into a flattened historical trajectory sequence by the following formula: , the length of which is: , and H is a positive integer greater than 0.

[0030] In the above formula, can be represented as: , Specifically, the historical trajectory of the agent 1 at the time -H is represented as Specifically, the historical trajectory of the agent 2 at the time -H is represented as Specifically, the historical trajectory of the agent N at the time -H is represented as Specifically, the historical trajectory of the agent 1 at the time -H+1 is represented as Specifically, the historical trajectory of the agent 2 at the time -H+1 is represented as Specifically, the historical trajectory of the agent N at the time 0 is represented as

[0031] In some embodiments of the present application, the trajectory sequence representation unit further comprises: a future trajectory sequence representation module, configured to assume that a future trajectory contains T time points (t=1,...,T), and the position of each agent n at the time point t is The future trajectory sequence is specifically represented by the following formula: The length is: T is a positive integer greater than 0.

[0032] In the above formula, can be represented as: , Specifically, the future trajectory of the agent 1 at the time 1 is represented as Specifically, the future trajectory of the agent 2 at the time 1 is represented as Specifically, the future trajectory of the agent N at the time 1 is represented as Specifically, the future trajectory of the agent 1 at the time 2 is represented as Specifically, the future trajectory of the agent 2 at the time 2 is represented as Specifically, the future trajectory of the agent 1 at the time T is represented as Specifically, the future trajectory of the agent N at the time T is represented as

[0033] In some embodiments of the present application, the trajectory sequence representation unit further comprises: a dynamic scene adaptation module, configured to remove the features corresponding to the missing agents from the historical trajectory sequence or add the features corresponding to the increased agents when the number of agents changes over time.

[0034] In the present embodiment, the dynamic scene adaptation module removes the features corresponding to the missing agents from the historical trajectory sequence when the number of agents changes over time, i.e., removes the features corresponding to the agents leaving the scene, or adds the features corresponding to the increased agents, i.e., adds the features corresponding to the agents entering the scene.

[0035] For example, a pedestrian leaves the scene at t = 3, the dynamic scene adaptation module directly truncates at the corresponding position of the historical trajectory sequence, and the model can still infer the interaction relationship through the remaining agent trajectory.

[0036] The application is based on a multi-agent trajectory joint representation method of flattened sequence, which converts the historical trajectory of the multi-agent into an "agent-time" flattened sequence (for example ), while retaining the association of agent index and timestamp features. This representation method breaks through the limitations of traditional methods such as independent sequence of Social LSTM and graph structure of GNN, enabling the Transformer to simultaneously process time dimension and social dimension dependencies. This sequence structure is the basis of joint modeling, directly determining the completeness of the interaction information between agents, and none of the prior art uses such flattened sequences.

[0037] The application provides a time encoder unit to add timestamp features to the historical trajectory sequence, solving the problem of the Transformer encoder-decoder unit being insensitive to time dimension information, and ensuring that the model can distinguish features at different times.

[0038] In some embodiments of the application, the time encoder unit comprises: a feature generation module for generating timestamp features using sine and cosine functions; a feature fusion module for splicing the historical trajectory sequence and the timestamp features, and then fusing them through linear transformation; a sequence output module for outputting the historical trajectory sequence with the added timestamp features.

[0039] In this embodiment, the feature generation module generates timestamp features using sine and cosine functions, and transmits the timestamp features to the feature fusion module; the feature fusion module splices the historical trajectory sequence and the timestamp features, and then fuses them through linear transformation; finally, the sequence output module outputs the historical trajectory sequence with the added timestamp features.

[0040] In some embodiments of the application, the feature generation module comprises: a feature generation sub-module for generating timestamp features through the following formula: , where the sine and cosine period is 10000, denotes the timestamp feature, denotes the timestamp dimension, which can be set to 256, for example, denotes mapping the historical time to a non-negative integer, for example, t = -H corresponds to 0 and t = 0 corresponds to H.

[0041] In some embodiments of the application, the feature fusion module comprises: a feature fusion sub-module, configured to splice the historical trajectory sequence with the timestamp feature and fuse by linear transformation: wherein is represented as a learnable weight matrix, is represented as splicing.

[0042] wherein the linear transformation matrix has a dimension of , is a historical trajectory sequence feature dimension, for example, 4 dimensions of position + speed.

[0043] The existing attention mechanism of the Transformer treats all elements in the sequence equally, which leads to confusion of two types of key information in the multi-agent scenario. Specifically: ① confusion of time sequence dependence and social interaction: for example, when predicting the future position of agent A, the model may assign the same attention weight to the historical position of A itself (time sequence dependence) and the current position of neighboring agent B (social interaction), resulting in a predicted trajectory that lacks rationality. ② loss of agent identity information: although agent index encoding (such as position embedding) can partially distinguish identity, it introduces artificial dependence (such as the model being sensitive to the order of agents), which destroys the permutation invariance.

[0044] To this end, the application provides a Transformer encoder-decoder unit, which designs a double-path attention mechanism to achieve accurate interaction modeling by explicitly distinguishing agent identity, i.e., simultaneously capturing time dimension dependence (time sequence correlation of the same agent) and social dimension interaction (correlation of different agents), and solving the information loss problem caused by separate modeling.

[0045] In some embodiments of the application, the Transformer encoder-decoder unit comprises: an encoder module, configured to output an encoded historical feature sequence after inputting the historical trajectory sequence with the added timestamp feature; an attention mechanism module, configured to generate two groups of key-value pairs of the plurality of agents by double projection matrices using an attention mechanism, the two groups of key-value pairs corresponding to queries, keys and values of the same agent at different time points and queries, keys and values of different agents at the same time point, respectively; a decoder module, configured to generate a future feature sequence based on the encoded historical feature sequence and a known future trajectory sequence.

[0046] In the embodiment, after the encoder module inputs the historical trajectory sequence added with the timestamp feature, it outputs the encoded historical feature sequence, which contains the time dimension and the social dimension; then, the attention mechanism module generates two sets of key-value pairs of multiple agents by using the attention mechanism through the double projection matrix, and the two sets of key-value pairs correspond to the query, key and value of the same agent at different time points and the query, key and value of different agents at the same time point respectively; the decoder module generates the future feature sequence based on the encoded historical feature sequence and the known future trajectory sequence.

[0047] In some embodiments of the present application, the encoder module comprises: a sequence receiving sub-module, configured to receive the historical trajectory sequence added with the timestamp feature; an encoding sub-module, configured to encode by using a 2-layer Transformer encoder, wherein each layer uses 8-headed agent-aware attention and a feed-forward network, and the hidden dimension can be 512; a sequence outputting sub-module, configured to output the encoded historical feature sequence, wherein the historical feature sequence is represented as , which contains the time dimension and the social dimension.

[0048] In some embodiments of the present application, the encoder module and the decoder module use the same design structure.

[0049] The encoder module encodes by using a 2-layer Transformer encoder, wherein each layer uses 8-headed agent-aware attention and a feed-forward network, and similarly, the decoder module encodes by using a 2-layer Transformer encoder, wherein each layer uses 8-headed agent-aware attention and a feed-forward network.

[0050] In some embodiments of the present application, the Transformer encoder-decoder unit further comprises: a mask module, configured to dynamically generate a mask matrix according to the identity of the agent to which the query-key belongs, so as to selectively activate the projection matrix.

[0051] In the embodiment, by using the mask module, a mask matrix can be dynamically generated according to the identity of the agent to which the query-key belongs, so as to selectively activate one of the two projection matrices in the double projection matrix.

[0052] In some embodiments of the present application, the mask module comprises: a mask sub-module, configured to selectively activate the projection matrix by , wherein L represents the sequence length; when the query and the key belong to the same agent, , the projection matrix matching the same agent at different time points is activated; otherwise activate the projection matrix matching the same moment different agent.

[0053] The attention mechanism module and the mask module provided by the application distinguish "intra-agent attention" and "inter-agent attention" through double projection matrices and masks, that is, time sequence dependence of the same agent and social interaction of different agents, and the calculation formula is: , For the intra-agent branch, a special key-value pair is generated , which only focuses on the historical and future states of the same agent. For example, when predicting the position of agent A at t=5, this branch focuses on the historical position of agent A at t=3.

[0054] For the Inter-Agent branch: generate key-value pairs , which capture the same period interaction between different agents. For example, when predicting the position of A at t=5, this branch focuses on the position of neighboring agent B at t=4.

[0055] The mask matrix controls which branch is activated, which can also be understood as controlling which projection matrix is activated. The mask matrix is dynamically generated according to the agent index. Specifically, when the query (Query) and the key (Key) belong to the same agent, , the intra-agent branch is activated; otherwise , the inter-agent branch is activated. For example, when predicting the future trajectory of agent A, the mask matrix will force the model to assign different attention weights to the historical position of agent A itself (intra-agent) and the current position of the surrounding vehicles (inter-agent), avoiding confusion between the two types of dependencies.

[0056] The application designs an agent-aware attention mechanism, which distinguishes the identity of the agent through double projection matrices (P for intra-agent attention, for inter-agent attention) and mask matrix M (M if and only if the query and the key belong to the same agent). The calculation of the attention weight matrix A considers both types of attention, ensuring that time sequence dependence and social interaction are not confused. Experiments show that removing this mechanism will cause a 21% performance drop, thus fully proving the importance of this mechanism.

[0057] In the prior art, the potential intentions of each agent are usually modeled independently, resulting in generated trajectories lacking social rationality. For example, the "crossing the road" intention of a pedestrian needs to be associated with the "slow down to avoid" intention of a vehicle through an additional post-processing module, which often fails to capture complex dynamic interactions (e.g., multi-vehicle coordinated avoidance).

[0058] To this end, the present application provides a CVAE prior and posterior unit to model the potential intentions of agents (e.g., agent turning, going straight), and to enable mutual influence between potential intentions through joint modeling, thereby improving the social rationality of trajectories.

[0059] In some embodiments of the present application, the CVAE prior and posterior unit comprises: a prior network module configured to perform agent-wise pooling on a historical feature sequence of each agent, and output a mean and a variance of a potential intention of the agent through an MLP to form a joint Gaussian prior distribution; a posterior network module configured to perform agent-wise pooling on a historical feature sequence and a future feature sequence of each agent, and output a mean and a variance of a potential intention of the agent through an MLP to form a joint Gaussian approximate posterior distribution.

[0060] In the present embodiment, the prior network module performs agent-wise pooling on a historical feature sequence of each agent, and outputs a mean and a variance of a potential intention of the agent through an MLP to form a joint Gaussian prior distribution. For example, in a crossroad scenario, if the historical feature sequence of a pedestrian shows "approaching the zebra crossing", the joint Gaussian prior distribution will infer that its potential intention is "crossing the street", and simultaneously adjust the potential intention of a vehicle, for example, to "slow down to avoid".

[0061] Specifically, the historical feature sequence C of each agent is first input, and then agent-wise pooling is performed on the historical feature sequence C of each agent, that is, the mean of the historical feature sequence C of each agent is calculated, and the calculation formula is as follows: Finally, the MLP is mapped to the joint Gaussian prior distribution , and the formula is: where the hidden dimension of the MLP is 512→256, is the potential intention of the agent.

[0062] In the present embodiment, the posterior network module performs agent-wise pooling on a historical feature sequence and a future feature sequence of each agent, and outputs a mean and a variance of a potential intention of the agent through an MLP to form a joint Gaussian approximate posterior distribution.

[0063] ​Specifically, the historical feature sequence C and the future feature sequence of each agent are input first, then agent-wise pooling is performed on the historical feature sequence C and the future feature sequence of each agent, that is, the mean of the future feature sequence of each agent is calculated, and finally the joint Gaussian approximate posterior distribution is mapped to the joint Gaussian approximate posterior distribution through the MLP , and the formula is .

[0064] In the above embodiment, in the joint modeling process, the joint Gaussian approximate posterior distribution of the latent intention is inferred from the future trajectories of all agents, so as to ensure the mutual correlation between the latent intentions. The technical parameters can be: the latent intention dimension , the MLP activation function is ReLU, and the dropout rate is 0.1.

[0065] In some specific embodiments of the present application, the CVAE prior and posterior unit further comprises: A KL divergence calculation module for minimizing the KL divergence between the joint Gaussian prior distribution and the joint Gaussian approximate posterior distribution in the training stage.

[0066] The KL divergence minimization is realized through the formula: , so as to ensure the consistency of the latent intention and the actual trajectory. The technical parameters can be: the KL loss coefficient is 1.

[0067] In combination with the above embodiment, the CVAE prior and posterior unit can significantly promote the transmission of social influence. Specifically, in the prediction process, the future trajectory decoder unit generates the future trajectory based on the latent intention Z obtained by CVAE sampling. For example, if Z1 of the pedestrian represents “crossing the street”, Z2 of the vehicle will be forced to adjust to “slow down and avoid”, so as to generate a trajectory that conforms to the social rules.

[0068] Based on the joint latent intention modeling framework of the CVAE, the CVAE prior and posterior unit performs the prior and posterior processes, and the latent intentions of all agents are jointly modeled. During training, the latent codes are inferred from the future trajectories of all agents; during prediction, the mutually influencing latent intentions are generated through joint sampling, so as to ensure the social consistency of the trajectories.

[0069] The present application provides a semantic map encoding unit, which fuses scene semantic information (such as roads, sidewalks, traffic lights), and improves the scene adaptability of trajectory prediction (for example, vehicles will not predict to the sidewalk).

[0070] In some specific embodiments of the present application, the semantic map encoding unit comprises: a map cropping module, configured to rotate and crop a local map region of a fixed size according to the orientation of each agent at the current position of the agent as a center; a CNN feature extraction module, configured to extract visual features from the local map region by a convolutional neural network (CNN); a feature concatenation module, configured to concatenate the visual features and trajectory features of the agent.

[0071] In the embodiment, the map cropping module rotates and crops a local map region of a fixed size according to the orientation of each agent at the current position of the agent as a center, and transmits the local map region to the CNN feature extraction module; the CNN feature extraction module extracts visual features from the local map region by a convolutional neural network (CNN) after receiving the local map region, and transmits the visual features to the feature concatenation module; the feature concatenation module is configured to concatenate the visual features and trajectory features of the agent, and finally generate semantic map features, wherein the trajectory features of the agent can be historical trajectory or current trajectory of the agent.

[0072] For example, for each agent, the semantic map is rotated according to the orientation, a 100*100 pixel patch centered on the agent is cropped, the resolution is 3m / pixel, and 3 types of semantics including drivable areas, road dividers, etc. are included; then a 4-layer CNN is used to extract 32-dimensional visual features, wherein the channel is 32→32→32→1, the kernel is 5*5→5*5→5*5→3*3, and the stride is 2→2→1→1; finally, the visual features and the trajectory features of the agent are concatenated to generate semantic map features, and the semantic map features are transmitted to the future trajectory decoder unit.

[0073] The fusion mechanism of the semantic map and the trajectory features is designed, the semantic map patch (containing road, sidewalk, etc.) around each agent is cropped, the visual features are extracted by the CNN, and the trajectory features are concatenated and input into the encoder. This mechanism improves the scene adaptability of trajectory prediction. Through the cropping of the map features based on the orientation of the agent, the CNN structure of 4-layer convolution + stride design, and the fusion mode of the trajectory features, the ablation experiment shows that the ADE increases by 5% after removing the map features.

[0074] The application provides a future trajectory decoder unit, which generates a future trajectory based on a historical feature sequence, a latent intention and semantic map features, and ensures time sequence consistency and long-term prediction accuracy.

[0075] In some specific embodiments of the application, the future trajectory decoder unit comprises: an initial sequence module, configured to take the initial positions of the agents as initial inputs; An iterative decoding module is used to fuse the historical feature sequence, the latent intent sampled by the CVAE prior and posterior units, and the semantic map features to generate the trajectory at the next moment. The termination condition module is used to output the complete future trajectory after T iterations.

[0076] In this embodiment, the initial sequence module takes the initial position of each agent as the initial input and transmits the initial position of each agent to the iterative decoding module; the iterative decoding module integrates the historical feature sequence, the potential intent and semantic map features obtained by CVAE prior and posterior unit sampling, and generates the next moment trajectory of each agent based on the initial position of each agent; the termination condition module iterates T times and finally outputs the complete future trajectory of each agent.

[0077] Specifically, starting with the agent's initial position (Right now Starting with the positional features in the sequence, construct an initial sequence. The initial sequence generated is input during the t'th iteration. The trajectory sequence is concatenated with the underlying intent, and represented as follows: This process forms a feature sequence F; then, timestamp features are added to feature sequence F, and semantic map features, such as road masks, are concatenated; finally, the trajectory for the next time step is decoded and generated. After repeating T times, the iteration terminates, generating a complete future trajectory. .

[0078] In some specific embodiments of this application, the future trajectory decoder unit further includes: The causal masking module is used to restrict the iterative decoding module to only call trajectory features before a certain moment when generating the trajectory at a certain moment.

[0079] In this embodiment, the causal masking module employs an upper triangular mask to restrict the iterative decoding module from calling trajectory features prior to a given moment when generating the trajectory of an agent at a specific time, thereby preventing information leakage. Compared to existing technologies, traditional non-autoregressive decoders are prone to temporal inconsistencies (e.g., trajectory jumps); this future trajectory decoder unit utilizes the causal masking module to autoregressively generate time-bound trajectories, achieving a 4% reduction in ADE (average displacement error) for long-term (e.g., 6s) predictions on the nuScenes dataset compared to Trajectron++.

[0080] The application designs an end-to-end autoregressive future trajectory decoder, which takes the sequence of initial positions as the starting point, feeds back to the input after generating the next time trajectory each time, and generates the complete future trajectory for T times. During the process, the historical feature sequence, the potential intention and the semantic map feature are combined, and the causal mask module is used to ensure the time sequence consistency. The decoder overcomes the time sequence breaking problem of the non-autoregressive method, and the iterative generation mechanism and the feature fusion mode can ensure the accuracy of long-time prediction, and the error accumulation of the traditional seq2seq decoder can be reduced by 30%.

[0081] The application provides a trajectory sampler unit, which generates diversified future trajectory prediction samples, covers more possible future modes, and solves the sample homogenization problem of the traditional sampling method.

[0082] In some embodiments of the present application, the trajectory sampler unit comprises: A latent code generation module is configured to perform linear transformation on standard Gaussian noise by using a learnable non-singular matrix and an offset vector. A loss optimization module is configured to optimize the non-singular matrix and the offset vector by using a loss function comprising a trajectory error term, a KL divergence term and a diversity penalty term. A parallel sampling module is configured to generate K sets of latent codes in parallel by using the optimized non-singular matrix and the offset vector, and output K trajectory samples, wherein K is an integer greater than 1.

[0083] In the embodiment, the latent code generation module performs linear transformation on standard Gaussian noise by using a learnable non-singular matrix and an offset vector; the loss optimization module optimizes the non-singular matrix and the offset vector by using a loss function comprising a trajectory error term, a KL divergence term and a diversity penalty term; and the parallel sampling module generates K sets of latent codes in parallel by using the optimized non-singular matrix and the offset vector, and drives the future trajectory decoder unit to output K trajectory samples, wherein K is an integer greater than 1.

[0084] In some embodiments of the present application, the latent code generation module comprises: A latent code generation submodule is configured to generate a set of latent codes for each sample k, and the latent code is represented as wherein , is a non-singular matrix, is an offset vector.

[0085] In some embodiments of the present application, the loss optimization module comprises: A loss optimization submodule is configured to optimize by using the following loss function: , Where the first term encourages the sample to be close to the ground truth trajectory, the second term ensures the latent code to conform to the prior distribution, and the third term promotes diversity by penalizing the similarity between samples.

[0086] In the above embodiment, the technical parameters can be: the sampling number K = 20, the diversity loss scaling factor (ETH / UCY), 20 (nuScenes), the training iteration is 50 epochs, and the initial learning rate is 10 -4 .

[0087] The present application has a diversified trajectory sampler and loss function design. The trajectory sampler unit generates K sets of latent codes through linear transformation. The loss function includes three terms: the error of the sample and the true trajectory, the KL divergence (constraint latent distribution), and the diversity penalty term (encourage sample difference). This design solves the homogenization problem of traditional random sampling. Through the linear transformation parameterization of the sampler and the weight regulation strategy of the diversity loss, diversified trajectory samples are generated. Experiments show that the sample coverage can be improved by 24%.

[0088] Comparing the present application with the prior art (Trajectron++) can obtain the following table 1: Table 1 It should be noted that in the above table 1, the slightly higher ADE is because the present application pays more attention to long-term prediction and diversity, and the FDE (long-term error) is better.

[0089] Comparing the present application with the optimal data of the prior art can obtain the following table 2: Table 2 The present application has the following significant advantages and beneficial effects through the innovative system architecture, joint modeling mechanism and diversified sampling strategy, as follows: I. Significant improvement in prediction accuracy and long-term dependence capturing ability ① The advantage of joint modeling of time and social dimensions Existing technologies (such as Social LSTM and Transformer-TF) process the temporal and social dimensions separately, leading to the loss of long-term interaction information between agents. This invention achieves joint modeling of both dimensions through the flattened sequence representation of Agent-Aware Transformer and the agent's perceptual attention mechanism, enabling the state of an agent at any given time to directly influence the future states of other agents. For example, in autonomous driving scenarios, when vehicle A suddenly decelerates, this invention can directly transmit this state to the future trajectory prediction of vehicle B, while traditional methods require multiple steps of temporal encoding and social pooling, resulting in information delays. Experimental data shows that on the nuScenes dataset (6-second long temporal prediction), the average displacement error (ADE) of this invention is 1.45 meters, a 4% reduction compared to Trajectron++'s 1.51 meters; the final displacement error (FDE) is 2.86 meters, significantly better than DSF-AF's 3.71 meters and DLow-AF's 3.58 meters. On the ETH / UCY pedestrian dataset, the FDE of this invention is 0.39 meters, which is 26% lower than STAR's 0.53 meters. It performs better in long-term prediction, demonstrating its stronger ability to capture long-range dependencies.

[0090] ② Improved accuracy in retaining agent identity information Traditional Transformers, by ignoring agent identities, often confuse "temporal dependencies of the same agent" with "social interactions between different agents." The agent-aware attention mechanism of this invention utilizes a dual projection matrix (…). , The mask matrix M and the inner-agent matrix M are used to model intra-agent and inter-agent associations, respectively, so that identity information can be preserved. Ablation experiments show that after removing agent-aware attention (“w / o AA attention”), the average FDE on the ETH / UCY dataset increases from 0.39 meters to 0.47 meters, a performance decrease of 21%, demonstrating the crucial role of this mechanism in accuracy.

[0091] II. Breakthroughs in Social Rationality and Multimodal Prediction Capabilities ① Social Consistency Based on Joint Latent Intent Modeling The existing method (such as Trajectron++) independently models the potential intention of the agent, which leads to unreasonable situations such as "pedestrians crossing the road while vehicles not avoiding". The present application jointly models the potential intention of all agents through the CVAE framework, so that the potential intentions influence each other. For example, when the "crossing intention" of the pedestrian is detected, the "avoidance intention" of the vehicle will be activated at the same time. The experimental visualization results show that 87% of the trajectories generated by the present application can reflect natural social behavior (such as following and collision avoidance), while the traditional method is only 62%.

[0092] ②, Diversified sampling covers more real scene patterns The random sampling of the existing technology (such as the method based on GAN) is prone to lead to homogenization of trajectory samples, and it is difficult to cover low-probability but key scenes (such as emergency avoidance). The trajectory sampler unit of the present application generates trajectory samples with an average distance of 42% higher than traditional methods through a diversity loss function (penalizing the similarity between samples). On the ETH / UCY dataset, when K=20, the trajectory samples of the present application can cover 92% of the real trajectory patterns, while SGAN only covers 68%.

[0093] Three, the enhancement of system stability and dynamic scene adaptability ①, parameter regularization and gradient control for stability improvement The present application introduces KL divergence regularization in the CVAE prior and posterior units to constrain the distribution range of the potential intention; and through a mask in the attention mechanism, invalid interactions (such as agents with a distance of more than 100 meters) are controlled to avoid abnormal fluctuations of parameters. Experiments show that the convergence speed of the model of the present application is 20% faster than that of the RNN-based method, and the loss remains stable (fluctuation amplitude <3%) after 500 iterations, while the fluctuation amplitude of the traditional method can reach 8%-12%.

[0094] ②, Robustness optimization of dynamic scenes The existing method needs to rebuild the model when the number of agents changes (such as pedestrians suddenly leaving the scene), which leads to performance degradation. The present application can directly handle the increase and decrease of agents through dynamic sequence length (without filling missing values) and flexible attention mask. In the "vehicle merging and merging out" scene of nuScenes, the ADE fluctuation of the present application is only 0.08 meters, while that of Trajectron++ is 0.15 meters, proving that the present application is more adaptable to dynamic scenes.

[0095] Four, reduction of computational efficiency and deployment cost ①, high efficiency of lightweight architecture The Agent-Aware Transformer of the present application adopts a 2-layer encoder-decoder structure (hidden dimension of 512), which reduces the calculation amount by 50% compared with the 4-layer Transformer of STAR. On the same hardware (such as NVIDIA Tesla V100), when processing the trajectories of 100 agents, the inference speed of the present application is 32 FPS, which is 45% higher than the 22 FPS of Trajectron++, meeting the real-time requirements (≥20 FPS) of autonomous driving.

[0096] ②, hardware compatibility and deployment flexibility The prior art (such as methods based on GNN) relies on special-purpose graphics processors to accelerate graph convolution operations, while the core modules (such as attention mechanisms, MLPs, etc.) of the present application can run on general-purpose CPUs and can be deployed only with video acquisition devices and ordinary computing platforms. Compared with laser radar assisted prediction systems, the hardware cost is reduced by more than 60%, and the energy consumption is reduced by 35%.

[0097] Five, operation convenience and expansion of application scenarios ①, full-process automation and low-threshold operation The present application realizes automation from trajectory sequence conversion, feature encoding to trajectory generation without human intervention. The system provides a graphical user interface (GUI) to support real-time visualization of trajectory prediction results, and the operator can flexibly control the prediction diversity by adjusting the sampling number (K value). Non-professionals can master it after 1 hour of training.

[0098] ②, cross-scene adaptability and multi-domain application The present application does not rely on prior knowledge of specific scenarios and can be directly applied to multi-agent trajectory prediction such as pedestrian (ETH / UCY), vehicle (nuScenes), etc. In intelligent monitoring, it can predict the flow of people in a shopping mall; in robot cooperation, it can plan obstacle avoidance paths for multiple robots, and the application range is expanded by more than 3 times compared with special-purpose models (such as PECNet which is only applicable to pedestrian).

[0099] Reference Figure 2 The embodiment of the present application provides a multi-agent trajectory prediction method, comprising the following steps: Step 100, converting the received historical trajectories of multiple agents into a flattened historical trajectory sequence; Step 200, generating a timestamp feature and adding the timestamp feature to the historical trajectory sequence; Step 300, encoding the historical trajectory sequence added with the timestamp feature to jointly model the time dimension and the social dimension, and outputting an encoded historical feature sequence, and generating a future feature sequence based on the encoded historical feature sequence and a known future trajectory sequence; Step 400, modeling a prior distribution of all agent potential intentions according to the historical feature sequence by a prior network, and modeling an approximate posterior distribution of all agent potential intentions according to the historical feature sequence and the future feature sequence by a posterior network; Step 500, processing semantic map information to generate semantic map features; Step 600, generating a future trajectory of each agent based on the historical feature sequence, the sampled potential intention, and the semantic map features; Step 700, generating a plurality of diversified future trajectory prediction samples by optimizing a strategy to generate a plurality of diversified potential intention sampling values.

[0100] In some embodiments of the present application, the received historical trajectories of a plurality of agents are converted into a flattened historical trajectory sequence, including the following steps: Suppose there are N agents, and the historical trajectory contains H+1 time points, and the state of each agent n at time t is The historical trajectory is converted into a flattened historical trajectory sequence by the following formula: The length is: .

[0101] In some embodiments of the present application, the received historical trajectories of a plurality of agents are converted into a flattened historical trajectory sequence, including the following steps: When the number of agents changes over time, the features corresponding to missing agents are removed from the historical trajectory sequence, or the features corresponding to added agents are added.

[0102] In some embodiments of the present application, the timestamp feature is generated and added to the historical trajectory sequence, including the following steps: A sine-cosine function is used to generate a timestamp feature; The historical trajectory sequence and the timestamp feature are spliced and fused by linear transformation; The historical trajectory sequence added with the timestamp feature is output.

[0103] In some embodiments of the present application, the sine-cosine function is used to generate a timestamp feature, including the following steps: The timestamp feature is generated by the following formula: , wherein, is represented as a timestamp feature, is represented as a timestamp dimension, is represented as mapping historical time steps to non-negative integers.

[0104] In some embodiments of the present application, the concatenation of the historical trajectory sequence and the timestamp feature, and the fusion through linear transformation, include the following steps: concatenating the historical trajectory sequence with the timestamp feature and fusing through linear transformation: wherein, is represented as a learnable weight matrix, is represented as concatenation.

[0105] In some embodiments of the present application, the encoding of the historical trajectory sequence added with the timestamp feature for jointly modeling time dimension and social dimension, and the output of the encoded historical feature sequence, and the generation of the future feature sequence based on the encoded historical feature sequence and the known future trajectory sequence, include the following steps: outputting the encoded historical feature sequence after inputting the historical trajectory sequence added with the timestamp feature; generating two groups of key-value pairs of the plurality of agents through a double projection matrix using an attention mechanism, the two groups of key-value pairs respectively corresponding to queries, keys and values of the same agent at different time steps, and queries, keys and values of different agents at the same time step; generating the future feature sequence based on the encoded historical feature sequence and the known future trajectory sequence.

[0106] In some embodiments of the present application, the output of the encoded historical feature sequence after inputting the historical trajectory sequence added with the timestamp feature includes: receiving the historical trajectory sequence added with the timestamp feature; encoding using a 2-layer Transformer encoder, wherein each layer uses 8-headed agent-aware attention and a feed-forward network; outputting the encoded historical feature sequence, wherein the historical feature sequence is represented as , containing time dimension and social dimension.

[0107] In some embodiments of the present application, the same design structure is used in the process of encoding and in the process of decoding.

[0108] In some embodiments of the present application, the history trajectory sequence to which the timestamp feature is added is encoded to jointly model the time dimension and the social dimension, and an encoded history feature sequence is output, and a future feature sequence is generated based on the encoded history feature sequence and a known future trajectory sequence, including the following steps: A mask matrix is dynamically generated according to the identity of the agent to which the query-key belongs, to selectively activate the projection matrix.

[0109] In some embodiments of the present application, the mask matrix is dynamically generated according to the identity of the agent to which the query-key belongs, to selectively activate the projection matrix, including the following steps: By Controlling the activation of the projection matrix, wherein L represents the sequence length; when the query and the key belong to the same agent, , the projection matrix matching the same agent at different time points is activated; otherwise , the projection matrix matching different agents at the same time is activated.

[0110] In some embodiments of the present application, the prior network models the prior distribution of the potential intentions of all agents according to the history feature sequence, and the posterior network models the approximate posterior distribution of the potential intentions of all agents according to the history feature sequence and the future feature sequence, including the following steps: After performing agent-wise pooling on the history feature sequence of each agent, the mean and variance of the potential intention of the agent are output through the MLP to form a joint Gaussian prior distribution; After performing agent-wise pooling on the history feature sequence and the future feature sequence of each agent, the mean and variance of the potential intention of the agent are output through the MLP to form a joint Gaussian approximate posterior distribution.

[0111] In some embodiments of the present application, the prior network models the prior distribution of the potential intentions of all agents according to the history feature sequence, and the posterior network models the approximate posterior distribution of the potential intentions of all agents according to the history feature sequence and the future feature sequence, including the following steps: In the training phase, the KL divergence between the joint Gaussian prior distribution and the joint Gaussian approximate posterior distribution is minimized.

[0112] In some embodiments of the present application, the semantic map information is processed to generate semantic map features, including the following steps: A local map region of a fixed size is rotated according to the orientation of each agent and centered on the current position of the agent; perform feature extraction on the local map region through a convolutional neural network to obtain visual features; concatenate the visual features and the trajectory features of the agent.

[0113] In some embodiments of the present application, generating the future trajectory of each agent based on the historical feature sequence, the sampled potential intention, and the semantic map feature comprises the following steps: taking the initial position of each agent as an initial input; fusing the historical feature sequence, the sampled potential intention, and the semantic map feature to generate a trajectory at the next time; outputting a complete future trajectory after T iterations.

[0114] In some embodiments of the present application, generating the future trajectory of each agent based on the historical feature sequence, the sampled potential intention, and the semantic map feature comprises the following steps: limiting the calling of the trajectory features before a certain time when generating a trajectory at the time.

[0115] In some embodiments of the present application, generating a plurality of diversified future trajectory prediction samples by generating a plurality of diversified potential intention sampling values through an optimization strategy comprises the following steps: linearly transforming standard Gaussian noise with a learnable non-singular matrix and an offset vector; optimizing the non-singular matrix and the offset vector with a loss function comprising a trajectory error term, a KL divergence term, and a diversity penalty term; generating K groups of potential codes in parallel using the optimized non-singular matrix and the offset vector and outputting K trajectory samples, wherein K is an integer greater than 1.

[0116] In some embodiments of the present application, linearly transforming standard Gaussian noise with a learnable non-singular matrix and an offset vector comprises the following steps: for each sample k, generating a group of potential codes, the potential code being represented as wherein , is a non-singular matrix, is an offset vector.

[0117] In some embodiments of the present application, generating K groups of potential codes in parallel using the optimized non-singular matrix and the offset vector and outputting K trajectory samples, wherein K is an integer greater than 1, comprises the following steps: optimizing through the following loss function: , where the first term encourages samples to be close to the true trajectory, the second term ensures the latent code conforms to the prior distribution, and the third term promotes diversity by penalizing the similarity between samples.

[0118] The specific implementation of the multi-agent trajectory prediction method provided in the application can refer to the specific implementation of the multi-agent trajectory prediction device provided in the above, and will not be described here, The computer device provided in the embodiment of the application includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the multi-agent trajectory prediction method when executing the computer program.

[0119] The embodiment of the application further provides a storage device having a computer program (instruction) stored thereon, and the computer program (instruction) implements the steps of the multi-agent trajectory prediction method when executed by a processor.

[0120] For example, the computer program can be divided into one or more modules, and the one or more modules are stored in the memory and executed by the processor to complete the application. One or more modules can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the computer device. For example, the computer program can be divided into the steps of the multi-agent trajectory prediction method provided in each method embodiment.

[0121] Those skilled in the art can understand that the above description of the computer device is only an example and does not constitute a limitation on the computer device, and can include more or fewer components than the above description, or combine certain components, or different components, for example, can include input and output devices, network access devices, buses, etc.

[0122] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The processor is the control center of the computer device, and connects all parts of the computer device through various interfaces and lines.

[0123] The memory can be configured to store the computer programs and / or modules, and the processor realizes various functions of the computer device by running or executing the computer programs and / or modules stored in the memory, and calling data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as an interface display function, an interface interaction function, etc.), and the like; and the data storage area can store data created according to the use of the mobile phone (such as a map interface, a selection interface, etc.), and the like. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0124] The modules / units integrated in the computer device, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage device. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage device. The computer program can realize the steps of each method embodiment when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms. The computer readable medium can include any entity or device capable of carrying the computer program code, recording media, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, electrical signals, and software distribution media, etc.

[0125] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily mean the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0126] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the present application. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the present application, several modifications and improvements can be made, which are within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A Transformer-based multi-agent trajectory prediction apparatus, characterized in that, The multi-agent trajectory prediction device comprises: a trajectory sequence representation unit configured to convert a plurality of received historical trajectories of agents into a flattened historical trajectory sequence; a time encoder unit configured to generate timestamp features and add the timestamp features to the historical trajectory sequence; a Transformer encoder-decoder unit configured to encode the historical trajectory sequence to which the timestamp features are added, to jointly model a time dimension and a social dimension, and output an encoded historical feature sequence, and generate a future feature sequence based on the encoded historical feature sequence and a known future trajectory sequence; a CVAE prior and posterior unit configured to model a prior distribution of potential intentions of all agents according to the historical feature sequence through a prior network, and model an approximate posterior distribution of potential intentions of all agents according to the historical feature sequence and the future feature sequence through a posterior network; a future trajectory decoder unit configured to generate a future trajectory of each agent based on the historical feature sequence and the potential intentions sampled by the CVAE prior and posterior unit.

2. The Transformer-based multi-agent trajectory prediction apparatus according to claim 1, wherein, The trajectory sequence representation unit comprises: a historical trajectory sequence representation module configured to, assuming there are N agents, the historical trajectory contains H+1 time points, and the state of each agent n at time t is convert the historical trajectory into a flattened historical trajectory sequence by the following formula: with a length of: .

3. The Transformer-based multi-agent trajectory prediction apparatus according to claim 2, wherein, The trajectory sequence representation unit further comprises: a dynamic scenario adaptation module configured to remove features corresponding to missing agents from the historical trajectory sequence or add features corresponding to increased agents when the number of agents changes over time.

4. The Transformer-based multi-agent trajectory prediction apparatus according to claim 1, wherein, The Transformer encoder-decoder unit comprises: an encoder module configured to output an encoded historical feature sequence after inputting the historical trajectory sequence to which the timestamp features are added; an attention mechanism module configured to generate two groups of key-value pairs of the plurality of agents by a double projection matrix using an attention mechanism, the two groups of key-value pairs corresponding to queries, keys and values of the same agent at different time points and queries, keys and values of different agents at the same time point, respectively; a decoder module configured to generate a future feature sequence based on the encoded historical feature sequence and a known future trajectory sequence.

5. The Transformer-based multi-agent trajectory prediction apparatus according to claim 4, wherein, The encoder module comprises: a sequence receiving submodule configured to receive the historical trajectory sequence to which the timestamp features are added; an encoding submodule configured to encode using a 2-layer Transformer encoder, wherein each layer uses an 8-head agent-aware attention and a feedforward network; The sequence output submodule is configured to output the encoded historical feature sequence, wherein the historical feature sequence is represented as , and comprises a time dimension and a social dimension. 6.The Transformer-based multi-agent trajectory prediction apparatus of claim 4, wherein, The encoder module and the decoder module use the same design structure.

7. The Transformer-based multi-agent trajectory prediction apparatus according to claim 4, characterized in that, The Transformer encoder-decoder unit further comprises: a mask module configured to dynamically generate a mask matrix according to the identities of the agents to which the queries and keys belong, to selectively activate the projection matrix.

8. The Transformer-based multi-agent trajectory prediction apparatus according to claim 7, characterized in that, The mask module comprises: The mask submodule is configured to activate a projection matrix by controlling, wherein L represents the sequence length; when the query and the key belong to the same agent, , the projection matrix matching the same agent at different time points is activated; otherwise , the projection matrix matching different agents at the same time is activated.

9. The Transformer-based multi-agent trajectory prediction apparatus of claim 1, wherein, The multi-agent trajectory prediction device further comprises: a semantic map encoding unit configured to process semantic map information to generate semantic map features, and the future trajectory decoder unit generates a future trajectory of each agent based on the historical feature sequence, the potential intentions sampled by the CVAE prior and posterior unit, and the semantic map features; a trajectory sampler unit configured to generate a plurality of sets of diversified latent intention sampling values by an optimization policy, so as to drive the future trajectory decoder unit to generate a plurality of diversified future trajectory prediction samples.

10. A method for multi-agent trajectory prediction based on a Transformer, characterized in that, comprising the following steps: converting a plurality of received historical trajectories of agents into a flattened historical trajectory sequence; generating a timestamp feature and adding the timestamp feature to the historical trajectory sequence; encoding the historical trajectory sequence to which the timestamp feature is added to jointly model the time dimension and the social dimension, and output an encoded historical feature sequence, and generating a future feature sequence based on the encoded historical feature sequence and a known future trajectory sequence; modeling a prior distribution of all agent latent intentions according to the historical feature sequence by a prior network, and modeling an approximate posterior distribution of all agent latent intentions according to the historical feature sequence and the future feature sequence by a posterior network; generating a future trajectory of each agent based on the historical feature sequence and the sampled latent intention. a trajectory sampler unit configured to generate a plurality of sets of diversified latent intention sampling values by an optimization policy, so as to drive the future trajectory decoder unit to generate a plurality of diversified future trajectory prediction samples. comprising the following steps: converting a plurality of received historical trajectories of agents into a flattened historical trajectory sequence; generating a timestamp feature and adding the timestamp feature to the historical trajectory sequence; encoding the historical trajectory sequence to which the timestamp feature is added to jointly model the time dimension and the social dimension, and output an encoded historical feature sequence, and generating a future feature sequence based on the encoded historical feature sequence and a known future trajectory sequence; modeling a prior distribution of all agent latent intentions according to the historical feature sequence by a prior network, and modeling an approximate posterior distribution of all agent latent intentions according to the historical feature sequence and the future feature sequence by a posterior network; generating a future trajectory of each agent based on the historical feature sequence and the sampled latent intention.

Images