Multi-agent trajectory prediction device and method

Abstract

The invention belongs to the technical field of automatic driving, and particularly relates to a multi-agent trajectory prediction device and method, and the device comprises a Transform encoder-decoder unit, a CVAE prior and posterior unit, a semantic map coding unit, a future trajectory decoder unit and a trajectory sampler unit. According to the invention, diversified sampling strategies are provided, track samples with higher coverage are generated, and the adaptability of a dynamic scene is enhanced.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving technology, and in particular relates to a multi-agent trajectory prediction device and method. Background Technology

[0002] Multi-agent trajectory prediction is one of the core technologies in fields such as autonomous driving and intelligent monitoring. Its core objective is to accurately predict the future movement trajectories of multiple intelligent agents such as pedestrians and vehicles based on historical trajectory data.

[0003] Existing multi-agent trajectory prediction methods are based on recurrent neural networks (RNNs), with Social LSTM as a representative. These methods model the trajectory of each agent in the time dimension using a Long Short-Term Memory (LST) network to capture its historical motion patterns. Simultaneously, they introduce a "social pooling layer" to aggregate the state information of surrounding agents, achieving simple social interaction modeling. The core modules include a temporal feature extraction module, a social interaction module, and a trajectory prediction module. The temporal feature extraction module uses LSTM to encode the historical position and velocity sequences of a single agent, outputting a temporal feature vector. The social interaction module aggregates the temporal features of other agents within a certain range around the target agent through pooling operations (e.g., max pooling, average pooling) to generate social features. The trajectory prediction module fuses the temporal and social features and outputs the future trajectory through a decoder.

[0004] Existing multi-agent trajectory prediction methods can also be based on traditional Transformer approaches. These methods apply Transformer to trajectory prediction and utilize self-attention mechanisms to capture long-term dependencies, but they are not optimized for multi-agent scenarios. Their core modules include a sequence transformation module, a self-attention module, and an independent prediction module. The sequence transformation module converts each agent's historical trajectory into an independent sequence, marking time information through position encoding. The self-attention module uses a standard self-attention mechanism to calculate the correlation between elements within the sequence, capturing temporal dependencies. The independent prediction module independently decodes the features of each agent and outputs the future trajectory, without considering interactions between agents.

[0005] Existing multi-agent trajectory prediction methods can also be based on graph neural networks (GNNs, such as GraphAttention Networks), exemplified by Social-BiGAT. These methods first extract temporal features using RNNs, and then use GNNs to model social interactions between agents. Their core modules include a temporal encoding module, a social graph construction module, a graph convolution module, and a joint prediction module. The temporal encoding module uses an RNN to process the historical trajectory of each agent to obtain temporal features. The social graph construction module constructs a social graph with agents as nodes and distance or interaction strength as edges. The graph convolution module propagates node features through GNNs to capture social interactions. The joint prediction module fuses temporal and social features to predict future trajectories.

[0006] Existing multi-agent trajectory prediction methods can also be based on generative models, such as the Trajectron++ method. These methods use conditional variational autoencoders (CVAEs) or generative adversarial networks (GANs) to model the multimodal distribution of trajectories, attempting to capture the uncertainty of future trajectories. Their core modules include a feature encoding module, a latent distribution modeling module, and a multimodal sampling module. The feature encoding module fuses historical trajectories with scene information (e.g., semantic maps) to generate latent features. The latent distribution modeling module outputs the distribution parameters of latent variables through the encoder of the CVAE. The decoder generates future trajectories based on the latent variables. The multimodal sampling module generates diverse trajectory candidates by sampling latent variables.

[0007] Existing methods suffer from insufficient predictive diversity and low coverage. Their trajectory sampling strategies tend to generate similar trajectory samples, making it difficult to cover all possible future patterns. For example, in an intersection scenario, vehicles may go straight, turn left, or turn right, but existing sampling methods often favor a particular pattern, resulting in insufficient coverage of low-probability but critical trajectories such as emergency avoidance.

[0008] Furthermore, existing methods suffer from poor adaptability to dynamic scenarios and insufficient robustness. When the number of agents changes over time, existing methods need to fill in missing data or rebuild the model, leading to performance degradation. For example, in surveillance video, when a pedestrian suddenly walks out of the frame, the recursive structure of an RNN cannot effectively encode temporal features due to incomplete input, and the graph structure of a GNN also needs to be rebuilt, resulting in poor adaptability. Summary of the Invention

[0009] Based on this, the purpose of this invention is to provide a multi-agent trajectory prediction device and method to fundamentally solve the problems of low trajectory coverage and unstable performance in dynamic scenes caused by the defects of existing sampling strategies.

[0010] The present invention is implemented as follows: a multi-agent trajectory prediction device is provided, comprising: The Transformer encoder-decoder unit is used to encode the received historical trajectory sequence to jointly model the time and social dimensions, and output the encoded historical feature sequence, as well as generate the future feature sequence based on the encoded historical feature sequence and the known future trajectory sequence. The CVAE prior and posterior units are used to model the prior distribution of all agents' potential intentions based on the historical feature sequence through a prior network, and to model the approximate posterior distribution of all agents' potential intentions based on the historical feature sequence and the future feature sequence through a posterior network. Semantic map encoding unit is used to process semantic map information and generate semantic map features; The future trajectory decoder unit is used to generate the future trajectory of each agent based on the historical feature sequence, the latent intent sampled by the CVAE prior and posterior units, and the semantic map features. The trajectory sampler unit is used to generate multiple sets of diverse potential intent sample values ​​through an optimization strategy, thereby driving the future trajectory decoder unit to generate multiple diverse future trajectory prediction samples.

[0011] In some embodiments, the Transformer encoder-decoder unit includes: The encoder module is used to output an encoded historical feature sequence after inputting the historical trajectory sequence with the timestamp feature added. The attention mechanism module is used to generate two sets of key-value pairs for the multiple agents through a dual projection matrix using the attention mechanism. The two sets of key-value pairs correspond to the query, key and value of the same agent at different times, and the query, key and value of different agents at the same time. The decoder module is used to generate a future feature sequence based on the encoded historical feature sequence and the known future trajectory sequence.

[0012] In some embodiments, the encoder module includes: A sequence receiving submodule is used to receive the historical trajectory sequence with the timestamp feature added; The encoding submodule is used for encoding with a 2-layer Transformer encoder, where each layer uses an 8-head agent perception attention and feedforward network; The sequence output submodule is used to output the encoded historical feature sequence, wherein the historical feature sequence is represented as... It includes both time and social dimensions.

[0013] In some implementations, the encoder module and the decoder module adopt the same design structure.

[0014] In some embodiments, the semantic map encoding unit includes: The map cropping module is used to crop a local map area of ​​a fixed size, centered on the current position of each agent and rotating it according to its orientation. The CNN feature extraction module is used to extract features from the local map region through a convolutional neural network to obtain visual features; The feature stitching module is used to stitch the visual features with the trajectory features of the intelligent agent.

[0015] In some embodiments, the trajectory sampler unit includes: The latent code generation module is used to perform a linear transformation on standard Gaussian noise with a learnable non-singular matrix and an offset vector. The loss optimization module is used to optimize the non-singular matrix and the offset vector using a loss function that includes a trajectory error term, a KL divergence term, and a diversity penalty term. The parallel sampling module is used to generate K sets of latent codes in parallel using the optimized non-singular matrix and the offset vector, and output K trajectory samples, where K is an integer greater than 1.

[0016] In some implementations, the latent code generation module includes: The latent code generation submodule is used to generate a set of latent codes for each sample k, where the latent codes are represented as follows: ,in , It is a non-singular matrix. This is the offset vector.

[0017] In some implementations, the loss optimization module includes: The loss optimization submodule is used for optimization using the following loss function: , The first measure encourages samples to closely approximate real trajectories, the second ensures that potential codes conform to prior distributions, and the third enhances diversity by penalizing similarities between samples.

[0018] In some embodiments, the multi-agent trajectory prediction device further includes: The trajectory sequence representation unit is used to convert the received historical trajectories of multiple agents into a flattened historical trajectory sequence; A time encoder unit is used to generate timestamp features, add the timestamp features to the historical trajectory sequence, and transmit the historical trajectory sequence with the added timestamp features to the Transformer encoder-decoder unit.

[0019] This invention also provides a multi-agent trajectory prediction method, comprising the following steps: The received historical trajectory sequence is encoded to jointly model the time and social dimensions, and the encoded historical feature sequence is output. Based on the encoded historical feature sequence and the known future trajectory sequence, a future feature sequence is generated. The prior network models the prior distribution of all agents' potential intentions based on the historical feature sequence, and the posterior network models the approximate posterior distribution of all agents' potential intentions based on the historical feature sequence and the future feature sequence. Process semantic map information to generate semantic map features; The future trajectories of each agent are generated based on the historical feature sequence, the sampled latent intentions, and the semantic map features. By optimizing the strategy, multiple sets of diverse potential intent sample values ​​are generated, thereby generating multiple diverse future trajectory prediction samples.

[0020] The multi-agent trajectory prediction device and method provided in this invention include: a Transformer encoder-decoder unit that encodes received historical trajectory sequences to jointly model the time and social dimensions, outputting an encoded historical feature sequence; and a future feature sequence generated based on the encoded historical feature sequence and known future trajectory sequences. A CVAE prior and posterior unit models the prior distribution of all agents' potential intentions based on the historical feature sequence using a prior network, and an approximate posterior distribution of all agents' potential intentions based on the historical and future feature sequences using a posterior network. A semantic map encoding unit processes semantic map information and generates semantic map features. A future trajectory decoder unit generates the future trajectories of each agent based on the historical feature sequence, the potential intentions sampled by the CVAE prior and posterior units, and the semantic map features. A trajectory sampler unit generates multiple sets of diverse potential intention sample values ​​through an optimization strategy, thereby driving the future trajectory decoder unit to generate multiple diverse future trajectory prediction samples. This invention proposes a diversified sampling strategy to generate trajectory samples with higher coverage and enhance adaptability to dynamic scenes. Attached Figure Description

[0021] Figure 1 This is a structural block diagram of the multi-agent trajectory prediction device provided in the embodiments of the present invention; Figure 2 This is a flowchart of the multi-agent trajectory prediction method provided in the embodiments of the present invention. Detailed Implementation

[0022] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0023] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0025] refer to Figure 1 This invention provides a multi-agent trajectory prediction device, comprising: The trajectory sequence representation unit is used to convert the received historical trajectories of multiple agents into a flattened historical trajectory sequence; A time encoder unit is used to generate timestamp features and add the timestamp features to the historical trajectory sequence; The Transformer encoder-decoder unit is used to encode the historical trajectory sequence with the timestamp feature added to jointly model the time dimension and the social dimension, and output the encoded historical feature sequence, as well as generate the future feature sequence based on the encoded historical feature sequence and the known future trajectory sequence. The CVAE prior and posterior units are used to model the prior distribution of all agents' potential intentions based on the historical feature sequence through a prior network, and to model the approximate posterior distribution of all agents' potential intentions based on the historical feature sequence and the future feature sequence through a posterior network. Semantic map encoding unit is used to process semantic map information and generate semantic map features; The future trajectory decoder unit is used to generate the future trajectory of each agent based on the historical feature sequence, the latent intent sampled by the CVAE prior and posterior units, and the semantic map features. The trajectory sampler unit is used to generate multiple sets of diverse potential intent sample values ​​through an optimization strategy, thereby driving the future trajectory decoder unit to generate multiple diverse future trajectory prediction samples.

[0026] In this invention, the Transformer and CVAE frameworks are used to achieve joint modeling of the time and social dimensions, agent identity perception, latent intent modeling, and diverse trajectory generation. Specifically, an external device, such as a computer, transmits the historical trajectories of multiple agents to a trajectory sequence representation unit. The trajectory sequence representation unit converts the received historical trajectories of the multiple agents into a flattened historical trajectory sequence to adapt to the input format of the subsequent Transformer encoder-decoder unit, and transmits the historical trajectory sequence to a time encoder unit. The time encoder unit generates timestamp features and adds these features to the received historical trajectory sequence, preserving the time dimension information, and transmits the time-stamped historical trajectory sequence to the Transformer encoder-decoder unit. The Transformer encoder-decoder unit encodes the received time-stamped historical trajectory sequence to jointly model the time and social dimensions, outputting an encoded historical feature sequence. Based on the encoded historical feature sequence and a known future trajectory sequence, a future feature sequence is generated. The historical and future feature sequences are then transmitted to the CVAE prior and posterior units, and the historical feature sequence is transmitted to the future trajectory decoding unit. The multi-agent trajectory prediction device comprises several units: a CVAE prior and posterior unit, which models the prior distribution of all agents' potential intentions based on historical feature sequences using a prior network, and an approximate posterior distribution of all agents' potential intentions based on historical and future feature sequences using a posterior network; a semantic map encoding unit, which processes semantic map information, generates semantic map features, and integrates scene information (e.g., road, sidewalk) to enhance the scene adaptability of the prediction; and a future trajectory decoder unit, which generates future trajectories for each agent based on the received historical feature sequences, potential intentions sampled by the CVAE prior and posterior units, and semantic map features. This autoregressive generation of future trajectories ensures temporal consistency. A trajectory sampler unit generates multiple sets of diverse potential intention sample values ​​through optimization strategies, thereby driving the future trajectory decoder unit to generate multiple diverse future trajectory prediction samples, optimizing the distribution of potential intentions, and improving coverage. Finally, the multi-agent trajectory prediction device outputs the optimal future trajectory prediction sample or all generated future trajectory prediction samples for use by downstream decision-making systems.

[0027] This invention offers the following advantages: ① It employs a joint modeling framework to simultaneously capture temporal dependencies and social interactions; ② It designs an agent identity perception mechanism to retain agent identity information and accurately distinguish between temporal dependencies and social interactions; ③ It constructs a latent intent model, enabling agents' latent intents to influence each other and improving the social rationality of trajectories; ④ It proposes diverse sampling strategies to generate trajectory samples with higher coverage and enhance adaptability to dynamic scenarios. The multi-agent trajectory prediction device provided by this invention can provide more accurate, reasonable, and diverse trajectory prediction results for intelligent mobile devices in fields such as autonomous driving and intelligent monitoring, thereby improving the safety decision-making capabilities of intelligent mobile devices.

[0028] In existing technologies, the time dimension (e.g., the historical trajectory of a single agent) and the social dimension (e.g., the interaction between multiple agents) are often processed in stages. For example, the historical trajectory of each agent is first independently encoded using a time-series model such as LSTM, and then the interaction information is aggregated through a social pooling layer. This "time-series encoding → social aggregation" model has the following two drawbacks: ① Information fragmentation: Time-series dependencies (e.g., the agent's own movement trend) and social interactions (e.g., avoiding nearby vehicles) are separated into independent processing steps, resulting in the loss of key correlation information across time and across agents. For example, the deceleration behavior of vehicle A at t=2 cannot be directly correlated with the avoidance action of vehicle B at t=5. ② Poor dynamic adaptability: When the number of agents changes over time (e.g., pedestrians entering or leaving the scene), traditional methods need to fill in invalid data or reinitialize the model, leading to prediction bias.

[0029] To address this, the present invention provides a trajectory sequence representation unit that, through a flattened sequence joint representation method, converts the historical trajectories of multiple agents into a flattened historical trajectory sequence suitable for subsequent processing by the Transformer encoder-decoder unit, while retaining the association information between agents and time, thus achieving deep integration of the time dimension and the social dimension.

[0030] In some specific embodiments of this application, the trajectory sequence representation unit includes: The historical trajectory sequence representation module is used to represent the historical trajectory sequence, assuming there are N agents and the historical trajectory contains H+1 time points (t=-H, ..., 0). The state of each agent n at time t is... This state includes position, velocity, etc., and the historical trajectory is converted into a flattened historical trajectory sequence using the following formula: The length is: H is a positive integer greater than 0.

[0031] In the above formula, it can be expressed as: , Specifically, it can be represented as the historical trajectory of agent 1 at time -H. Specifically, it can be represented as the historical trajectory of agent 2 at time -H. Specifically, it can be represented as the historical trajectory of agent N at time -H. Specifically, it is represented as the historical trajectory of agent 1 at time -H+1. Specifically, it is represented as the historical trajectory of agent 2 at time -H+1. Specifically, it is represented as the historical trajectory of agent N at time 0.

[0032] In some specific embodiments of this application, the trajectory sequence representation unit further includes: The future trajectory sequence representation module is used to assume that the future trajectory contains T time points (t=1,...,T), and the position of each agent n at time t is... The future trajectory sequence can be specifically represented by the following formula: The length is: T is a positive integer greater than 0.

[0033] In the above formula, it can be expressed as: , Specifically, it represents the future trajectory of agent 1 at time 1. Specifically, it represents the future trajectory of agent 2 at time 1. Specifically, it represents the future trajectory of agent N at time 1. Specifically, it represents the future trajectory of agent 1 at time 2. Specifically, it represents the future trajectory of agent 2 at time 2. Specifically, it represents the future trajectory of agent 1 at time T. Specifically, it represents the future trajectory of agent N at time T.

[0034] In some specific embodiments of this application, the trajectory sequence representation unit further includes: The dynamic scene adaptation module is used to remove features corresponding to missing agents from the historical trajectory sequence or add features corresponding to agents as the number of agents changes over time.

[0035] In this embodiment, when the number of agents changes over time, such as when some agents leave or enter the scene, the dynamic scene adaptation module removes the features corresponding to the missing agents from the historical trajectory sequence, that is, removes the features corresponding to the agents that leave the scene, or adds the features corresponding to the agents that enter the scene.

[0036] For example, if a pedestrian leaves the scene at time t=3, the dynamic scene adaptation module directly truncates the corresponding position in the historical trajectory sequence, and the model can still infer the interaction relationship through the trajectory of the remaining agents.

[0037] This invention is based on a multi-agent trajectory joint representation method using flattened sequences, which converts the historical trajectories of multiple agents into "agent-time" flattened sequences (e.g., This method preserves the association between agent indices and timestamp features. It overcomes the limitations of traditional methods such as the independent sequences of Social LSTM and the graph structures of GNNs, enabling Transformer to handle dependencies in both the temporal and social dimensions simultaneously. This sequence structure forms the basis of joint modeling and directly determines the integrity of interaction information between agents; no existing technology employs such a flattened sequence.

[0038] This invention provides a time encoder unit that adds timestamp features to historical trajectory sequences, solving the problem that the Transformer encoder-decoder unit is insensitive to time dimension information, and ensuring that the model can distinguish features at different times.

[0039] In some specific embodiments of this application, the time encoder unit includes: The feature generation module is used to generate timestamp features using sine and cosine functions; The feature fusion module is used to concatenate the historical trajectory sequence and the timestamp feature, and then fuse them through linear transformation; The sequence output module is used to output the historical trajectory sequence with the timestamp feature added.

[0040] In this embodiment, the feature generation module generates timestamp features using sine and cosine functions and transmits the timestamp features to the feature fusion module; the feature fusion module concatenates the historical trajectory sequence and timestamp features and then fuses them through linear transformation; finally, the sequence output module outputs the historical trajectory sequence with added timestamp features.

[0041] In some specific embodiments of this application, the feature generation module includes: The feature generation submodule is used to generate timestamp features using the following formula: , The period of the sine and cosine is 10000. Represented as a timestamp feature, This can be represented as a timestamp dimension, for example, it can be set to 256. This means mapping historical moments to non-negative integers, for example, t=-H corresponds to 0, and t=0 corresponds to H.

[0042] In some specific embodiments of this application, the feature fusion module includes: The feature fusion submodule is used to integrate historical trajectory sequences. With timestamp features The pieces are then spliced ​​together and merged using a linear transformation: ,in Represented as a learnable weight matrix, This is represented as splicing.

[0043] Wherein, the dimension of the linear transformation matrix is , The historical trajectory sequence features are dimensional; for example, position + velocity is 4-dimensional.

[0044] The existing Transformer attention mechanism treats all elements in the sequence equally, leading to confusion between two types of key information in multi-agent scenarios. Specifically: ① Confusion between temporal dependencies and social interactions: For example, when predicting the future position of agent A, the model may assign the same attention weight to A's own historical position (temporal dependency) and the current position of neighboring agent B (social interaction), resulting in an unreasonable predicted trajectory. ② Loss of agent identity information: Although agent index encoding (e.g., position embedding) can partially distinguish identities, it introduces artificial dependencies (e.g., the model is sensitive to agent order), violating permutation invariance.

[0045] To address this, the present invention provides a Transformer encoder-decoder unit and designs a dual-path attention mechanism. By explicitly distinguishing the identities of agents, it achieves accurate interaction modeling, that is, it simultaneously captures the dependence in the temporal dimension (the temporal association of the same agent) and the interaction in the social dimension (the association of different agents), thus solving the problem of information loss caused by separate modeling.

[0046] In some specific embodiments of this application, the Transformer encoder-decoder unit includes: The encoder module is used to output an encoded historical feature sequence after inputting the historical trajectory sequence with the timestamp feature added. The attention mechanism module is used to generate two sets of key-value pairs for the multiple agents through a dual projection matrix using the attention mechanism. The two sets of key-value pairs correspond to the query, key and value of the same agent at different times, and the query, key and value of different agents at the same time. The decoder module is used to generate a future feature sequence based on the encoded historical feature sequence and the known future trajectory sequence.

[0047] In this embodiment, after the encoder module takes the historical trajectory sequence with added timestamp features as input, it outputs the encoded historical feature sequence, which includes time and social dimensions. Then, the attention mechanism module uses the attention mechanism to generate two sets of key-value pairs for multiple agents through a dual projection matrix. The two sets of key-value pairs correspond to the queries, keys and values ​​of the same agent at different times, and the queries, keys and values ​​of different agents at the same time. The decoder module generates the future feature sequence based on the encoded historical feature sequence and the known future trajectory sequence.

[0048] In some specific embodiments of this application, the encoder module includes: A sequence receiving submodule is used to receive the historical trajectory sequence with the timestamp feature added; The encoding submodule is used to encode using a 2-layer Transformer encoder, where each layer uses an 8-head intelligent agent perception attention and feedforward network, and the hidden dimension can be 512. The sequence output submodule is used to output the encoded historical feature sequence, wherein the historical feature sequence is represented as... It includes both time and social dimensions.

[0049] In some specific embodiments of this application, the encoder module and the decoder module adopt the same design structure.

[0050] The encoder module uses a 2-layer Transformer encoder for encoding, with each layer employing an 8-agent perception attention and feedforward network. Similarly, the decoder module uses a 2-layer Transformer encoder for encoding, with each layer employing an 8-agent perception attention and feedforward network.

[0051] In some specific embodiments of this application, the Transformer encoder-decoder unit further includes: The masking module is used to dynamically generate a mask matrix based on the identity of the agent to which the query key belongs, so as to selectively activate the projection matrix.

[0052] In this embodiment, the mask module can dynamically generate a mask matrix based on the identity of the smart agent to which the query key belongs, so as to selectively activate one of the projection matrices in the dual projection matrix.

[0053] In some specific embodiments of this application, the mask module includes: The mask submodule is used to pass through Control the activation projection matrix, where L represents the sequence length; when the query and key belong to the same agent. Activate the projection matrix that matches the same agent at different times; otherwise Activate the projection matrix that matches different agents at the same time.

[0054] The attention mechanism module and masking module provided in this invention distinguish between "intra-agent attention," i.e., the temporal dependence of the same agent, and "inter-agent attention," i.e., the social interaction between different agents, through a dual projection matrix and a mask. The calculation formula is as follows: , For the intra-agent branch, specialized key-value pairs are generated. This branch focuses only on the historical and future states of the same agent. For example, when predicting the position of agent A at t=5, this branch focuses on the historical position of agent A at t=3.

[0055] For the Inter-Agent branch: Generate key-value pairs This captures simultaneous interactions between different agents. For example, when predicting A's position at t=5, this branch focuses on neighboring agent B's position at t=4.

[0056] Through the mask matrix This controls which branch is activated, which can also be understood as controlling the activation of a certain projection matrix. The mask matrix is ​​dynamically generated based on the agent's index; specifically, when the query and key belong to the same agent, Activate the intra-agent branch; otherwise This activates the inter-agent branch. For example, when predicting the future trajectory of agent A, the mask matrix forces the model to assign different attention weights to agent A's own historical position (intra-agent) and the current position of surrounding vehicles (inter-agent), avoiding confusion between the two types of dependencies.

[0057] This invention designs an agent-aware attention mechanism, which utilizes a dual projection matrix ( Used for intra-agent attention (used for inter-agent attention) and mask matrix M ( The mechanism distinguishes agent identities by recognizing that the query and the key belong to the same agent (if and only if the query and the key belong to the same agent). The calculation of the attention weight matrix A considers both types of attention, ensuring that temporal dependencies and social interactions are not confused. Experiments show that removing this mechanism results in a 21% performance decrease, thus fully demonstrating its importance.

[0058] In existing technologies, the potential intentions of each agent are typically modeled independently, resulting in a lack of social plausibility in the generated trajectories. For example, a pedestrian's intention to "cross the road" and a vehicle's intention to "slow down and avoid" need to be correlated through an additional post-processing module, which often cannot capture complex dynamic interactions (such as multi-vehicle cooperative avoidance).

[0059] To address this, the present invention provides CVAE prior and posterior units to model the agent's potential intentions (e.g., the agent turns or goes straight), and through joint modeling, enables potential intentions to influence each other, thereby improving the social rationality of the trajectory.

[0060] In some specific embodiments of this application, the CVAE prior and posterior units include: The prior network module is used to perform agent-wise pooling on the historical feature sequence of each agent, and then output the mean and variance of the agent's potential intention through MLP to form a joint Gaussian prior distribution. The posterior network module is used to perform agent-wise pooling on the historical and future feature sequences of each agent, and then output the mean and variance of the agent's potential intention through MLP to form a joint Gaussian approximate posterior distribution.

[0061] In this embodiment, the prior network module performs agent-wise pooling on the historical feature sequence of each agent, and then outputs the mean and variance of the agent's potential intention through MLP to form a joint Gaussian prior distribution. For example, in a crossroads scenario, if the pedestrian's historical feature sequence shows "approaching the zebra crossing", the joint Gaussian prior distribution will infer that its potential intention is "crossing the street", and simultaneously adjust the vehicle's potential intention, for example, to "slow down and avoid".

[0062] Specifically, we first input the historical feature sequence C of each agent, and then perform agent-wise pooling on the historical feature sequence C of each agent, that is, calculate the mean of its historical feature sequence C for each agent. The calculation formula is as follows: Finally, it is mapped via MLP. To the joint Gaussian prior distribution The formula is: The hidden dimension of MLP is 256, which is 512 to 256. The potential intention of the intelligent agent.

[0063] In this embodiment, the posterior network module performs agent-wise pooling on the historical feature sequence and future feature sequence of each agent, and then outputs the mean and variance of the agent's potential intention through MLP to form a joint Gaussian approximate posterior distribution.

[0064] Specifically, the process first inputs the historical feature sequence C and the future feature sequence for each agent. Then, agent-wise pooling is performed on the historical feature sequence C and the future feature sequence for each agent, which means calculating the mean of the future feature sequence for each agent. Finally, the result is mapped to a joint Gaussian approximate posterior distribution through an MLP. The formula is: .

[0065] In the above implementation, during the joint modeling process, latent intent The joint Gaussian approximation posterior distribution is inferred jointly from the future trajectories of all agents, thus ensuring the interrelationship between potential intentions. Technical parameters can be: potential intention dimension. The MLP activation function is ReLU, and the dropout rate is 0.1.

[0066] In some specific embodiments of this application, the CVAE prior and posterior unit further includes: The KL divergence calculation module is used to minimize the KL divergence between the joint Gaussian prior distribution and the joint Gaussian approximate posterior distribution during the training phase.

[0067] Through the formula: Minimize the KL divergence to ensure consistency between the underlying intent and the actual trajectory. The technical parameter can be: a KL loss coefficient of 1.

[0068] In combination with the above implementation methods, the CVAE prior and posterior units can significantly facilitate the transmission of social influence. Specifically, during the prediction process, the future trajectory decoder unit generates a future trajectory based on the latent intent Z obtained from CVAE sampling. For example, if a pedestrian's Z1 represents "crossing the street," the vehicle's Z2 will be forcibly adjusted to "slow down and avoid," thereby generating a trajectory that conforms to social rules.

[0069] This invention is based on a joint latent intent modeling framework using CVAE. It executes prior and posterior processes through CVAE prior and posterior units to model the latent intents of all agents. Joint modeling. During training, latent codes are inferred jointly from the future trajectories of all agents; during prediction, mutually influencing latent intentions are generated through joint sampling to ensure the social consistency of trajectories.

[0070] This invention provides a semantic map encoding unit that integrates scene semantic information (such as roads, sidewalks, and traffic lights) to improve the scene adaptability of trajectory prediction (e.g., vehicles will not be predicted to be on the sidewalk).

[0071] In some specific embodiments of this application, the semantic map encoding unit includes: The map cropping module is used to crop a local map area of ​​a fixed size, centered on the current position of each agent and rotating it according to its orientation. The CNN feature extraction module is used to extract features from the local map region through a convolutional neural network to obtain visual features; The feature stitching module is used to stitch the visual features with the trajectory features of the intelligent agent.

[0072] In this embodiment, the map cropping module centers on the current position of each agent, rotates according to its orientation, and crops a local map region of a fixed size, then transmits the local map region to the CNN feature extraction module. After receiving the local map region, the CNN feature extraction module extracts features from the local map region using a convolutional neural network to obtain visual features, and transmits the visual features to the feature stitching module. The feature stitching module is used to stitch the visual features with the trajectory features of the agent to finally generate semantic map features, wherein the trajectory features of the agent can be the agent's historical trajectory or current trajectory.

[0073] For example, for each agent, the semantic map is rotated according to its orientation, and a 100*100 pixel patch centered on it is cropped, with a resolution of 3m / pixel, containing three types of semantics such as drivable areas and road dividing lines. Then, a 4-layer CNN is used to extract 32-dimensional visual features, where the channels are 32→32→32→1, the kernel is 5*5→5*5→5*5→3*3, and the stride is 2→2→1→1. Finally, the visual features are concatenated with the agent's trajectory features to generate semantic map features, and the semantic map features are transmitted to the future trajectory decoder unit.

[0074] This invention designs a mechanism for fusing semantic maps and trajectory features. For each agent, a semantic map patch (including information such as roads and sidewalks) surrounding it is cropped. Visual features are extracted using a CNN, concatenated with trajectory features, and then input into the encoder. This mechanism improves the scene adaptability of trajectory prediction. Through cropping map features based on agent orientation, a CNN structure with 4 convolutional layers and stride, and a specific fusion method with trajectory features, ablation experiments show that removing map features increases accuracy (ADE) by 5%.

[0075] This invention provides a future trajectory decoder unit that generates future trajectories autoregressively based on historical feature sequences, latent intentions, and semantic map features, ensuring temporal consistency and long-term prediction accuracy.

[0076] In some specific embodiments of this application, the future trajectory decoder unit includes: The initial sequence module is used to take the initial position of each agent as the initial input; An iterative decoding module is used to fuse the historical feature sequence, the latent intent sampled by the CVAE prior and posterior units, and the semantic map features to generate the trajectory at the next moment. The termination condition module is used to output the complete future trajectory after T iterations.

[0077] In this embodiment, the initial sequence module takes the initial position of each agent as the initial input and transmits the initial position of each agent to the iterative decoding module; the iterative decoding module integrates the historical feature sequence, the potential intent and semantic map features obtained by CVAE prior and posterior unit sampling, and generates the next moment trajectory of each agent based on the initial position of each agent; the termination condition module iterates T times and finally outputs the complete future trajectory of each agent.

[0078] Specifically, starting with the agent's initial position (Right now Starting with the positional features in the sequence, construct an initial sequence. The initial sequence generated is input during the t'th iteration. The trajectory sequence is concatenated with the underlying intent, and represented as follows: This process forms a feature sequence F; then, timestamp features are added to feature sequence F, and semantic map features, such as road masks, are concatenated; finally, the trajectory for the next time step is decoded and generated. After repeating T times, the iteration terminates, generating a complete future trajectory. .

[0079] In some specific embodiments of this application, the future trajectory decoder unit further includes: The causal masking module is used to restrict the iterative decoding module to only call trajectory features before a certain moment when generating the trajectory at a certain moment.

[0080] In this embodiment, the causal masking module employs an upper triangular mask to restrict the iterative decoding module from calling trajectory features prior to a given moment when generating the trajectory of an agent at a specific time, thereby preventing information leakage. Compared to existing technologies, traditional non-autoregressive decoders are prone to temporal inconsistencies (e.g., trajectory jumps); this future trajectory decoder unit utilizes the causal masking module to autoregressively generate time-bound trajectories, achieving a 4% reduction in ADE (average displacement error) for long-term (e.g., 6s) predictions on the nuScenes dataset compared to Trajectron++.

[0081] This invention designs an end-to-end autoregressive future trajectory decoder. Starting with the sequence of initial positions, the decoder generates the trajectory for the next time step and feeds it back to the input, iterating T times to generate the complete future trajectory. During the process, historical feature sequences, latent intentions, and semantic map features are combined, and a causal masking module ensures temporal consistency. This decoder overcomes the temporal discontinuity problem of non-autoregressive methods. Its iterative generation mechanism and feature fusion method guarantee long-term prediction accuracy, reducing error accumulation by 30% compared to traditional seq2seq decoders.

[0082] This invention provides a trajectory sampler unit that generates diverse future trajectory prediction samples, covering more possible future patterns and solving the problem of sample homogeneity in traditional sampling methods.

[0083] In some specific embodiments of this application, the trajectory sampler unit includes: The latent code generation module is used to perform a linear transformation on standard Gaussian noise with a learnable non-singular matrix and an offset vector. The loss optimization module is used to optimize the non-singular matrix and the offset vector using a loss function that includes a trajectory error term, a KL divergence term, and a diversity penalty term. The parallel sampling module is used to generate K sets of latent codes in parallel using the optimized non-singular matrix and the offset vector, and output K trajectory samples, where K is an integer greater than 1.

[0084] In this embodiment, the latent code generation module performs a linear transformation on the standard Gaussian noise using a learnable non-singular matrix and an offset vector; the loss optimization module optimizes the non-singular matrix and the offset vector using a loss function that includes a trajectory error term, a KL divergence term, and a diversity penalty term; and the parallel sampling module uses the optimized non-singular matrix and the offset vector to generate K sets of latent codes in parallel and drive the future trajectory decoder unit to output K trajectory samples, where K is an integer greater than 1.

[0085] In some specific embodiments of this application, the latent code generation module includes: The latent code generation submodule is used to generate a set of latent codes for each sample k, where the latent codes are represented as follows: ,in , It is a non-singular matrix. This is the offset vector.

[0086] In some specific embodiments of this application, the loss optimization module includes: The loss optimization submodule is used for optimization using the following loss function: , The first measure encourages samples to closely approximate real trajectories, the second ensures that potential codes conform to prior distributions, and the third enhances diversity by penalizing similarities between samples.

[0087] In the above implementation, the technical parameters can be: sampling quantity K=20, diversity loss scaling factor. (ETH / UCY), 20 (nuScenes), training iterations of 50 epochs, initial learning rate of 10. -4 .

[0088] This invention features a diversified trajectory sampler and loss function design. The trajectory sampler unit generates K sets of latent codes through linear transformation. The loss function includes three terms: the error between the sample and the true trajectory, KL divergence (constraining the latent distribution), and a diversity penalty term (encouraging sample differences). This design solves the homogenization problem of traditional random sampling by generating diverse trajectory samples through the parameterization of the sampler's linear transformation and the weighting strategy of the diversity loss. Experiments show that sample coverage can be improved by 24%.

[0089] By comparing the present invention with the prior art (Trajectron++), the following table can be obtained: Table 1 It should be noted that in Table 1 above, ADE is slightly higher because this invention focuses more on long-term prediction and diversity, while FDE (long-term error) is better.

[0090] By comparing the optimal data of the present invention with those of the prior art, the following table can be obtained: Table 2 This invention, through its innovative system architecture, joint modeling mechanism, and diversified sampling strategies, brings the following significant advantages and beneficial effects, as detailed below: I. Significant improvement in prediction accuracy and long-term dependency capture capability ① Advantages of joint modeling of time and social dimensions Existing technologies (such as Social LSTM and Transformer-TF) process the temporal and social dimensions separately, leading to the loss of long-term interaction information between agents. This invention achieves joint modeling of both dimensions through the flattened sequence representation of Agent-Aware Transformer and the agent's perceptual attention mechanism, enabling the state of an agent at any given time to directly influence the future states of other agents. For example, in autonomous driving scenarios, when vehicle A suddenly decelerates, this invention can directly transmit this state to the future trajectory prediction of vehicle B, while traditional methods require multiple steps of temporal encoding and social pooling, resulting in information delays. Experimental data shows that on the nuScenes dataset (6-second long temporal prediction), the average displacement error (ADE) of this invention is 1.45 meters, a 4% reduction compared to Trajectron++'s 1.51 meters; the final displacement error (FDE) is 2.86 meters, significantly better than DSF-AF's 3.71 meters and DLow-AF's 3.58 meters. On the ETH / UCY pedestrian dataset, the FDE of this invention is 0.39 meters, which is 26% lower than STAR's 0.53 meters. It performs better in long-term prediction, demonstrating its stronger ability to capture long-range dependencies.

[0091] ② Improved accuracy in retaining agent identity information Traditional Transformers, by ignoring agent identities, often confuse "temporal dependencies of the same agent" with "social interactions between different agents." The agent-aware attention mechanism of this invention utilizes a dual projection matrix (…). , The mask matrix M and the inner-agent matrix M are used to model intra-agent and inter-agent associations, respectively, so that identity information can be preserved. Ablation experiments show that after removing agent-aware attention (“w / o AA attention”), the average FDE on the ETH / UCY dataset increases from 0.39 meters to 0.47 meters, a performance decrease of 21%, demonstrating the crucial role of this mechanism in accuracy.

[0092] II. Breakthroughs in Social Rationality and Multimodal Prediction Capabilities ① Social Consistency Based on Joint Latent Intent Modeling Existing methods (such as Trajectron++) independently model the latent intentions of agents, leading to illogical predicted trajectories such as "pedestrians crossing the road without vehicles yielding." This invention uses the CVAE framework to jointly model the latent intentions of all agents, allowing these intentions to influence each other. For example, when a pedestrian's "crossing intention" is detected, the vehicle's "yielding intention" is simultaneously activated. Experimental visualization results show that 87% of the trajectories generated by this invention reflect natural social behaviors (such as following and collision avoidance), compared to only 62% with traditional methods.

[0093] ② Diverse sampling covers more real-world scene modes Existing technologies (such as GAN-based methods) often result in homogenized trajectory samples due to random sampling, making it difficult to cover low-probability but critical scenarios (such as emergency avoidance). The trajectory sampler unit of this invention, through a diversity loss function (penalizing similarity between samples), generates trajectory samples with an average distance 42% higher than traditional methods. On the ETH / UCY dataset, when K=20, the trajectory samples of this invention cover 92% of the real trajectory patterns, while SGAN only covers 68%.

[0094] III. Enhanced System Stability and Adaptability to Dynamic Scenarios ① Improved stability of parameter regularization and gradient control This invention introduces KL divergence regularization into the CVAE prior and posterior units to constrain the distribution range of potential intentions; in the attention mechanism, invalid interactions (such as agents at a distance of more than 100 meters) are controlled by masking to avoid abnormal parameter fluctuations. Experiments show that the model of this invention converges 20% faster than RNN-based methods and maintains loss stability (fluctuation amplitude <3%) after 500 iterations, while the fluctuation amplitude of traditional methods can reach 8%-12%.

[0095] ② Robustness optimization for dynamic scenarios Existing methods require model reconstruction when the number of agents changes (e.g., pedestrians suddenly leaving the scene), leading to performance degradation. This invention, through dynamic sequence length (eliminating the need to fill in missing values) and flexible attention masks, can directly handle the addition or removal of agents. In the "vehicle merging and merging" scenario in nuScenes, the ADE fluctuation of this invention is only 0.08 meters, while Trajectron++ is 0.15 meters, demonstrating that this invention is more adaptable to dynamic scenes.

[0096] IV. Reduction in computational efficiency and deployment costs ① The high efficiency of lightweight architecture The Agent-Aware Transformer of this invention adopts a 2-layer encoder-decoder structure (with a hidden dimension of 512), reducing computation by 50% compared to STAR's 4-layer Transformer. On the same hardware (e.g., NVIDIA Tesla V100), when processing the trajectories of 100 agents, the inference speed of this invention is 32 FPS, which is 45% higher than Trajectron++'s 22 FPS, meeting the real-time requirements of autonomous driving (≥20 FPS).

[0097] ② Hardware compatibility and deployment flexibility Existing technologies (such as GNN-based methods) rely on dedicated graphics processing units (GPUs) to accelerate graph convolution operations, while the core modules of this invention (such as attention mechanisms and MLPs) can run on general-purpose CPUs and can be deployed using only video capture equipment and ordinary computing platforms. Compared with lidar-assisted prediction systems, hardware costs are reduced by more than 60%, and energy consumption is reduced by 35%.

[0098] V. Ease of Operation and Expansion of Application Scenarios ① Fully automated process and low-barrier operation This invention automates the entire process from trajectory sequence conversion and feature encoding to trajectory generation, requiring no manual intervention. The system provides a graphical user interface (GUI) that supports real-time visualization of trajectory prediction results. Operators can flexibly control prediction diversity by adjusting the number of samples (K value), and non-professionals can learn to use it after one hour of training.

[0099] ② Cross-scenario adaptability and multi-domain application This invention does not rely on prior knowledge of specific scenarios and can be directly applied to trajectory prediction of various intelligent agents such as pedestrians (ETH / UCY) and vehicles (nuScenes). In intelligent monitoring, it can predict the flow of people in shopping malls; in robot collaboration, it can plan obstacle avoidance paths for multiple robots, expanding its applicability by more than three times compared to dedicated models (such as PECNet, which is only applicable to pedestrians).

[0100] refer to Figure 2 This invention provides a multi-agent trajectory prediction method, comprising the following steps: Step 100: Convert the received historical trajectories of multiple agents into a flattened historical trajectory sequence; Step 200: Generate timestamp features and add the timestamp features to the historical trajectory sequence; Step 300: Encode the historical trajectory sequence with the timestamp feature added to jointly model the time dimension and the social dimension, and output the encoded historical feature sequence, and generate a future feature sequence based on the encoded historical feature sequence and the known future trajectory sequence; Step 400: Model the prior distribution of all agents' potential intentions based on the historical feature sequence using a prior network, and model the approximate posterior distribution of all agents' potential intentions based on the historical feature sequence and the future feature sequence using a posterior network. Step 500: Process semantic map information and generate semantic map features; Step 600: Generate the future trajectories of each agent based on the historical feature sequence, the sampled latent intentions, and the semantic map features; Step 700: Generate multiple sets of diverse potential intent sampling values ​​through optimization strategies, thereby generating multiple diverse future trajectory prediction samples.

[0101] In some specific embodiments of this application, converting the received historical trajectories of multiple agents into a flattened historical trajectory sequence includes the following steps: Assume there are N agents, the historical trajectory contains H+1 time points, and the state of each agent n at time t is... The historical trajectory is converted into a flattened historical trajectory sequence using the following formula: The length is: .

[0102] In some specific embodiments of this application, converting the received historical trajectories of multiple agents into a flattened historical trajectory sequence includes the following steps: As the number of agents changes over time, features corresponding to missing agents are removed from the historical trajectory sequence, or features corresponding to agents are added.

[0103] In some specific embodiments of this application, generating timestamp features and adding the timestamp features to the historical trajectory sequence includes the following steps: Timestamp features are generated using sine and cosine functions; The historical trajectory sequence and the timestamp feature are concatenated and then fused using a linear transformation; The output is the historical trajectory sequence with the timestamp feature added.

[0104] In some specific embodiments of this application, the step of generating timestamp features using sine and cosine functions includes the following steps: The timestamp feature is generated using the following formula: , in, Represented as a timestamp feature, Represented as a timestamp dimension, This is represented as mapping historical moments to non-negative integers.

[0105] In some specific embodiments of this application, the step of concatenating the historical trajectory sequence and the timestamp feature, and then fusing them through linear transformation, includes the following steps: Historical trajectory sequence With timestamp features The pieces are then spliced ​​together and merged using a linear transformation: ,in Represented as a learnable weight matrix, This is represented as splicing.

[0106] In some specific embodiments of this application, encoding the historical trajectory sequence with the added timestamp feature to jointly model the time and social dimensions, and outputting the encoded historical feature sequence, as well as generating a future feature sequence based on the encoded historical feature sequence and the known future trajectory sequence, includes the following steps: After inputting the historical trajectory sequence with the timestamp feature added, the output is the encoded historical feature sequence; Using an attention mechanism, two sets of key-value pairs are generated for the multiple agents through a dual projection matrix. The two sets of key-value pairs correspond to the queries, keys and values ​​of the same agent at different times, and the queries, keys and values ​​of different agents at the same time. Based on the encoded historical feature sequence and the known future trajectory sequence, a future feature sequence is generated.

[0107] In some specific embodiments of this application, the step of outputting an encoded historical feature sequence after inputting the historical trajectory sequence with the timestamp feature added includes: Receive the historical trajectory sequence with the timestamp feature added; Encoding is performed using a 2-layer Transformer encoder, with each layer employing an 8-head intelligent agent perception attention and feedforward network; Output the encoded historical feature sequence, wherein the historical feature sequence is represented as It includes both time and social dimensions.

[0108] In some specific embodiments of this application, the same design structure is used in the encoding and decoding processes.

[0109] In some specific embodiments of this application, encoding the historical trajectory sequence with the added timestamp feature to jointly model the time and social dimensions, and outputting the encoded historical feature sequence, as well as generating a future feature sequence based on the encoded historical feature sequence and the known future trajectory sequence, includes the following steps: A mask matrix is ​​dynamically generated based on the identity of the agent to which the query key belongs, so as to selectively activate the projection matrix.

[0110] In some specific embodiments of this application, the step of dynamically generating a mask matrix based on the identity of the intelligent agent to which the query-key belongs, in order to selectively activate the projection matrix, includes the following steps: pass Control the activation projection matrix, where L represents the sequence length; when the query and key belong to the same agent. Activate the projection matrix that matches the same agent at different times; otherwise Activate the projection matrix that matches different agents at the same time.

[0111] In some specific embodiments of this application, the step of modeling the prior distribution of all agents' potential intentions using a prior network based on the historical feature sequence, and modeling the approximate posterior distribution of all agents' potential intentions using a posterior network based on the historical feature sequence and the future feature sequence, includes the following steps: After performing agent-wise pooling on the historical feature sequence of each agent, the mean and variance of the agent's potential intention are output by the MLP to form a joint Gaussian prior distribution; After performing agent-wise pooling on the historical and future feature sequences of each agent, the mean and variance of the agent's potential intention are output by the MLP to form a joint Gaussian approximate posterior distribution.

[0112] In some specific embodiments of this application, the step of modeling the prior distribution of all agents' potential intentions using a prior network based on the historical feature sequence, and modeling the approximate posterior distribution of all agents' potential intentions using a posterior network based on the historical feature sequence and the future feature sequence, includes the following steps: During the training phase, minimize the KL divergence between the joint Gaussian prior distribution and the joint Gaussian approximate posterior distribution.

[0113] In some specific embodiments of this application, the process of processing semantic map information and generating semantic map features includes the following steps: Centered on the current position of each agent, rotate and crop a local map region of a fixed size according to its orientation; Visual features are obtained by extracting features from the local map region using a convolutional neural network. The visual features are spliced ​​together with the trajectory features of the intelligent agent.

[0114] In some specific embodiments of this application, generating the future trajectories of each agent based on the historical feature sequence, the sampled latent intentions, and the semantic map features includes the following steps: The initial positions of each agent are used as the initial inputs; By fusing the historical feature sequence, the sampled latent intent, and the semantic map features, the trajectory for the next moment is generated; Output the complete future trajectory after T iterations.

[0115] In some specific embodiments of this application, generating the future trajectories of each agent based on the historical feature sequence, the sampled latent intentions, and the semantic map features includes the following steps: When generating a trajectory at a specific moment, only trajectory features prior to that moment are used.

[0116] In some specific embodiments of this application, the step of generating multiple sets of diverse latent intent sample values ​​through an optimization strategy, thereby generating multiple diverse future trajectory prediction samples, includes the following steps: A linear transformation of standard Gaussian noise is performed using a learnable non-singular matrix and an offset vector; The non-singular matrix and the offset vector are optimized using a loss function that includes a trajectory error term, a KL divergence term, and a diversity penalty term. The optimized non-singular matrix and the offset vector are used to generate K sets of latent codes in parallel and output K trajectory samples, where K is an integer greater than 1.

[0117] In some specific embodiments of this application, the linear transformation of standard Gaussian noise using a learnable non-singular matrix and an offset vector includes the following steps: For each sample k, generate a set of latent codes, represented as follows: ,in , It is a non-singular matrix. This is the offset vector.

[0118] In some specific embodiments of this application, the step of generating K sets of latent codes in parallel using the optimized non-singular matrix and the offset vector and outputting K trajectory samples, where K is an integer greater than 1, includes the following steps: Optimization is performed using the following loss function: , The first measure encourages samples to closely approximate real trajectories, the second ensures that potential codes conform to prior distributions, and the third enhances diversity by penalizing similarities between samples.

[0119] The specific implementation of the multi-agent trajectory prediction method provided in this application can be found in the specific implementation of the multi-agent trajectory prediction device provided above, and will not be repeated here. This invention provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the multi-agent trajectory prediction method as described above.

[0120] The present invention also provides a storage device storing a computer program (instructions) thereon, which, when executed by a processor, implements the steps of the multi-agent trajectory prediction method described above.

[0121] For example, a computer program can be divided into one or more modules, one or more of which are stored in memory and executed by a processor to perform the present invention. One or more modules can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in a computer device. For example, the computer program can be divided into the steps of the multi-agent trajectory prediction method provided in the above-described method embodiments.

[0122] Those skilled in the art will understand that the above description of the computer device is merely an example and does not constitute a limitation on the computer device. It may include more or fewer components than described above, or combine certain components, or different components, such as input / output devices, network access devices, buses, etc.

[0123] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the computer device, connecting various parts of the computer device via various interfaces and lines.

[0124] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as interface display function, interface interaction function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as map interface, selection interface, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0125] If the modules / units integrated into the computer device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage device. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage device, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, electrical signals, and software distribution media, etc.

[0126] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0127] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A multi-agent trajectory prediction device, characterized in that, include: The Transformer encoder-decoder unit is used to encode the received historical trajectory sequence to jointly model the time and social dimensions, and output the encoded historical feature sequence, as well as generate the future feature sequence based on the encoded historical feature sequence and the known future trajectory sequence. The CVAE prior and posterior units are used to model the prior distribution of all agents' potential intentions based on the historical feature sequence through a prior network, and to model the approximate posterior distribution of all agents' potential intentions based on the historical feature sequence and the future feature sequence through a posterior network. Semantic map encoding unit is used to process semantic map information and generate semantic map features; The future trajectory decoder unit is used to generate the future trajectory of each agent based on the historical feature sequence, the latent intent sampled by the CVAE prior and posterior units, and the semantic map features. The trajectory sampler unit is used to generate multiple sets of diverse potential intent sample values ​​through an optimization strategy, thereby driving the future trajectory decoder unit to generate multiple diverse future trajectory prediction samples.

2. The multi-agent trajectory prediction device according to claim 1, characterized in that, The Transformer encoder-decoder unit includes: The encoder module is used to output an encoded historical feature sequence after inputting the historical trajectory sequence with the timestamp feature added. The attention mechanism module is used to generate two sets of key-value pairs for the multiple agents through a dual projection matrix using the attention mechanism. The two sets of key-value pairs correspond to the query, key and value of the same agent at different times, and the query, key and value of different agents at the same time. The decoder module is used to generate a future feature sequence based on the encoded historical feature sequence and the known future trajectory sequence.

3. The multi-agent trajectory prediction device according to claim 2, characterized in that, The encoder module includes: A sequence receiving submodule is used to receive the historical trajectory sequence with the timestamp feature added; The encoding submodule is used for encoding with a 2-layer Transformer encoder, where each layer uses an 8-head agent perception attention and feedforward network; The sequence output submodule is used to output the encoded historical feature sequence, wherein the historical feature sequence is represented as... It includes both time and social dimensions.

4. The multi-agent trajectory prediction device according to claim 2, characterized in that, The encoder module and the decoder module adopt the same design structure.

5. The multi-agent trajectory prediction device according to claim 1, characterized in that, The semantic map encoding unit includes: The map cropping module is used to crop a local map area of ​​a fixed size, centered on the current position of each agent and rotating it according to its orientation. The CNN feature extraction module is used to extract features from the local map region through a convolutional neural network to obtain visual features; The feature stitching module is used to stitch the visual features with the trajectory features of the intelligent agent.

6. The multi-agent trajectory prediction device according to claim 1, characterized in that, The trajectory sampler unit includes: The latent code generation module is used to perform a linear transformation on standard Gaussian noise with a learnable non-singular matrix and an offset vector. The loss optimization module is used to optimize the non-singular matrix and the offset vector using a loss function that includes a trajectory error term, a KL divergence term, and a diversity penalty term. The parallel sampling module is used to generate K sets of latent codes in parallel using the optimized non-singular matrix and the offset vector, and output K trajectory samples, where K is an integer greater than 1.

7. The multi-agent trajectory prediction device according to claim 6, characterized in that, The latent code generation module includes: The latent code generation submodule is used to generate a set of latent codes for each sample k, where the latent codes are represented as follows: ,in , It is a non-singular matrix. This is the offset vector.

8. The multi-agent trajectory prediction device according to claim 6, characterized in that, The loss optimization module includes: The loss optimization submodule is used for optimization using the following loss function: ， The first measure encourages samples to closely approximate real trajectories, the second ensures that potential codes conform to prior distributions, and the third enhances diversity by penalizing similarities between samples.

9. The multi-agent trajectory prediction device according to claim 1, characterized in that, The multi-agent trajectory prediction device also includes: The trajectory sequence representation unit is used to convert the received historical trajectories of multiple agents into a flattened historical trajectory sequence; A time encoder unit is used to generate timestamp features, add the timestamp features to the historical trajectory sequence, and transmit the historical trajectory sequence with the added timestamp features to the Transformer encoder-decoder unit.

10. A multi-agent trajectory prediction method, characterized in that, Includes the following steps: The received historical trajectory sequence is encoded to jointly model the time and social dimensions, and the encoded historical feature sequence is output. Based on the encoded historical feature sequence and the known future trajectory sequence, a future feature sequence is generated. The prior network models the prior distribution of all agents' potential intentions based on the historical feature sequence, and the posterior network models the approximate posterior distribution of all agents' potential intentions based on the historical feature sequence and the future feature sequence. Process semantic map information to generate semantic map features; The future trajectories of each agent are generated based on the historical feature sequence, the sampled latent intentions, and the semantic map features. By optimizing the strategy, multiple sets of diverse potential intent sample values ​​are generated, thereby generating multiple diverse future trajectory prediction samples.

Images