Multi-agent trajectory generation method and apparatus

Abstract

The present disclosure provides a multi-agent trajectory generation method and device, the multi-agent trajectory generation method comprising: obtaining historical trajectory data of each participating entity in a multi-agent scene and semantic data describing a scene event, the historical trajectory data representing motion trajectory features, and the semantic data containing semantic information; detecting a kinematic feature mutation time of a target object in the historical trajectory data; correcting the timestamp of the semantic data based on a time anchor point determined at the kinematic feature mutation time, to achieve time domain alignment of the motion trajectory features and the semantic information; performing cross-modal feature fusion based on the aligned historical trajectory data and the semantic data, to construct conditional features with semantic intent constraints; and generating future motion trajectories of each participating entity that conform to the semantic intent constraints based on the conditional features.

Description

Technical Field

[0001] This disclosure relates to the field of multi-agent trajectory prediction technology, specifically to a method and apparatus for generating multi-agent trajectories. Background Technology

[0002] Multi-agent trajectory prediction and generation has significant theoretical and practical value in applications such as team sports tactical analysis, autonomous driving systems, and robot collaborative operations. Accurately predicting or generating the future trajectories of multiple participating entities plays a crucial role in improving tactical decision-making efficiency, proactively mitigating risks, and optimizing task planning.

[0003] Existing trajectory prediction methods primarily rely on statistical or deep learning modeling based on historical motion data, inferring future trajectories by analyzing an entity's historical position, velocity, and other kinematic characteristics. However, these methods have the following limitations:

[0004] 1) Trajectory data has temporal and spatial characteristics, while language description has discrete symbolic characteristics. The two have significant differences in representation, making it difficult to fuse and align multimodal data.

[0005] 2) Traditional trajectory prediction models typically output a single deterministic prediction result or can only generate a limited number of random variations. They are difficult to systematically explore the trajectory possibilities under different assumptions and cannot meet the needs of counterfactual reasoning and hypothetical scenario analysis. Therefore, the generation diversity of existing methods is limited.

[0006] 3) Existing methods lack the ability to understand and respond to high-level semantic intent. In practical applications, decision-makers often need to explore possible trajectory schemes based on specific tactical intentions, strategic goals, or scenario assumptions. For example, in team sports, coaches may want to know "how players should position themselves if a fast counter-attack tactic is adopted"; in autonomous driving scenarios, planning systems may need to generate trajectories with different styles, such as "conservative overtaking" or "aggressive lane changing." Existing methods struggle to translate these high-level intentions expressed in natural language into specific trajectory outputs. Summary of the Invention

[0007] One of the purposes of this disclosure is to provide a multi-agent trajectory generation method that can ensure that the generated trajectory accurately responds to semantic instructions in terms of physical actions.

[0008] According to one aspect of this disclosure, a multi-agent trajectory generation method includes: acquiring historical trajectory data of each participating entity in a multi-agent scenario and semantic data describing scenario events, wherein the historical trajectory data represents motion trajectory features and the semantic data contains semantic information; detecting the abrupt change in the kinematic features of the target object in the historical trajectory data; correcting the timestamp of the semantic data based on the time anchor point determined by the abrupt change in the kinematic features to achieve temporal alignment between the motion trajectory features and the semantic information; performing cross-modal feature fusion based on the aligned historical trajectory data and semantic data to construct conditional features with semantic intent constraints; and generating future motion trajectories of each participating entity that conform to the semantic intent constraints based on the conditional features.

[0009] Optionally, detecting the moment of abrupt change in the kinematic characteristics of the target object in the historical trajectory data may include: real-time monitoring of the rate of change of the acceleration, velocity vector direction, or angular velocity of the target object in the historical trajectory data; in response to the acceleration amplitude exceeding a preset threshold, or the instantaneous change in the velocity vector direction exceeding a preset angle, or the angular velocity undergoing abrupt change, identifying the corresponding time point as the moment of abrupt change in kinematic characteristics.

[0010] Optionally, based on the time anchor points determined at the moments of abrupt changes in kinematic features, the timestamps of the semantic data are corrected to achieve temporal alignment between motion trajectory features and semantic information. This may include: extracting local extreme points from the acceleration curves of historical trajectory data as candidate time anchor points; calculating the relative spatial distance between each participating entity and the target object in the scene at each candidate time anchor point; selecting candidate time anchor points whose relative spatial distance with the participating entities meets a preset minimum distance condition as the final precise time anchor points; and performing temporal translation of the semantic data based on the precise time anchor points to correct the timestamps of the semantic data.

[0011] Optionally, generating future motion trajectories of each participating entity that conform to semantic intent constraints based on conditional features may include: introducing random latent variables that characterize motion uncertainty; performing multimodal fusion of the random latent variables and conditional features to construct target generation features; and performing decoding processing based on the target generation features to generate multiple candidate future trajectory sequences with diversity under semantic intent constraints.

[0012] Optionally, multimodal fusion of random latent variables and conditional features may include: using transformations such as concatenation, weighted summation, multiplicative interaction, or mapping to a shared feature space to fuse random latent variables and conditional features along the feature dimension.

[0013] Optionally, decoding based on target generation features to generate multiple candidate future trajectory sequences with diversity under semantic intent constraints may include: mapping target generation features to a spatiotemporal state space in the future time domain, obtaining predicted state information of each participating entity at multiple future times through temporal recursion or parallel mapping; and reconstructing the continuous motion path of each participating entity under semantic intent constraints based on the predicted state information to form a candidate future trajectory sequence.

[0014] Optionally, cross-modal feature fusion is performed based on aligned historical trajectory data and semantic data to construct conditional features with semantic intent constraints. This may include: modeling the multi-agent scenario as a spatiotemporal association graph, where each participating entity is a node, the interaction relationship between participating entities is an edge, and semantic information is used as the global semantic attribute of the spatiotemporal association graph; and capturing the correlation between node features and global semantic attributes in the spatiotemporal association graph through encoding processing to obtain conditional features that integrate local interaction features and global intent features.

[0015] Optionally, generating future motion trajectories of each participating entity that conform to semantic intent constraints based on conditional features may include: obtaining the location information of the future time to be generated, and generating a future temporal domain location code based on the location information; using the future temporal domain location code as a query vector and interacting with the motion trajectory features represented by historical trajectory data to determine the cross-temporal contribution of historical time to different future time; and mapping the historical information contained in the conditional features to each future time based on the cross-temporal contribution to construct the future motion trajectory of each participating entity that evolves continuously in the time dimension.

[0016] Optionally, the multi-agent scenario can be a sports competition scenario or a robot collaboration scenario; the participating entities can include at least one of athletes, competition equipment, robots, and their operated objects; semantic data can be obtained by extracting text features from the narration audio, instruction text, or scenario annotation information of the multi-agent scenario.

[0017] According to another aspect of this disclosure, a multi-agent trajectory generation apparatus includes: a memory storing a program or instructions; and a processor that, when the program or instructions are executed by the processor, causes the processor to execute the multi-agent trajectory generation method described above.

[0018] The multi-agent trajectory generation method according to embodiments of this disclosure takes the historical trajectory sequence of multiple participant entities and natural language description as input, and achieves automatic generation of future trajectories that conform to language intent through cross-modal information fusion and spatiotemporal dependency modeling, supporting diverse outputs and counterfactual scenario reasoning.

[0019] The multi-agent trajectory generation method according to the embodiments of this disclosure can better reproduce the uncertainties in human behavior (such as feints in football and sudden turns in traffic), making the simulation results closer to the dynamic changes in the real world.

[0020] The multi-agent trajectory generation method according to the embodiments of this disclosure realizes natural language-controlled trajectory generation. Users can specify high-level semantic information such as tactical intentions and action styles through language descriptions, and generate trajectory schemes that conform to the intentions accordingly, thereby improving the naturalness and convenience of human-computer interaction.

[0021] The multi-agent trajectory generation method according to the embodiments of this disclosure effectively integrates trajectory dynamic features and linguistic semantic information, achieves deep alignment of spatiotemporal data and text data, and improves the semantic consistency of the generated results.

[0022] The multi-agent trajectory generation method according to the embodiments of this disclosure supports diverse trajectory generation and counterfactual scenario reasoning. Under the same language conditions, it can generate a variety of different trajectory schemes, meet the needs of hypothesis exploration and scheme comparison, and expand the application scope of trajectory prediction technology.

[0023] The multi-agent trajectory generation method according to the embodiments of this disclosure can simultaneously capture spatial interaction and temporal evolution patterns, and the generated trajectory has good spatiotemporal coherence and physical rationality.

[0024] The multi-agent trajectory generation method according to the embodiments of this disclosure has good versatility and can be applied to various multi-agent decision support scenarios such as team sports tactical analysis, autonomous driving scenario prediction, and multi-robot collaborative planning. Attached Figure Description

[0025] The above and other objects and features of exemplary embodiments of this disclosure will become clearer from the following description taken in conjunction with the accompanying drawings, which exemplarily illustrate the embodiments, wherein:

[0026] Figure 1 This is a flowchart illustrating a multi-agent trajectory generation method according to an embodiment of the present disclosure;

[0027] Figure 2 This is a schematic diagram illustrating data alignment methods according to embodiments of the present disclosure;

[0028] Figure 3 This is a flowchart illustrating a data alignment method according to an embodiment of the present disclosure;

[0029] Figure 4 This is a schematic diagram illustrating the overall flow and data preprocessing logic of a multi-agent trajectory generation method according to an embodiment of the present disclosure;

[0030] Figure 5 This is a flowchart illustrating a multi-agent trajectory generation method according to an embodiment of the present disclosure;

[0031] Figure 6 This is a schematic diagram illustrating a motion trajectory generation logic architecture based on cross-modal feature fusion according to an embodiment of the present disclosure. Detailed Implementation

[0032] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings, examples of which are illustrated in the drawings, wherein the same reference numerals always refer to the same parts. The embodiments will now be described with reference to the accompanying drawings in order to explain this disclosure.

[0033] The following detailed description is provided to aid in obtaining a full understanding of the methods, apparatus, and / or systems described herein. However, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein; equivalent substitutions or changes may be made, except for operations that must occur or be performed in a specific order. Furthermore, for clarity and conciseness, descriptions of content well-known in the art will be omitted or simplified.

[0034] Unless otherwise specified, the same reference numerals generally refer to the same elements (e.g., components, steps, and methods). Reference numerals described in previous embodiments that reappear in later embodiments may be omitted. Furthermore, technical features described in different or the same embodiments can be combined in any way, as long as the combined embodiment or technical solution is complete and can solve the technical problems of this application or achieve the technical effects described or not described in this disclosure but which can be determined based on the complete technical solution described above.

[0035] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains upon understanding this disclosure. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and in this disclosure, and shall not be interpreted in an idealized or overly formalistic manner.

[0036] This disclosure addresses the issue of significant discrepancies between trajectory and linguistic representation through a data preprocessing stage. Specifically, it utilizes kinematic features (such as acceleration extrema) in the tracking data as physical anchors to dynamically correct the timestamps of discrete event semantic data in time series. Through this physically driven calibration mechanism, continuous spatial coordinates and discrete symbolic features are mapped to a unified time reference, eliminating initial asynchronous biases for subsequent cross-modal coding.

[0037] In the conditional trajectory generation layer, this disclosure introduces random latent variables that represent motion uncertainty and interacts with fusion features in a multi-dimensional manner. This enables diverse outputs to be achieved by sampling the latent space under the same linguistic intent constraint, thereby allowing users to systematically simulate counterfactual scenario reasoning of "how the trajectory changes if the intent is fine-tuned", which meets the hypothesis analysis needs in complex decision-making.

[0038] Furthermore, this disclosure achieves a deep response to high-level tactical intentions through spatiotemporal graph construction and dependency modeling. The multi-agent scenario is modeled as an association graph, and the high-level semantics described by language are transformed into global intent attributes, thereby constraining the local interactions of each entity. Combined with cross-temporal dependency modeling, this ensures that the generated trajectory not only conforms to the logic of group collaboration in space but also exhibits a strong consistency in response to high-level strategic objectives (such as "rapid counterattack" or "conservative lane change") in time. The embodiments of this disclosure are described in detail below with reference to the accompanying drawings.

[0039] Figure 1 This is a flowchart illustrating a multi-agent trajectory generation method according to an embodiment of the present disclosure; Figure 2 This is a schematic diagram illustrating data alignment methods according to embodiments of the present disclosure; Figure 3 This is a flowchart illustrating a data alignment method according to an embodiment of the present disclosure; Figure 4 This is a schematic diagram illustrating the overall flow and data preprocessing logic of a multi-agent trajectory generation method according to an embodiment of the present disclosure; Figure 5 This is a flowchart illustrating a multi-agent trajectory generation method according to an embodiment of the present disclosure; Figure 6 This is a schematic diagram illustrating a motion trajectory generation logic architecture based on cross-modal feature fusion according to an embodiment of the present disclosure.

[0040] Reference Figure 1 The multi-agent trajectory generation method according to the embodiments of the present disclosure may include steps S100, S200, S300, S400 and S500.

[0041] In step S100, historical trajectory data and semantic data are acquired. For example, historical trajectory data of each participating entity in a multi-agent scenario and semantic data describing scenario events can be acquired. The historical trajectory data represents the characteristics of the motion trajectory, and the semantic data contains semantic information.

[0042] The multi-agent scenarios according to embodiments of this disclosure may include sports competition scenarios, robot collaboration, and road traffic scenarios. Sports competition scenarios may include ball games, such as those with strong competitiveness, rigorous tactical logic, and frequent interactions between entities (athletes and equipment). Road traffic scenarios may include autonomous driving simulation environments, such as those with high safety constraints, multi-target (vehicle and human) obstacle avoidance interactions, and complex traffic rule constraints.

[0043] The historical trajectory data here may include trajectory tracking data, which is spatial location recorded at fixed time intervals, continuous time series, and further calculated kinematic characteristics such as velocity and acceleration.

[0044] As an example, historical trajectory data can also be cleaned data, which may include interpolation and filtering. Semantic data can include event-labeled data, recording attribute information such as the type, time, entity, location, and result of key events. In the aforementioned multi-agent scenario, participating entities can include at least one of athletes, competition equipment, robots and their manipulated objects, motor vehicles, and pedestrians. Semantic data can be obtained by extracting textual features from the narration audio, instruction text, or scene annotation information of the multi-agent scenario.

[0045] Taking football applications as an example, historical trajectory data can refer to the coordinate sequence of each player and the ball on the field over the past few seconds (e.g., 10 seconds). For example, a sensor collects data 50 times per second, forming a continuous coordinate matrix. In the case of autonomous driving, historical trajectory data can refer to the three-dimensional coordinate sequence of the vehicle, surrounding vehicles, and pedestrians over the past few seconds (e.g., 5 seconds). This data is typically collected by onboard radar or cameras.

[0046] Historical trajectory features refer to high-order dynamic information derived mathematically from the original spatial coordinates, which characterizes the motion state and physical laws of an entity. Historical trajectory features can include not only basic displacements, but also calculated velocity vectors, acceleration amplitudes, and rates of change of heading angle (angular velocity). For example, when a player "shoots" or "brakes suddenly," the motion trajectory features will show local extrema of the acceleration curve. These physical abrupt changes provide crucial anchoring information for spatiotemporal alignment.

[0047] Semantic data refers to non-numerical information, existing in the form of natural language, symbols, or labels, used to describe the essence of scene events, tactical intentions, or decision-making objectives, giving logical meaning to physical motion. For example, semantic data could be the audio-visual text of a match commentator saying "the player makes a run down the wing and crosses," or the decision instruction in an autonomous driving system saying "take a safe detour at the next intersection." It is transformed into global intent attributes through text feature extraction technology, serving as constraints to guide the model in generating future paths that conform to specific logical objectives (such as tactical execution or risk avoidance).

[0048] To address the noise and missing data issues in trajectory tracking data, data preprocessing methods such as smoothing and interpolation can be used to process the collected data. For example, for short-term missing data points, linear interpolation or mean-based filling can be used to complete them; for high-frequency noise in continuous trajectories, sliding window smoothing or low-pass filtering methods can be used to suppress it, improving data stability while preserving the motion trend.

[0049] In step S200, the moment of abrupt change in the kinematic characteristics of the target object in the historical trajectory data is detected.

[0050] As an example, historical trajectory data (e.g., trajectory coordinates) can be processed by temporal differentiation to capture the instability point of physical motion state by monitoring the acceleration amplitude, velocity vector direction and rate of change of angular velocity of the target object in continuous time frames in real time.

[0051] As an example, when the instantaneous amplitude of acceleration exceeds the preset dynamic threshold, or the direction of the velocity vector deflects by more than a preset angle in a very short time, or the angular velocity exhibits a discontinuous abrupt change, these feature points can be preliminarily identified as candidate abrupt change moments of kinematic features.

[0052] To ensure the physical accuracy of capturing abrupt changes, local extrema extraction can be performed on the acceleration curve to lock the instantaneous point with the strongest dynamic response as a candidate time anchor.

[0053] As an example, a spatial constraint mechanism can be introduced. By calculating the relative spatial distances between each participating entity and the target object at these candidate anchor points, the extreme point that meets the preset minimum distance condition is selected as the final accurate moment. The above method combines a multi-dimensional detection approach, including dynamic threshold filtering, curve extreme value search, and spatial geometry verification, which can eliminate trajectory noise interference and accurately pinpoint the real instant of physical interaction such as hitting a ball, braking, or turning.

[0054] Therefore, it can monitor the acceleration, velocity vector direction, or angular velocity rate of change of target objects in historical trajectory data in real time. In response to the acceleration amplitude exceeding a preset threshold, the instantaneous change in velocity vector direction exceeding a preset angle, or a sudden change in angular velocity, the corresponding time point is identified as the moment of sudden change in kinematic characteristics.

[0055] However, the methods for detecting abrupt changes in kinematic characteristics disclosed herein are not limited to this. For example, energy / power consumption characteristics can also be monitored, and drastic changes in the force state of an entity can be captured by the spikes in motor current or instantaneous power. Spatiotemporal topological potential energy can also be monitored, and the extreme values ​​of the rate of change of relative distance between entities can be used to identify structural transitions in interactive relationships. Furthermore, trajectory frequency domain characteristics can be analyzed, and high-frequency fluctuations in curvature can be detected by waveform transformation to identify complex obstacle avoidance or change-of-direction actions.

[0056] In step S300, data alignment can be performed. For example, the timestamps of semantic data can be corrected based on the time anchors determined at the moments of abrupt changes in kinematic features, so as to achieve temporal alignment between motion trajectory features and semantic information.

[0057] To address the inconsistency in timestamps between trajectory tracking data and event annotation data, this disclosure presents a time anchor detection and alignment mechanism based on kinematic features.

[0058] As described above, local extreme points in the acceleration curve of the detectable target object (e.g., a soccer ball) can be used as candidate time anchor points. These extreme points typically correspond to the moment when the motion state changes abruptly due to external forces. Near each candidate anchor point, the spatial distance between the target object and each participating entity can be further calculated, and the moment with the smallest distance is selected as the precise occurrence time of the event. Through this alignment mechanism, the deviation between the event labeling time and the actual occurrence time can be effectively eliminated.

[0059] Reference Figure 3 The data alignment method according to the embodiments of this disclosure may include steps S310, S320, S330 and S340.

[0060] In step S310, local extreme points are extracted from the acceleration curve of historical trajectory data as candidate time anchor points.

[0061] Specifically, quadratic difference or numerical differentiation can be performed on the continuous historical trajectory coordinates to calculate the instantaneous acceleration amplitude of each sampling frame, and an acceleration curve evolving over time can be plotted. Subsequently, by using a sliding window or derivative zero-crossing detection method, peaks where the value changes from increasing to decreasing or troughs where it changes from decreasing to increasing can be found on the curve, i.e., local extrema. These extrema physically correspond to the instants when the force on the entity is most significant, such as the moment a football is struck or a vehicle brakes, and are thus locked as candidate time anchor points.

[0062] In step S320, the relative spatial distance between each participating entity and the target object in the scene at each candidate time anchor point is calculated.

[0063] After obtaining candidate time anchor points, the real-time spatial coordinates of the target object (e.g., a soccer ball) and all participating entities (e.g., players) in the scene at each anchor point can be extracted. Using the three-dimensional or two-dimensional Euclidean distance formula, the linear geometric distance between the target object and each participating entity can be calculated in parallel for each candidate time point, thereby constructing a set of relative spatial distances distributed with the candidate anchor points.

[0064] In step S330, candidate time anchor points whose relative spatial distance with the participating entities meets the preset minimum distance condition are selected as the final precise time anchor points.

[0065] The system can filter the set of relative spatial distances calculated at each candidate time anchor point. It compares the minimum distance between the target object and each participating entity at each candidate moment with a preset physical interaction distance threshold, retaining only anchor points whose spatial distance falls within the preset minimum range. This geometric constraint verification eliminates interference points caused by sensor noise, occlusion, or isolated actions from dynamic abrupt changes, thus determining the precise instant of actual physical interaction between the target object and participating entities. Precise time anchor points can be used as a time reference for synchronizing semantic data and trajectory data.

[0066] In step S340, the semantic data is shifted in the time domain based on the precise time anchor point in order to correct the timestamp of the semantic data.

[0067] Based on the determined precise time anchor point, its time offset from the original recorded time in the semantic data can be calculated, and the semantic data can be temporally shifted and adjusted based on this offset. By aligning the logical timestamps of semantic tags (such as event descriptions or tactical instructions) to the actual instant of physical motion mutation, the initial deviation caused by manual annotation delays or sensor asynchrony can be eliminated, thereby ensuring that the generated trajectory can accurately respond to semantic instructions in terms of physical actions.

[0068] For details, please refer to Figure 2 Extracting kinematic features of the target object from tracking data, and identifying instantaneous changes in motion state through acceleration extreme point detection, such as... Figure 2 At times 1022 and 1029, these were selected as candidate time anchor points. Subsequently, to eliminate interference from non-interactive factors, spatial distance detection was introduced to calculate the relative distance between the player with the ball and the ball at each candidate time. Only when the distance met a preset minimum threshold (…) Figure 2Only when the exact interaction points (times 1024 and 1027) are reached are the original times corrected to the accurate physical occurrence times; that is, 1022 is corrected to 1024 and 1029 to 1027. Then, the event data is time-domain shifted using this accurate time as a reference to eliminate annotation delay. In this way, the tracking trajectory and semantic events can be aligned with high precision in the temporal domain, providing a reliable consistency benchmark for subsequent feature fusion in the neural network.

[0069] In step S400, conditional features are constructed. For example, cross-modal feature fusion can be performed based on aligned historical trajectory data and semantic data to construct conditional features with semantic intent constraints.

[0070] As an example, neural networks (e.g., Spatiotemporal Graph Convolutional Networks (ST-GCN) or Long Short-Term Memory Networks (LSTM)) can be used to extract spatiotemporal motion features of multi-agent systems from aligned historical trajectory data, capturing their physical inertia and interactive dynamics. Additionally, pre-trained language models or natural language processing tools can be used to extract intent features from corrected semantic data, transforming tactical instructions, event descriptions, etc., into semantic vectors. Then, an attention mechanism is used to treat these semantic vectors as queries, matching and weighting them with spatiotemporal motion features to calculate the contribution of semantic information across different spatiotemporal dimensions.

[0071] However, this is merely an example. This disclosure can organize preprocessed multi-agent data into a spatiotemporal relational graph structure to facilitate subsequent feature extraction and relational modeling.

[0072] For example, a multi-agent scenario can be modeled as a spatiotemporal relational graph, where each participating entity is a node and the interaction relationship between participating entities is an edge. Semantic information is used as the global semantic attribute of the spatiotemporal relational graph. The correlation between node features and global semantic attributes in the spatiotemporal relational graph is captured through encoding processing to obtain conditional features that integrate local interaction features and global intent features.

[0073] Specifically, a spatiotemporal relationship graph can be defined as a six-tuple structure, including a set of nodes, a set of edges, a sequence of node features, a relation matrix, a time index, and a language description, wherein the set of nodes contains all participating entities and target objects in the scene.

[0074] As an example, in a team sports scenario, a node can include multiple athletes and the ball used in the game.

[0075] As an example, in an autonomous driving scenario, a node can include multiple vehicles and pedestrians.

[0076] As an example, in a robot collaboration scenario, a node may include multiple robots and the objects being operated on.

[0077] In addition, the edge set defines the relationships between nodes. It supports multiple edge construction strategies: the fully connected strategy connects all nodes pairwise, which is suitable for scenarios with a small number of entities and complex interaction relationships; the sparse connected strategy only establishes edges between node pairs that meet specific conditions (such as spatial distance thresholds, group relationships, etc.), which is suitable for scenarios with a large number of entities.

[0078] The node feature sequence records the dynamic attributes of each node over time. For each node, its position, velocity, and other features at each moment within the historical time window are extracted to form a feature matrix. The time length and sampling interval of the node feature sequence can be configured according to specific application requirements.

[0079] In addition, the relation matrix stores edge information between nodes in the form of an adjacency matrix, with matrix elements indicating whether a relationship exists between corresponding node pairs. The time index records the timestamp information corresponding to the node feature sequence, used for subsequent temporal coding and cross-temporal alignment. The language description is the global semantic attribute of the graph, derived from event annotation information, describing the semantic features of the current scene fragment. The language description may include information such as event type, participating entities, tactical intent, and action style.

[0080] The core function of encoding processing (e.g., cross-modal coding modules) is to map trajectory temporal features and linguistic semantic information to a unified representation space, forming a conditional representation that integrates dynamic features and semantic intent, which is used to guide subsequent trajectory generation.

[0081] As an example, encoding processing can be achieved through a timing encoder, a language encoder, and a cross-modal fusion unit.

[0082] A temporal encoder is responsible for encoding the node feature sequence and assigning temporal position codes to each moment in the time series to explicitly represent the temporal order information. The temporal position codes can use learnable embedding vectors or fixed functional codes (such as sine and cosine codes). The temporal position codes are fused with the node features to obtain node representations that contain temporal information.

[0083] The language encoder is responsible for encoding the natural language description, segmenting the language description into word sequences, mapping each word to a high-dimensional vector representation using a language embedding model, and aggregating the word vector sequences to obtain sentence-level semantic representation vectors. The language embedding model can be initialized using pre-trained language model parameters to leverage semantic knowledge learned from large-scale corpora.

[0084] The cross-modal fusion unit can jointly encode temporal and semantic representations. The temporal and semantic encoded vectors are concatenated or fused by attention to form a unified cross-modal conditional representation. This conditional representation simultaneously carries the dynamic features of the historical trajectory and the semantic intent of the linguistic description, providing an informational foundation for subsequent conditional trajectory generation.

[0085] Cross-modal conditional representations continue to participate in computation in subsequent network layers. Through interaction with hidden layer features, they permeate linguistic semantic information into the entire trajectory generation process, enabling language to controllably adjust the generation results.

[0086] Dependency modeling can capture the spatial interaction relationships and temporal evolution patterns between entities in multi-agent scenarios, providing structured contextual information for trajectory generation.

[0087] For example, spatial dependency modeling units can capture the interaction relationships between different entities at the same time. In multi-agent scenarios, the movements of each entity are not independent but are influenced by factors such as the position, speed, and intent of other entities. Based on a graph structure, the spatial dependency modeling unit exchanges information between nodes through a message passing mechanism, enabling the representation of each node to incorporate the features of its neighboring nodes. Spatial dependency modeling can be implemented using various methods such as graph neural networks and attention mechanisms; however, this disclosure is not limited to these, and spatial dependency modeling can employ various implementation methods capable of effectively capturing the interaction patterns between nodes.

[0088] Temporal dependency modeling can capture the state evolution patterns of the same entity at different times. The future state of an entity depends not only on the characteristics at the current moment, but also on its historical trajectory.

[0089] As an example, a temporal dependency modeling unit can be used to model the node feature sequence in the time dimension to extract the temporal evolution pattern. As an example, temporal dependency modeling can be implemented using various methods such as recurrent neural networks, temporal attention mechanisms, and temporal convolution. However, this disclosure is not limited to these methods, and temporal dependency modeling can be implemented using various methods that can effectively model dependencies in the time dimension.

[0090] By adopting a joint modeling strategy of spatial and temporal dependencies, information interaction between the spatial and temporal dimensions is carried out simultaneously in each layer of the network. The modeling results of the two dimensions are integrated through a fusion mechanism, which can adopt various methods such as weighted summation, gated fusion, and post-projection. The key is to adaptively balance the contributions of spatial and temporal dependencies.

[0091] In step S500, the future motion trajectories of each participating entity that conform to the semantic intent constraints are generated based on the conditional features.

[0092] As an example, a cross-temporal alignment mechanism projects conditional features containing historical dynamics and semantic intent to the future temporal domain. Since trajectory generation requires predicting future states, an information transmission channel between the past and the future can be established: First, information about the future moments to be generated is acquired and a future temporal domain position code is generated; then, this position code is used as a query vector and interacted with the conditional features to determine the cross-temporal contribution of historical features to different future moments; finally, based on this contribution, the effective information in the conditional features is mapped to each future moment to construct the future motion trajectory of each participating entity that evolves continuously in the time dimension.

[0093] Taking a football match scenario as an example, the aligned player movement trajectory can be fused with the semantic intent of "winger cutting inside and crossing" to construct conditional features that include physical inertia and tactical objectives. Since the encoder acquires the player's already performed actions (such as dribbling and accelerating), while the prediction task needs to generate movement patterns in the next few seconds, it is necessary to establish an information mapping from historical states to future moments. Specifically, the future time series points to be generated can be obtained first and converted into temporal position codes as "query" information to retrieve related motion cues from historical conditional features. For example, when predicting the trajectory of the football after it leaves the foot, the dynamic parameters of the "moment of contact" in the historical features can be calculated to have the highest contribution to that future moment. Based on the calculated cross-temporal contribution, the tactical intent and physical energy contained in the conditional features can be mapped onto the future timeline. This ensures that the generated trajectory not only physically continues the player's previous movement trend but also reflects semantic constraints in temporal evolution. For example, the striker's future trajectory will automatically accelerate towards the penalty area based on the projection of the intent to "cross," rather than blindly maintaining a straight line. Thus, the motion trajectories of each participating entity in the time dimension can be constructed as continuous, smooth, and consistent with tactical logic.

[0094] As described above, future trajectory sequences that conform to linguistic intent can be generated based on cross-modal conditional representations and dependency encoding features, and diverse generation methods can be supported to meet the needs of counterfactual reasoning and hypothesis exploration. For example, the conditional trajectory generation module can receive the hidden features and cross-modal conditional representations output by the dependency modeling module as input. The hidden features contain spatial and temporal dependency information of historical trajectories, while the cross-modal conditional representations contain the semantic intent of the linguistic description. The decoding network progressively decodes the hidden features into trajectory predictions for each entity in the future time series. The structure of the decoding network corresponds to that of the encoding network, using a multi-layer stacking approach to progressively reconstruct trajectory details. During the decoding process, the cross-modal conditional representation continuously participates in the computation, ensuring that the generated results are consistent with the linguistic intent.

[0095] As an example, a random sampling mechanism can be introduced to support the generation of diverse trajectories.

[0096] The steps for generating future motion trajectories of participating entities that conform to semantic intent constraints based on conditional features may include: introducing random latent variables that characterize motion uncertainty; performing multimodal fusion of the random latent variables and conditional features to construct target generation features; and performing decoding processing based on the target generation features to generate multiple candidate future trajectory sequences with diversity under the semantic intent constraints.

[0097] Based on deterministic coding features, additional sampleable random variables are introduced. By controlling the values ​​of these random variables, various trajectory schemes can be generated. This random sampling can be implemented using various techniques such as variational inference, diffusion processes, and adversarial generation. However, this disclosure is not limited to these methods; various approaches that can introduce controllable diversity while maintaining trajectory rationality can be employed.

[0098] In the process of multimodal fusion, transformation methods such as splicing, weighted summation, multiplicative interaction, or mapping to a shared feature space can be used to fuse random latent variables and conditional features in the feature dimension.

[0099] In one example, the target generation features can be mapped to the spatiotemporal state space of the future time domain, and the predicted state information of each participating entity at multiple future times can be obtained through temporal recursion or parallel mapping; based on the predicted state information, the continuous motion path of each participating entity under the semantic intent constraint can be reconstructed to form a candidate future trajectory sequence.

[0100] In a preferred embodiment, a variational autoencoder framework can be used to achieve diversity generation. The encoder maps input features to parameters of a latent distribution, samples latent variables from the latent distribution, and fuses these latent variables with deterministic features before inputting them into the decoder. By optimizing the variational lower bound objective function, the latent distribution is made to approximate a preset prior distribution, while maximizing the similarity between the generated trajectory and the real trajectory. During the inference phase, multiple samplings from the prior distribution can generate various trajectory schemes that conform to the language intent but have different specific paths. The output layer converts the decoded features into the final trajectory prediction result, including the position coordinates and velocity vectors of each entity at various future times. The time length and sampling interval of the output trajectory can be configured according to specific application requirements. The following will combine... Figure 4 The architecture and data flow of this disclosure are described in detail.

[0101] Reference Figure 4 The multi-agent trajectory generation method according to the embodiments of the present disclosure may include steps S410, S420, S430, S440, S450, S460 and S470.

[0102] In steps S410 and S420, raw tracking data and event data in the intelligent agent scenario can be obtained respectively. As mentioned above, the tracking data includes the high-frequency spatial coordinates of each participating entity in a continuous time series, and the event data may include semantic tags manually labeled or recorded by the system and corresponding timestamps, etc.

[0103] In step S430, the original tracking data and time data can be processed and analyzed. Specifically, firstly, the tracking data missing due to data loss is supplemented by interpolation methods, and noise in the trajectory is removed by filtering algorithms. Then, acceleration extreme point detection is performed, the possible time of event occurrence is identified based on the location of the extreme point, and anchor points are selected based on the spatial distance between entities in combination with the event data.

[0104] As an example, the dynamic characteristics of an agent's motion can be extracted by calculating the rate of change of velocity between adjacent coordinate points, especially for analyzing potential intervals where the direction or speed of motion changes drastically. Furthermore, interpolation techniques can be used to fill in trajectory breaks caused by sensor packet loss, ensuring the continuity and smoothness of the motion trajectory along the time axis. Candidate abrupt change points can be screened based on spatial distance constraints between multiple agents to determine precise physical interaction anchor points. Specifically, all candidate moments in the trajectory data can be traversed, and the Euclidean distance between the target object (e.g., a ball) and each participating entity (e.g., a player) can be calculated in real time. Only moments when the distance between entities is less than a preset threshold and is within a motion abrupt change interval are considered valid interaction points.

[0105] In step S440, tracking-event data alignment can be performed based on the determined physical anchor points to eliminate the temporal deviation between semantic labels and motion trajectories. As described above, by calculating the offset between the physical anchor points and the original event timestamps, a temporal translation mapping can be performed on the entire semantic data sequence, so that semantic commands such as "cut inside" and "shoot" are synchronized or substantially synchronized with their actual trajectory segments on the timeline.

[0106] In step S450, the aligned tracking data can be transformed into a structured spatiotemporal graph, from which topological features between agents can be extracted. As mentioned above, the coordinates and motion states of each participating entity are encoded as node features, and the interaction logic between entities is dynamically represented through edge relationships. Through deep analysis of the spatiotemporal graph, the implicit group cooperation patterns and spatial competition dynamics in the scene can be captured, elevating the physical trajectory into a high-order structured vector.

[0107] In step S460, textual information can be extracted from the semantic data and transformed into a semantic feature space compatible with spatiotemporal graph vectors. As mentioned above, a pre-trained language model or semantic encoding network can be used to transform discrete tactical instructions into continuous embedding vectors, extracting the intent and behavioral constraints contained in the instructions.

[0108] In step S470, the generated spatiotemporal graph vector and semantic vector can be input into a neural network (e.g., a conditional encoder-decoder network) to ultimately produce a future motion trajectory that conforms to semantic constraints.

[0109] As an example, the network can achieve deep interweaving of cross-modal features through a cross-attention mechanism, constructing conditional features with intent constraints, and projecting this information onto the prediction window using a cross-temporal alignment mechanism. The decoder performs autoregressive generation based on the projected features, outputting a future trajectory sequence that is highly consistent in both physical logic and tactical intent. The following section combines... Figure 5 The specific algorithm implementation of the multi-agent trajectory generation method disclosed herein is described in detail.

[0110] Reference Figure 5 The multi-agent trajectory generation method according to the embodiments of the present disclosure may include steps S510, S520, S530, S540, S550 and S560.

[0111] In step S510, historical trajectory data of the target intelligent agent can be acquired, and the historical trajectory data can be standardized. The trajectory data records the spatiotemporal coordinate sequence of the entity.

[0112] The aligned data can be used to construct a spatiotemporal graph, which transforms the physical location relationships of entities such as players and soccer balls into a topological network.

[0113] In step S520, semantic data describing scene events can be obtained, and the semantic data can be standardized. The semantic data contains preset tactical intentions or action tags. The semantic data can be text-encoded to transform discrete semantic tags into mathematical expressions in a high-dimensional vector space.

[0114] In step S530, the embedded spatiotemporal graph and semantic vector can be input into the cross-modal encoder to perform cross-modal encoding processing. The encoder calculates the correlation weight between historical trajectory features and semantic intent features through a cross-attention mechanism, identifying which motion states are strongly correlated with specific semantic descriptions.

[0115] In step S540, a spatiotemporal graph vector representing the dynamics of entity interactions can be extracted and generated based on the results of cross-modal coding. This vector not only preserves the spatial relative positions and movement trends between agents, but also incorporates weight information under semantic constraints. For example, under the intent of "passing," this vector will enhance the focus on potential receiving entities within the penalty area.

[0116] In step S550, text vectors representing the scene constraint logic can be generated simultaneously. These text vectors are a deep refinement of the original semantic intent. During cross-modal encoding, they incorporate feedback from the current physical environment, forming tactical guidance vectors with spatiotemporal awareness. These text vectors can guide the model in subsequent generation processes to make target-oriented corrections to the generated trajectories based on the actual distribution on the field, ensuring that the generated behavior conforms to the global intent without deviating from the current physical reality.

[0117] In step S560, the generated spatiotemporal graph vector and text vector can be fused to construct conditional features with semantic intent constraints. This fused feature can be projected onto the representation space of the future time axis using a cross-temporal alignment unit. The conditional features are then input into a conditional encoder-decoder network to generate the motion trajectories of each participating entity within the future time window. The decoder can output the spatial coordinate sequence of the entities frame-by-frame using an autoregressive mechanism based on the projected temporal features. Because the generated trajectories are constrained by both the aligned text vector and the spatiotemporal vector, they not only exhibit smooth displacement trajectories physically but also logically restore the initial semantic requirements, achieving high-fidelity multi-agent cooperative motion prediction.

[0118] Reference Figure 6 Taking a football match scenario as an example, tracking data of players dribbling the ball and event data of "passing the ball to XX" can be obtained. In the preprocessing stage, by detecting the extreme points of the football's acceleration and verifying its spatial distance from the player, the precise kicking moment (anchor point) is locked, perfectly aligning the "passing" intention with the physical action in the temporal domain. Next, a spatiotemporal graph reflecting the interaction between players can be constructed, and textual features of semantic instructions can be extracted. The cross-modal encoding process deeply integrates these two, generating tactically oriented conditional feature vectors. Finally, the conditional encoder-decoder network infers based on these features, and the generated future trajectory not only conforms to the physical laws of player running but also reflects the tactical intention: the football draws an arc to avoid the defense and flies towards a specific teammate, while other players also make reasonable overlapping movements, realizing intention-oriented multi-agent collaborative evolution.

[0119] The above has been referred to Figures 1 to 6A multi-agent trajectory generation method according to exemplary embodiments of the present disclosure is described. However, it should be understood that the devices, units, apparatuses, etc., shown in the figures can be configured as software, hardware, firmware, or any combination thereof to perform specific functions. For example, these units and devices may correspond to dedicated integrated circuits, pure software code, or modules combining software and hardware. Furthermore, one or more functions implemented by these systems or devices may also be uniformly executed by components in a physical entity device (e.g., a processor, client, or server).

[0120] The instructions stored in the aforementioned computer-readable storage medium can be executed in environments deployed in computer devices such as clients, hosts, agent devices, and servers. It should be noted that the instructions can also be used to perform additional steps beyond those described above, or to perform more specific processing while executing the aforementioned steps. The details of these additional steps and further processing are already provided in the reference... Figures 1 to 6 As mentioned in the description of the relevant systems and methods, they will not be repeated here to avoid repetition.

[0121] It should be noted that the process generation method according to the exemplary embodiments of this disclosure can rely entirely on the operation of computer programs or instructions to achieve the corresponding functions. That is, each device corresponds to each step in the functional architecture of the computer program, so that the entire system is called through a special software package (e.g., a lib library) to achieve the corresponding functions.

[0122] This disclosure provides a multi-agent trajectory generation device, which may include a memory and a processor. The memory stores programs or instructions, and when the programs or instructions are executed by the processor, the processor is prompted to execute the above-described multi-agent trajectory generation method.

[0123] This disclosure provides a computer-readable storage medium that can store a program or instructions that, when executed by a processor, cause the processor to perform the above-described multi-agent trajectory generation method.

[0124] The multi-agent trajectory generation method according to embodiments of the present disclosure can ensure that the generated trajectory accurately responds to semantic instructions in terms of physical actions.

[0125] The multi-agent trajectory generation method according to embodiments of this disclosure takes the historical trajectory sequence of multiple participant entities and natural language description as input, and achieves automatic generation of future trajectories that conform to language intent through cross-modal information fusion and spatiotemporal dependency modeling, supporting diverse outputs and counterfactual scenario reasoning.

[0126] The multi-agent trajectory generation method according to the embodiments of this disclosure can better reproduce the uncertainties in human behavior (such as feints in football and sudden turns in traffic), making the simulation results closer to the dynamic changes in the real world.

[0127] The multi-agent trajectory generation method according to the embodiments of this disclosure realizes natural language-controlled trajectory generation. Users can specify high-level semantic information such as tactical intentions and action styles through language descriptions, and generate trajectory schemes that conform to the intentions accordingly, thereby improving the naturalness and convenience of human-computer interaction.

[0128] The multi-agent trajectory generation method according to the embodiments of this disclosure supports diverse trajectory generation and counterfactual scenario reasoning. Under the same language conditions, it can generate a variety of different trajectory schemes, meet the needs of hypothesis exploration and scheme comparison, and expand the application scope of trajectory prediction technology.

[0129] The multi-agent trajectory generation method according to the embodiments of this disclosure can simultaneously capture spatial interaction and temporal evolution patterns, and the generated trajectory has good spatiotemporal coherence and physical rationality.

[0130] The multi-agent trajectory generation method according to the embodiments of this disclosure has good versatility and can be applied to various multi-agent decision support scenarios such as team sports tactical analysis, autonomous driving scenario prediction, and multi-robot collaborative planning.

[0131] While specific terminology has been used to describe various embodiments of this disclosure, the specification and drawings are to be regarded as illustrative rather than restrictive in order to aid in understanding this disclosure. Various modifications and changes can be made by those skilled in the art (e.g., different features in different embodiments may be combined) without departing from the broader spirit and scope of this disclosure. Therefore, the scope of this disclosure is not limited by the specific embodiments and examples, but by the claims and their equivalents.

Claims

1. A method for generating multi-agent trajectories, characterized in that, include: Acquire historical trajectory data of each participating entity in a multi-agent scenario and semantic data describing scenario events. The historical trajectory data represents motion trajectory features, and the semantic data contains semantic information. Detect the moment of abrupt change in the kinematic characteristics of the target object in the historical trajectory data; Based on the time anchor point determined by the abrupt change in the kinematic features, the timestamp of the semantic data is corrected to achieve temporal alignment between the motion trajectory features and the semantic information; Based on the aligned historical trajectory data and the semantic data, cross-modal feature fusion is performed to construct conditional features with semantic intent constraints; Based on the conditional features, the future motion trajectories of each participating entity that conform to the semantic intent constraints are generated. Based on the conditional features, the future motion trajectories of each participating entity that conform to the semantic intent constraints are generated, including: Introduce random latent variables to characterize the uncertainty of motion; The random latent variables and the conditional features are fused in a multimodal manner by means of splicing, weighted summation, multiplicative interaction, or mapping to a shared feature space, so as to construct the target generated features; Decoding is performed based on the target generation features to generate multiple candidate future trajectory sequences with diversity under the semantic intent constraints.

2. The multi-agent trajectory generation method according to claim 1, characterized in that, Detecting abrupt changes in the kinematic characteristics of the target object in the historical trajectory data includes: Real-time monitoring of the rate of change of the target object's acceleration, velocity vector direction, or angular velocity in the historical trajectory data; In response to the acceleration amplitude exceeding a preset threshold, or the instantaneous change in the velocity vector direction exceeding a preset angle, or the angular velocity undergoing a sudden change, the corresponding time point is identified as the moment of the sudden change in kinematic characteristics.

3. The multi-agent trajectory generation method according to claim 2, characterized in that, Based on the time anchor points determined by the abrupt change in the kinematic features, the timestamps of the semantic data are corrected to achieve temporal alignment between the motion trajectory features and the semantic information, including: Local extreme points are extracted from the acceleration curves of the historical trajectory data as candidate time anchors; Calculate the relative spatial distance between each participating entity and the target object within the scene at each of the candidate time anchor points; Candidate time anchor points whose relative spatial distance to participating entities meets the preset minimum distance condition are selected as the final precise time anchor points; The semantic data is shifted in the time domain based on the precise time anchor point to correct the timestamp of the semantic data.

4. The multi-agent trajectory generation method according to claim 1, characterized in that, Decoding is performed based on the target generation features to generate multiple candidate future trajectory sequences with diversity under the semantic intent constraints, including: The target generation features are mapped to the spatiotemporal state space of the future time domain, and the predicted state information of each participating entity at multiple future times is obtained through temporal recursion or parallel mapping. Based on the predicted state information, the continuous motion paths of each participating entity under the semantic intent constraint are reconstructed to form the candidate future trajectory sequence.

5. The multi-agent trajectory generation method according to claim 1, characterized in that, Based on the aligned historical trajectory data and the semantic data, cross-modal feature fusion is performed to construct conditional features with semantic intent constraints, including: The multi-agent scenario is modeled as a spatiotemporal relational graph, where each participating entity is a node, the interaction relationship between the participating entities is an edge, and the semantic information is used as the global semantic attribute of the spatiotemporal relational graph. The correlation between node features in the spatiotemporal association graph and the global semantic attributes is captured by encoding processing to obtain the conditional features that integrate local interaction features and global intent features.

6. The multi-agent trajectory generation method according to claim 1, characterized in that, Based on the conditional features, the future motion trajectories of each participating entity that conform to the semantic intent constraints are generated, including: Obtain the location information of the future time to be generated, and generate the future time-domain location code based on the location information; The future temporal location code is used as a query vector and interacted with the motion trajectory features represented by the historical trajectory data to determine the cross-temporal contribution of historical moments to different future moments. Based on the cross-temporal contribution, the historical information contained in the conditional features is mapped to various future moments to construct the future motion trajectory of each participating entity in the time dimension.

7. The multi-agent trajectory generation method according to any one of claims 1 to 6, characterized in that, The multi-agent scenarios are sports competition scenarios and robot collaboration scenarios; The participating entities include at least one of athletes, competition equipment, robots, and objects they operate on. The semantic data is obtained by extracting text features from the audio narration, instruction text, or scene annotation information of the multi-agent scene.

8. A multi-agent trajectory generation device, characterized in that, include: Memory, which stores programs or instructions; The processor, when the program or instructions are executed by the processor, causes the processor to perform the multi-agent trajectory generation method according to any one of claims 1-7.

Images