An agent trajectory prediction method, system, device and medium under a frequency domain perspective

Abstract

The application belongs to the field of agent trajectory prediction, and discloses an agent trajectory prediction method, system, device and medium from a frequency domain perspective, which comprises the following steps: constructing a hierarchical prediction network to predict the future trajectory of an agent from different frequency scales, wherein the hierarchical prediction network comprises two sub-networks: a coarse-grained key point estimation sub-network, which is used for predicting the "minimum" spectrum of the agent trajectory on a plurality of "key" frequency parts, i.e., the overall planning of the agent movement, and is used for predicting the key point trajectory spectrum of the agent at a lower space-time resolution, and the encoder-decoder structure is designed to generate a plurality of potential prediction trajectories to describe the uncertainty and randomness of the future behavior of the agent; and a fine-grained spectrum interpolation sub-network, which is used for reconstructing a complete trajectory spectrum from a higher space-time resolution according to the key point trajectory spectrum with the help of interaction characteristics. The application reconsiders the trajectory prediction from the frequency domain perspective, and reflects the behavior preferences of the agent at different scales from the coarse-grained global planning to the more fine-grained interaction preference description.

Description

Technical Field

[0001] This invention belongs to the field of intelligent agent trajectory prediction, and particularly relates to a method, system, device and medium for intelligent agent trajectory prediction from a frequency domain perspective. Background Technology

[0002] Currently, trajectory prediction aims to infer the possible future trajectories of agents by considering potential influencing factors. This is an important but challenging task with wide applications in behavior analysis, robot navigation, autonomous driving, object detection and tracking, and other tasks. Existing work primarily investigates the impact of interaction factors on trajectory prediction, including social interaction and physical interaction. Other works explore better ways to model trajectories, often using different neural networks to encode agent trajectories, such as Long Short-Term Memory Networks (LSTM), Graph Convolutional Networks (GCN), and Transformers.

[0003] Most existing methods treat trajectory prediction as a time-series generation task, progressively generating trajectory points. However, this fails to accurately reflect the agent's behavioral preferences at different scales. In other words, current work primarily focuses on the dynamic changes in agent behavior but lacks a hierarchical description across different time scales. For example, when an agent moves, it often first determines its destination (movement trend) and then adjusts the specific movement details (behavioral preferences) during the actual movement. Although some methods use neural networks with attention mechanisms (such as Transformers) as the backbone network to model the agent's motion state, they still struggle to directly describe the detailed differences in the agent's movement across different time scales.

[0004] The Fourier transform (FT) has achieved great success in signal processing, such as image denoising, edge detection, and image super-resolution. The Fourier transform decomposes a signal into a series of sine curves with different amplitudes and phases at different frequencies. Furthermore, these sine curves possess the characteristic of reflecting differentiated frequency responses at different frequency scales, which is difficult to obtain directly from the original signal.

[0005] Some works have applied Fourier transform to tasks such as trajectory prediction. However, they only model the interactions between agents using graph Fourier transform, and do not directly model trajectories using Fourier transform.

[0006] Based on the above analysis, the problems and shortcomings of the existing technology are as follows:

[0007] (1) Existing methods lack an overall description of the trajectory, making it difficult to accurately depict the motion trend of the agent.

[0008] (2) Existing hierarchical trajectory prediction methods focus more on whether the "destination" (i.e. the overall trend of the trajectory) can be predicted more accurately, while ignoring the fine modeling of interactive behavior.

[0009] (3) Existing methods often treat trajectory prediction as a time series generation task and generate trajectory points step by step. However, this is difficult to reflect the behavioral preferences of the agent at different scales in detail. Summary of the Invention

[0010] To address the problems existing in the prior art, this invention provides a method, system, device, and medium for predicting the trajectory of an intelligent agent from a frequency domain perspective.

[0011] This invention is implemented as follows: a method for predicting the trajectory of an intelligent agent from a frequency domain perspective, the method comprising:

[0012] A hierarchical prediction network is constructed to predict the future trajectory of an agent at different frequency scales. This hierarchical prediction network comprises two sub-networks:

[0013] The coarse-grained keypoint estimation subnetwork predicts the "minimum" spectrum of the agent's trajectory in several "key" frequency parts, i.e. the overall planning of the agent's motion, and is used to predict the keypoint trajectory spectrum of the agent at a lower spatiotemporal resolution. By designing an encoder-decoder structure, multiple potential predicted trajectories are generated to describe the uncertainty and randomness of the agent's future behavior.

[0014] A fine-grained spectral interpolation subnetwork is used to reconstruct the complete trajectory spectrum from the key point trajectory spectrum at a higher spatiotemporal resolution with the help of interactive representations.

[0015] Furthermore, the coarse-grained keypoint estimation subnetwork includes:

[0016] Step 1: Perform Discrete Fourier Transform (DFT) on the agent's observation trajectory to obtain the agent's observation time [1,2,…,t]. h The trajectory spectrum of ];

[0017] Step 2: Integrate the use of multilayer perceptron (MLP) t Multilayer Perceptron II (MLP) i Obtain random noise features f i and high-dimensional features f t splicing together f e f_e represents the embedding vector;

[0018] Step 3, use keypoint Transformer T k (Long Short-Term Memory (LSTM) or Temporal Convolutional Network (TCN) can also be used here, as Transformer is better at modeling long-range relationships.) For the f... e Encode;

[0019] Step 4, use the decoder MLP e Predict the trajectory spectrum of N key points of the intelligent agent [A] key ,Φ key ];

[0020] Step 5: Optimize the coarse-grained keypoint estimation subnetwork by minimizing the average Euclidean distance between the true spatial coordinates and the predicted spatial coordinates of the keypoints, thereby predicting the trajectory spectrum of the keypoints.

[0021] Furthermore, the observation trajectory of the intelligent agent is as follows:

[0022] make Let {I} represent the two-dimensional coordinates of an agent at time t. Given a video segment {I}, where {I} contains M agents at times [1, 2, ..., t], let {I} be the coordinates of an agent at time [1, 2, ..., t]. h ] observation trajectory X i This represents the observation trajectory of the i-th agent, used to predict the future time interval [t] of the M agents. h +1,t h +2,…,t h +t f Potential trajectory coordinates

[0023] Furthermore, the specific process of step one is as follows:

[0024] Observation trajectory 1D-DFT is applied to each dimension to obtain the observed trajectory spectrum. Including amplitude spectrum A = {a x a y} and phase spectrum Φ={φ x , φ y}:

[0025]

[0026] Here, DFT stands for Discrete Fourier Transform. 1D-DFT: One-dimensional Discrete Fourier Transform.

[0027] Furthermore, the specific process of step two is as follows:

[0028] First, for random noise vectors Sampling is performed; then, a multilayer perceptron (MLP) is used. i Encoding the noise variable z yields the random noise feature f i ;

[0029] Using a multilayer perceptron (MLP) t Observation trajectory spectrum of the coded agent Obtain high-dimensional features f t ;

[0030] Finally, the f t With the f i The concatenation yields the embedding vector f. e The formula is:

[0031]

[0032] Among them, f e The expression [a,b] represents the concatenation of vectors {a,b} on the last dimension. To illustrate that the operation [,] represents concatenation, here a and b represent two vectors, similar to an example.

[0033] Furthermore, the specific process of step three is as follows:

[0034] First embed the vector f e It is passed to the Transformer encoder, which will observe the trajectory spectrum (a x ,a y ,φ x ,φ y The input is fed into the Transformer decoder; the Transformer encoder is used as a feature extractor and does not contain a final output layer.

[0035] Then a multilayer perceptron 3MLP was used. e By aggregating the features of nodes at different frequencies, the behavioral feature f is obtained, expressed as:

[0036]

[0037] Furthermore, the specific process of step four is as follows:

[0038] The trajectory spectrum [A] key ,Φ key The expression for ] is:

[0039]

[0040] Where, N key This indicates the number of key points in the trajectory spectrum.

[0041] Furthermore, the specific process of step five is as follows:

[0042] When training the coarse-grained subnetwork, the agent's real keypoint spatial coordinates y key As a label, the y key Obtained from the complete ground truth of the future trajectory; simultaneously, the spatial coordinates of the predicted key points are obtained using the inverse discrete Fourier transform (IDFT).

[0043] By minimizing the true y key and prediction The average Euclidean distance between them is used to optimize the subnetwork, thereby learning to predict the trajectory spectrum of the corresponding key points;

[0044] The average Euclidean distance is the average keypoint loss (L). AKL The expression is:

[0045]

[0046] in,

[0047]

[0048] {\hat{Y}}_{i=1}^m represents the predicted trajectories of M agents. Here, Y represents the trajectories of the M agents, and y represents the trajectory of a single agent. If Y has \hat, it indicates a predicted trajectory; if Y does not have \hat, it indicates the ground truth of the trajectory.

[0049] p is short for position, representing the coordinate point (x, y), and t_1,...,t_{N_key} represents the time of the key point.

[0050] Here, IDFT represents the inverse discrete Fourier transform function.

[0051] Furthermore, the fine-grained spectral interpolation sub-network includes:

[0052] First, the trajectory spectrum of the key points [A] key , φ key Input, obtain representation features :

[0053]

[0054] Among them, the MLP here t MLP in coarse-grained networks t Different sharing parameters, where N key <t f N keyIndicates the number of key points; t f This indicates the length of the predicted time period.

[0055] Then, the interaction representation C is obtained first, as follows:

[0056] Let I denote the scene image at time t. t In the subsequent model derivation process, the trajectory coordinates of the agent need to be normalized to a unified unit. The trajectory coordinates of the agent can be represented by world coordinates, pixel coordinates, or grid coordinates, where (x, y) is the world coordinate (in meters), (p...) is the pixel coordinate, (x, y) is the grid coordinate, (x, y) is the world coordinate (in meters), (p...) is the pixel coordinate, (x, y ... x ,p y ) are coordinates in pixels (in pixels), (g x ,g y The coordinates are grid coordinates (unit: 1). When the agent's trajectory coordinates are encoded to form a migration image, the unit on the image is pixels. Before encoding, the trajectory coordinates need to be converted to pixel coordinates. There is a mapping relationship between world coordinates and pixel coordinates: (p...) x ,p y ) = m(x,y), where m represents the conversion factor between world coordinates and pixel coordinates.

[0057] Through two independent sub-modules, the observation trajectory and scene image are transferred to a focused semantically aligned interactive representation C.

[0058] The physical transfer submodule uses all available trajectories in a given scene as supervision information during training on the dataset, thereby establishing a connection between scene images and agent activities. First, a kernel density estimation method is used to obtain the trajectory probability density (in pixels) in a given dataset, as follows:

[0059]

[0060] Where R represents the number of location coordinates of the agent recorded in the dataset, h represents the bandwidth of the kernel function, (x,y) represents the world coordinates, K(x,y) represents the kernel density function, and (p x i ,p y i ) represents the pixel coordinates of the corresponding i in the dataset (this should be strictly distinguished from agent i; here, i only represents the number i). To accelerate computation, in this invention, we take...

[0061] Secondly, using the mapping function (g) x ,g y ) = m p→g (p x ,p yThe scene image is divided into H×W grids to obtain the supervision label l[I(g) for the activity semantics of each grid. x ,g y )],as follows:

[0062]

[0063] Where, D={(x,y)|m p→g (x,y)=(g x ,g y )} represents the set of pixel coordinates covered by the grid, m p→g This represents the transformation factor between grid coordinates and pixel coordinates; therefore, the supervision label l[I(g) of activity semantics can be used when training the model. x ,g y )] and scene image I(g x ,g y ).

[0064] Then, a trainable convolutional neural network (denoted as G(·)) is introduced to generate the corresponding gridded agent activity probability distribution. Where R H×W The scene image is represented by a mesh, as shown below:

[0065]

[0066] In this invention, the activity semantic tag l[I(g) is minimized. x ,g y [ ] and the agent activity probability prediction given by a trainable convolutional neural network Pixel-level error. It ensures that the network relies solely on scene images during testing. Directly obtain the semantics of the activity

[0067] Social Migration Submodule: Given grid E i ∈R H×W The trajectory of agent i is X i The trajectory of the surrounding agents (denoted as J, where J represents the set of all agents surrounding i) is... Define the social interaction energy E of agent i i The energy function f is as follows:

[0068]

[0069] Among them, E i Let f represent the energy of social interaction relationships of agent i, and let f represent the energy function. It is the initial energy function, j represents the agents surrounding i, and X is the initial energy function. iThis represents the trajectory of agent i. Represents the trajectory of j ( The superscript " / i" indicates the surrounding agents other than i), (g x ,g y ) represents the grid coordinates of i. Let θ represent the grid coordinates of j, P0 be the prior predictor (outputting grid coordinates), and θ be the grid coordinates of j. ij Let {h1,h2,h3} represent the relative energy growth factor between agents i and j, {h1,h2,h3} be the bandwidth of the energy function f, and {λ1,λ2,λ3} be the hyperparameters.

[0070] Social interaction energy E i It consists of three sub-energys, the first sub-energy This represents the potential intention of agent i, i.e., its possible future destination, where, This represents the interaction between agent i and itself; the second term is sub-energy. Describe the interaction between agent i and other surrounding agents J (J represents the set of all surrounding agents i), let... Let represent the displacement direction of agent i during the observation time. Therefore, the relative energy growth factor can be expressed as follows:

[0071]

[0072] Third Sub-Energy Determine social intimacy (safe social distance) to limit the activities of agent i.

[0073] Therefore, this scheme, from the perspective of energy, ε={E i This represents social interaction relationships, rather than historical trajectories.

[0074] Contextual features: The physical transfer submodule obtains activity semantics from scene images. The social migration submodule, viewed from an energy perspective, ε={E i Describes social interaction relationships. Transfer imagery (a fusion representation of social and contextual interactions). If C represents the collaborative impact of social and contextual interaction behaviors on the future trajectory planning of an intelligent agent, then... i Defined as:

[0075]

[0076] Using MLP c The interaction representation C is encoded by using a CNN to encode the scene visual images and trajectories in the form of an energy map, which encodes social and physical interactions. The interaction representation C comes from the model CSCNet.

[0077] f c =MLP c (C)

[0078] fc represents the features extracted after the interaction representation is processed by MLPc.

[0079] Secondly, use interpolation Transformer T i To learn the trajectory spectrum deviation and then reconstruct the complete trajectory spectrum, the spliced ​​features are then used. Passed to T i The encoder, and the key point trajectory spectrum Passed to T i The decoder; T i Used to predict the complete trajectory spectrum Finally, the reconstructed trajectory was obtained using IDFT. :

[0080]

[0081] Therefore, the potential predicted trajectory is represented as follows:

[0082]

[0083] Among them, [t h :,:] represents the tensor slicing operation; th represents the length of the observation period;

[0084] tf represents the length of the prediction period.

[0085] The slicing operation means that the trajectory I finally reconstruct is the trajectory within the time interval th+tf, but I only need the prediction estimate within the prediction time interval of length tf [th+1,th+2,...,th+tf]. So it is equivalent to cutting off the trajectory within the first th length and only keeping the trajectory within the second tf time interval.

[0086] Finally, the fine-grained spectral interpolation subnetwork uses an interpolation Transformer to interpolate the trajectory spectrum of the keypoints into a complete trajectory spectrum, and then interpolates the remaining spectral portion (the spectrum remaining after removing the trajectory spectrum of the keypoints from the complete trajectory spectrum).

[0087] This is used to reflect more refined interaction details of the intelligent agent;

[0088] By using Average Point-wise Loss (L APL Optimize fine-grained subnetworks.

[0089]

[0090] in, y represents the predicted trajectory and the ground truth of the predicted trajectory, respectively. p t Let represent the predicted trajectory coordinates at time t and the true value of the predicted trajectory coordinates, respectively.

[0091] Furthermore, the method for predicting agent trajectories from a frequency domain perspective also includes:

[0092] The hierarchical prediction network is trained using a joint loss function L, which is:

[0093]

[0094] Another object of the present invention is to provide a frequency domain perspective agent trajectory prediction system for implementing the aforementioned frequency domain perspective agent trajectory prediction method, the frequency domain perspective agent trajectory prediction system comprising:

[0095] The coarse-grained keypoint estimation module is used to predict the keypoint trajectory spectrum of an agent at a lower spatiotemporal resolution. It generates multiple potential predicted trajectories by designing an encoder-decoder structure to describe the uncertainty and randomness of the agent's future behavior.

[0096] A fine-grained spectral interpolation module is used to reconstruct the complete trajectory spectrum from the key point trajectory spectrum at a higher spatiotemporal resolution with the help of interactive representation.

[0097] Another object of the present invention is to provide a computer device including a memory and a processor, the memory storing a computer program, which, when executed by the processor, causes the processor to perform the steps of the intelligent agent trajectory prediction method in the frequency domain perspective.

[0098] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the agent trajectory prediction method in the frequency domain.

[0099] Another objective of this invention is to provide an information data processing terminal for implementing the intelligent agent trajectory prediction system from the frequency domain perspective.

[0100] Based on the above technical solutions and the technical problems solved, please analyze the advantages and positive effects of the technical solution to be protected by this invention from the following aspects:

[0101] This invention provides a hierarchical prediction of an agent's future trajectory at different frequency scales, including: (1) overall planning, i.e., the agent's coarse movement trend. The low-frequency part (slowly changing part) of the trajectory spectrum will reflect the agent's overall movement plan. (2) interaction preferences, i.e., the agent's interaction relationships. The high-frequency part (rapidly changing part) of the trajectory spectrum will describe the agent's rapidly changing movement differences, thereby further characterizing its personalized interaction preferences. This allows for more accurate trajectory prediction from different dimensions, from coarse-grained global planning to more fine-grained description of interaction preferences.

[0102] This invention re-examines trajectory prediction from a frequency domain perspective, moving from coarse-grained global planning to more fine-grained descriptions of interaction preferences, thus providing a more comprehensive and three-dimensional reflection of the agent's behavioral preferences at different scales.

[0103] Most existing methods treat trajectories as time series, thus typically processing them in the time domain. However, to be precise, a trajectory is a spatiotemporal sequence, related not only to time but also to spatial location. Common time series, such as stock prices, only contain time variables and no spatial variables. Therefore, processing trajectories solely in the time domain is inadequate; modeling trajectories in the frequency domain yields a better representation of them.

[0104] In image processing, Fourier transform is used to process Lena.jpg. This invention discovers that the low-frequency components represent the image contours, while the high-frequency components represent the image details (abrupt changes). Similarly, in trajectory analysis, this invention describes the trajectory through its high and low frequencies. Further research can be conducted to explore how high and low frequencies influence the trajectory, thereby enabling controllable adaptive prediction of diverse trajectories based on scene differences.

[0105] The expected benefits and commercial value of the technical solution of this invention after transformation are as follows: It can provide a new approach to the partial occlusion problem in target detection and tracking, proposing a prediction-based detection or tracking algorithm. The target's position (coordinates) is obtained based on known detection / tracking algorithms, and then the prediction algorithm of this invention calculates the target's possible future position (coordinates) after entering occlusion. In behavior analysis, it can provide more reasonable or refined trajectories to meet sensory needs, further making character behavior in the metaverse / VR / AR more realistic.

[0106] Does the technical solution of this invention overcome technical bias? It overcomes the previous method of modeling trajectories only in the time domain. This invention processes trajectories in the frequency domain, achieving more accurate results and more reasonable predicted trajectories. Attached Figure Description

[0107] Figure 1 This is a diagram of the overall architecture for trajectory prediction provided in an embodiment of the present invention.

[0108] Figure 2 (a)---(l) shows that the method provided by the embodiments of the present invention can meet 12 different scenarios. In particular, in scenario h, it was found that the predicted trajectory given by the method of the present invention conforms to traffic rules and common sense, and the predicted trajectory all revolves around the roundabout. Detailed Implementation

[0109] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0110] To enable those skilled in the art to fully understand how the present invention is specifically implemented, this section provides an explanatory description of the embodiments that expand upon the technical solutions of the claims.

[0111] like Figure 1 As shown, the frequency domain perspective method for predicting agent trajectories provided in this embodiment of the invention includes:

[0112] A hierarchical prediction network is constructed to predict the future trajectory of an agent at different frequency scales. This hierarchical prediction network comprises two sub-networks:

[0113] The coarse-grained keypoint estimation subnetwork predicts the "minimum" spectrum of the agent's trajectory in several "key" frequency parts, i.e. the overall planning of the agent's motion, and is used to predict the keypoint trajectory spectrum of the agent at a lower spatiotemporal resolution. By designing an encoder-decoder structure, multiple potential predicted trajectories are generated to describe the uncertainty and randomness of the agent's future behavior.

[0114] A fine-grained spectral interpolation subnetwork is used to reconstruct the complete trajectory spectrum from the key point trajectory spectrum at a higher spatiotemporal resolution with the help of interactive representations.

[0115] Furthermore, the coarse-grained keypoint estimation subnetwork includes:

[0116] Step 1: First, use Discrete Fourier Transform (DFT) on the agent's observation trajectory to obtain the agent's observation time [1,2,…,t]. h The trajectory spectrum of ];

[0117] Step 2: Use a multilayer perceptron (MLP) t The observation trajectory spectrum of the intelligent agent Embedded into high-dimensional feature f t middle;

[0118] Step 3, use keypoint Transformer T k For f e Encode;

[0119] Step 4, use the decoder MLP e To predict N key points (N) of the agent key <t f The trajectory spectrum of ) [A key , φ key ];

[0120] Step 5: Minimize the average Euclidean distance between the true spatial coordinates and the predicted spatial coordinates of the key points to optimize the coarse-grained key point estimation subnetwork, thereby learning to predict the trajectory spectrum of the key points.

[0121] Furthermore, the observation trajectory of the intelligent agent is as follows:

[0122] make Let {I} represent the two-dimensional coordinates of an agent at time t. Given a video segment {I}, where {I} contains M agents at times [1, 2, ..., t], ... h ] observation trajectory (representing the observation trajectory of the i-th agent), trajectory prediction aims to predict the agent's observation trajectory in the future time period [t]. h +1,t h +2,…,t h +t f Potential trajectory coordinates

[0123] Furthermore, the specific process of step one is as follows:

[0124] Observation trajectory Apply 1D-DFT to each dimension to obtain their spectrum, including the amplitude spectrum A = {a x ,a y} and phase spectrum Φ={φ x , φ y}:

[0125]

[0126] Furthermore, the specific process of step two is as follows:

[0127] First, for random noise vectors Sampling is performed; then, another multilayer perceptron (MLP) is used. i Encoding the noise variable z yields the random noise feature f i Finally, f t with f i splicing to obtain f e The formula is:

[0128]

[0129] Among them, f eThe diversity of the agent's future trajectory is represented by [a,b], which represents the concatenation of vectors {a,b} on the last dimension.

[0130] The specific process of step three is as follows:

[0131] Embedded vector f e It is passed to the Transformer encoder to observe the trajectory spectrum (a x a y , φ x , φ y The input is fed into the Transformer decoder; the Transformer here acts as a feature extractor and does not contain a final output layer; another multilayer perceptron (MLP) is used. e By aggregating the features of nodes at different frequencies, behavioral features f can be inferred, expressed as:

[0132]

[0133] Furthermore, the specific process of step four is as follows:

[0134] The trajectory spectrum [A] key Φ key The expression for ] is:

[0135]

[0136] Where, N key Indicates the number of key points in the trajectory spectrum;

[0137] Furthermore, the specific process of step five is as follows:

[0138] When training the coarse-grained subnetwork, the keypoint spatial coordinates y of the agent are... key (Taken from the complete future trajectory ground truth) as labels; simultaneously, the key spatial coordinates of the predicted key points will be obtained using the inverse discrete Fourier transform (IDFT). This will be achieved by minimizing the true y. key and prediction The average Euclidean distance between them is used to optimize the subnetwork, thereby learning to predict the trajectory spectrum of the corresponding key points;

[0139] The average Euclidean distance is the average keypoint loss (L). AKL The expression is:

[0140]

[0141] in,

[0142]

[0143] Here, IDFT represents the inverse discrete Fourier transform function.

[0144] Furthermore, the fine-grained spectral interpolation sub-network includes:

[0145] First, input the trajectory spectrum of the key points [A] key Φ key Obtaining representation :

[0146]

[0147] Among them, the MLP here t MLP in coarse-grained networks t Different parameters;

[0148] Then, use MLP c The interaction representation C is encoded by using a CNN to encode the scene visual images and trajectories in the form of an energy map, which encodes social and physical interactions. The interaction representation C comes from the model CSCNet.

[0149] f c =MLP c (C)

[0150] Secondly, use interpolation Transformer T i To learn the trajectory spectrum deviation and then reconstruct the complete trajectory spectrum, the spliced ​​features are then used. Passed to T i The encoder, and the key point trajectory spectrum Passed to T i The decoder; T i Used to predict the complete trajectory spectrum Finally, the reconstructed trajectory was obtained using IDFT.

[0151]

[0152] Therefore, one of the potential predicted trajectories is represented as follows:

[0153]

[0154] Among them, [t h :,:] represents the tensor slicing operation;

[0155] Finally, the fine-grained spectral interpolation subnetwork uses an interpolation Transformer to interpolate the trajectory spectrum of key points into a complete trajectory spectrum, and uses the remaining spectral portion to reflect the more refined interaction details of the agent.

[0156] By using Average Point-wise Loss (L APL Optimize fine-grained subnetworks.

[0157]

[0158] Furthermore, the method for predicting agent trajectories from a frequency domain perspective also includes:

[0159] The hierarchical prediction network is trained using a joint loss function L, which is:

[0160]

[0161] This invention also provides an agent trajectory prediction system from a frequency domain perspective, comprising:

[0162] The coarse-grained keypoint estimation module is used to predict the keypoint trajectory spectrum of an agent at a lower spatiotemporal resolution. It generates multiple potential predicted trajectories by designing an encoder-decoder structure to describe the uncertainty and randomness of the agent's future behavior.

[0163] A fine-grained spectral interpolation module is used to reconstruct the complete trajectory spectrum from the key point trajectory spectrum at a higher spatiotemporal resolution with the help of interactive representation.

[0164] The embodiments of the present invention have achieved some positive results during the research and development or use process, and have indeed great advantages compared with the prior art. The following content describes the experimental process with data, charts and other information.

[0165]

[0166] The datasets include eth, hotel, univ, zaral, and zara2. In the figure, lower performance metrics are considered better. The table shows that the method of this invention achieved the best results, with performance metrics improved by up to 62.5% / 72% compared to previous methods.

[0167] Model Name Performance indicators S-GAN 27.25 / 41.44 SoPhie 16.27 / 29.38 Multiverse 14.78 / 27.09 SimAug 12.03 / 23.98 PECNet 9.96 / 15.88 MANTR 8.96 / 17.76 LB-EBM 8.87 / 15.61 SpecTGNN 8.21 / 12.41 Y-net 7.85 / 11.85 Our method 7.12 / 11.39

[0168] This table compares the quantitative results of the method provided by this invention with other methods on the SDD dataset. Lower performance metrics are better. As can be seen from the table, the method of this invention achieves the best results, with performance metrics improved by a maximum of 73.9% / 72.5% compared to previous methods.

[0169] The method provided in this invention provides trajectory prediction in different scenarios. The solid line represents the observed trajectory, and the clustered distribution represents the predicted trajectory. Figure 2The method provided in this embodiment of the invention can meet the needs of different scenarios. In particular, in scenario h, it was found that the predicted trajectory given by the method of the present invention conforms to traffic rules and common sense, and the predicted trajectory all revolves around the roundabout.

[0170] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented using hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or using software executed by various types of processors, or using a combination of the above-described hardware circuitry and software, such as firmware.

[0171] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for predicting the trajectory of an intelligent agent from a frequency domain perspective, characterized in that, The frequency domain perspective method for predicting agent trajector trajectories includes: A hierarchical prediction network is constructed to predict the future trajectory of an agent at different frequency scales. This hierarchical prediction network comprises two sub-networks: A coarse-grained keypoint estimation subnetwork is used to predict the keypoint trajectory spectrum of an agent at a lower spatiotemporal resolution. By designing an encoder-decoder structure, multiple potential predicted trajectories are generated to describe the uncertainty and randomness of the agent's future behavior. A fine-grained spectral interpolation subnetwork is used to reconstruct the complete trajectory spectrum from the key point trajectory spectrum at a higher spatiotemporal resolution; The coarse-grained keypoint estimation subnetwork includes: Step 1: Perform Discrete Fourier Transform (DFT) on the agent's observation trajectory to obtain the agent's observation time [1,2,…,t]. h The trajectory spectrum of ]; Step 2: Integrate the use of multilayer perceptron (MLP) t Multilayer Perceptron II (MLP) i Obtain random noise features f i and high-dimensional features f t splicing together f e ; Step 3, use keypoint Transformer T k Alternatively, a Long Short-Term Memory (LSTM) network or a Temporal Convolutional Network (TCN) can be used to process the f network. e Encode; Step 4, use the decoder MLP e Predict the trajectory spectrum of N key points of the intelligent agent [A] key ,Φ key ]; Step 5: Optimize the coarse-grained keypoint estimation subnetwork by minimizing the average Euclidean distance between the true spatial coordinates and the predicted spatial coordinates of the keypoints, thereby predicting the trajectory spectrum of the keypoints; the fine-grained spectral interpolation subnetwork includes: First, the trajectory spectrum of the key points [A] key ,Φ key Input, obtain representation features Among them, the MLP here t MLP in coarse-grained networks t Different sharing parameters, where N key <t f ; Then, use MLP c The interaction representation C is encoded by using a CNN to encode the scene visual images and trajectories in the form of an energy map, which encodes social and physical interactions. The interaction representation C comes from the model CSCNet. f c =MLP c (C) Secondly, use interpolation Transformer T i To learn the trajectory spectrum deviation and then reconstruct the complete trajectory spectrum, the spliced ​​features are then used. Passed to T i The encoder, and the key point trajectory spectrum Passed to T i The decoder; T i Used to predict the complete trajectory spectrum Finally, the reconstructed trajectory was obtained using IDFT. Therefore, one of the potential predicted trajectories is represented as follows: Among them, [t h :, :] represents the tensor slicing operation; Finally, the fine-grained spectral interpolation subnetwork uses an interpolation Transformer to interpolate the trajectory spectrum of key points into a complete trajectory spectrum, and uses the remaining spectral portion to reflect the more refined interaction details of the agent. Its variables will be passed through the average point-wise loss (L... APL Optimize fine-grained subnetworks.

2. The method for predicting agent trajectory from a frequency domain perspective as described in claim 1, characterized in that, The observation trajectory of the intelligent agent is: make Let {I} represent the two-dimensional coordinates of an agent at time t. Given a video segment {I}, where {I} contains M agents at times [1, 2, ..., t], let {I} be the coordinates of an agent at time [1, 2, ..., t]. h ] observation trajectory X i This represents the observation trajectory of the i-th agent, used to predict the agent's trajectory in the future time period [t]. h +1,t h +2,…,t h +t f Potential trajectory coordinates 3. The method for predicting agent trajectory from a frequency domain perspective as described in claim 1, characterized in that, The specific process of step one is as follows: Observation trajectory 1D-DFT is applied to each dimension to obtain the observed trajectory spectrum. Including amplitude spectrum A = {a x ,a y } and phase spectrum Φ={φ x ,φ y }: Here, DFT represents the Discrete Fourier Transform function.

4. The method for predicting agent trajectory from a frequency domain perspective as described in claim 1, characterized in that, The specific process of step two is as follows: First, for random noise vectors Sampling is performed; then, a multilayer perceptron (MLP) is used. i Encoding the noise variable z yields the random noise feature f i ; Using a multilayer perceptron (MLP) t Observation trajectory spectrum of the coded agent Obtain high-dimensional features f t ; Finally, the f t With the f i splicing to obtain f e The formula is: Among them, f e Let [a,b] represent the concatenation of vectors {a,b} along the last dimension.

5. The method for predicting agent trajectory from a frequency domain perspective as described in claim 1, characterized in that, The specific process of step three is as follows: First embed the vector f e It is passed to the Transformer encoder, which will observe the trajectory spectrum (a x a y , φ x , φ y The input is fed into the Transformer decoder; the Transformer encoder is used as a feature extractor and does not contain a final output layer. Then a multilayer perceptron 3MLP was used. e By aggregating the features of nodes at different frequencies, the behavioral feature f is obtained, expressed as: The specific process of step four is as follows: The trajectory spectrum [A] key ,Φ key The expression for ] is: Where, N key This indicates the number of key points in the trajectory spectrum.

6. The method for predicting agent trajectory from a frequency domain perspective as described in claim 1, characterized in that, The specific process of step five is as follows: When training the coarse-grained subnetwork, the agent's real keypoint spatial coordinates y key As a label, the y key Obtained from the complete ground truth of the future trajectory; simultaneously, the spatial coordinates of the predicted key points are obtained using the inverse discrete Fourier transform (IDFT). By minimizing the true y key and prediction The average Euclidean distance between them is used to optimize the subnetwork, thereby learning to predict the trajectory spectrum of the corresponding key points; The average Euclidean distance is the average keypoint loss (L). AKL The expression is: in, Here, IDFT represents the inverse discrete Fourier transform function.

7. The method for predicting agent trajectory from a frequency domain perspective as described in claim 1, characterized in that, The frequency domain perspective method for predicting agent trajector trajectories also includes: The hierarchical prediction network is trained using a joint loss function L, which is:

8. A frequency-domain perspective intelligent agent trajectory prediction system implementing the frequency-domain perspective intelligent agent trajectory prediction method as described in any one of claims 1-7, characterized in that, The frequency domain perspective intelligent agent trajectory prediction system includes: The coarse-grained keypoint estimation module is used to predict the keypoint trajectory spectrum of an agent at a lower spatiotemporal resolution. It generates multiple potential predicted trajectories by designing an encoder-decoder structure to describe the uncertainty and randomness of the agent's future behavior. A fine-grained spectral interpolation module is used to reconstruct the complete trajectory spectrum from the key point trajectory spectrum at a higher spatiotemporal resolution with the help of interactive representation.

9. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, causes the processor to perform the steps of the intelligent agent trajectory prediction method in the frequency domain as described in any one of claims 1-7.

Images