A method and apparatus for predicting the trajectory of an object

Abstract

The application provides a method and device for predicting the trajectory of an object, and belongs to the technical field of computer vision and image processing. The method comprises the following steps: reconstructing a multi-view time sequence two-dimensional image sequence of an acquired target object to obtain an original particle point cloud sequence, and generating a sparse key point cloud sequence after sampling; constructing a particle graph sequence based on the spatial position of a key point at each moment in the sequence, and performing spatial semantic completion; performing object-level dynamic space-time aggregation on the key point feature sequence after completion to obtain aggregated key point features, and inputting the aggregated key point features into a particle graph converter to output key point update features after long-range force propagation modeling by the converter; and decoding the key point displacement at a future moment based on the update features, and generating the future motion trajectory of the target object according to the key point displacement. A device for predicting the trajectory of an object is also provided based on the method. The application can simultaneously predict the fine appearance and stable and accurate motion trajectory of an object.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and image processing, and particularly relates to a method and device for predicting the trajectory of an object. BACKGROUND

[0002] Accurate prediction of the future trajectory of a moving object is of great significance in the fields of industrial robots, autonomous driving, augmented reality, etc. In the field of industrial robots, the robot needs to predict the trajectory of a workpiece moving quickly on a conveyor belt or in free fall, in order to achieve precise grasping or avoidance, and to improve automation efficiency and flexibility. In the field of autonomous driving, vehicles need to predict the motion trajectory of surrounding pedestrians, vehicles and other road users, in order to plan a safe path and avoid collisions. In the field of augmented reality, virtual objects need to interact realistically with moving objects in the real world, which relies on accurate prediction of the motion of real objects. In addition, when embodied intelligent robots (such as household service robots, warehouse logistics robots) perform tasks involving moving objects (such as picking up objects, tidying up the environment), accurate prediction of object motion is a prerequisite for reliable and natural interaction. Existing technologies usually obtain a particle representation of an object based on dynamic three-dimensional reconstruction, and use a graph neural network to model particle dynamics. Specifically, these methods sample sparse key points from the original particle point cloud of the reconstructed object, and input the key point coordinates as graph nodes into the graph neural network to predict the motion.

[0003] However, the existing methods have two core defects: first, the sparse key points severely lose the original geometric details and appearance information of the object, resulting in artifacts in the predicted appearance and deviations in the predicted trajectory; second, the inherent hop-by-hop message passing mechanism of the graph neural network cannot effectively model the force transmission across long distances in the motion of the object, resulting in feature homogenization of the force signal during transmission, and the inability to accurately express the force action from different sources and distances, which in turn causes the accumulation of prediction errors in the key point positions, resulting in significant drift in the predicted trajectory. SUMMARY

[0004] To solve the above technical problems, the present application proposes a method and device for predicting the trajectory of an object in motion. The method can simultaneously predict the fine appearance and stable, accurate trajectory of an object, and exhibits stronger adaptability and robustness in predicting the motion of complex or unknown material objects.

[0005] To achieve the above-mentioned purposes, the present application adopts the following technical solutions:

[0006] A method for predicting the trajectory of an object in motion, comprising the following steps:

[0007] reconstructing a multi-view time-series two-dimensional image sequence of a target object obtained to obtain an original particle point cloud sequence that can be tracked across frames; sampling each frame of the original particle point cloud to generate a corresponding sparse key point cloud sequence;

[0008] Based on the spatial position of each key point in the sparse key point cloud sequence, a particle graph sequence is constructed, the key points are taken as graph nodes, and edges are constructed according to the spatial proximity relationship between the key points;

[0009] The sparse key point cloud sequence is subjected to key point level spatial semantic completion to obtain a completed key point feature sequence containing original particle point cloud geometric details, and the completed key point feature sequence is subjected to object level dynamic space-time aggregation to obtain aggregated key point features;

[0010] The aggregated key point features are input into a particle graph converter, long-range interaction forces between the key points are modeled through a graph attention mechanism in the particle graph converter, and updated key point features after long-range force propagation modeling are output;

[0011] Based on the updated key point features, key point displacements at a future time are predicted, and a future motion trajectory of the target object is generated according to the key point displacements.

[0012] The application further provides an object motion trajectory prediction device, which comprises at least one processor and a memory, and the memory stores a computer program, and the computer program is executed by the at least one processor to realize the object motion trajectory prediction method.

[0013] The effects provided in the summary are only the effects of the embodiments, not all the effects of the application, and one of the above technical solutions has the following advantages or beneficial effects:

[0014] The application provides a method and device for predicting the trajectory of an object, and belongs to the technical field of computer vision and image processing. The method comprises the following steps: reconstructing a multi-view time sequence two-dimensional image sequence of an acquired target object to obtain an original particle point cloud sequence which can be tracked across frames; sampling the original particle point cloud of each frame to generate a corresponding sparse key point cloud sequence; constructing a particle graph sequence based on the spatial position of a key point at each moment in the sparse key point cloud sequence, taking the key point as a graph node, and constructing an edge according to the spatial proximity relationship between the key points; performing key point level spatial semantic completion on the sparse key point cloud sequence to obtain a completed key point feature sequence containing the geometric details of the original particle point cloud; performing object level dynamic space-time aggregation on the completed key point feature sequence to obtain an aggregated key point feature; inputting the aggregated key point feature into a particle graph converter, modeling the long-range interaction force between the key points through a graph attention mechanism in the particle graph converter, and outputting a key point update feature which is modeled after long-range force propagation; and decoding the key point displacement at a future moment based on the key point update feature, and generating the future motion trajectory of the target object according to the key point displacement. Based on the method for predicting the trajectory of an object, a device for predicting the trajectory of an object is also provided. Based on the space-time semantic completion and long-range force propagation modeling, the method can simultaneously predict the fine appearance and stable and accurate motion trajectory of an object, and has stronger adaptability and robustness for motion prediction of complex or unknown material objects.

[0015] The application effectively fuses the fine-grained geometric details of the original point cloud and the cross-frame motion information through key point level spatial semantic completion and object level dynamic space-time aggregation, and provides a key point representation basis which is more complete in information and more consistent in space-time for appearance prediction and motion decoding.

[0016] The application introduces a particle graph converter, uses the graph attention mechanism of the particle graph converter to establish a direct force conduction path between the key points, overcomes the limitations of traditional GNN message passing, can accurately express long-distance force action, and significantly suppresses the drift phenomenon in trajectory prediction. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 A flowchart of a method for predicting the trajectory of an object is provided for the embodiment 1 of the application;

[0018] Figure 2 A framework diagram of the implementation of the method for predicting the trajectory of an object is provided for the embodiment 1 of the application;

[0019] Figure 3 A schematic diagram of a device for predicting the trajectory of an object is provided for the embodiment 2 of the application. DETAILED DESCRIPTION

[0020] To clearly illustrate the technical features of this solution, the invention will be described in detail below through specific embodiments and in conjunction with the accompanying drawings. The following disclosure provides many different embodiments or examples for implementing different structures of the invention. To simplify the disclosure of the invention, components and arrangements of specific examples are described below. Furthermore, reference numerals and / or letters may be repeated in different examples. This repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed. It should be noted that the components illustrated in the drawings are not necessarily drawn to scale. Descriptions of well-known components, processing techniques, and processes are omitted in this invention to avoid unnecessarily limiting the invention.

[0021] Example 1

[0022] Embodiment 1 of this invention proposes a method for predicting the trajectory of an object, which is used to solve the problems of trajectory drift and inaccurate appearance caused by incomplete key point representation and insufficient long-range force transmission modeling.

[0023] This invention constructs an object dynamics modeling framework capable of simultaneously achieving accurate appearance prediction and stable, accurate motion trajectory prediction, effectively overcoming the limitations of existing methods. The implementation process of this invention includes: trackable particle representation based on dynamic reconstruction, spatiotemporal feature completion and aggregation mechanisms, and dynamics modeling based on a particle graph converter.

[0024] The task of predicting the trajectory of an object's motion is that the input is a set of... One perspective 2D visual observation images of video frames The output is the predicted future motion of the object through dynamic modeling. .

[0025] Figure 1 This is a flowchart of a method for predicting the trajectory of an object according to Embodiment 1 of the present invention;

[0026] In step S110, the multi-view temporal two-dimensional image sequence of the acquired target object is reconstructed to obtain an original particle point cloud sequence that can be tracked across frames; the original particle point cloud of each frame is sampled to generate a corresponding sparse key point cloud sequence.

[0027] Figure 2 This is a framework diagram of the object motion trajectory prediction method proposed in Embodiment 1 of the present invention;

[0028] The acquisition of multi-view temporal two-dimensional image sequences of the target object is reconstructed to obtain an original particle point cloud sequence that can be tracked across frames. Specifically, based on the dynamic 3D Gaussian reconstruction method, static 3D Gaussian reconstruction is performed on the first frame image to obtain the appearance descriptor, initial position, and rotation of the particles. The appearance descriptor is frozen, and the current frame is initialized based on the particle position and rotation of the previous frame in a recursive optimization manner. The particle position and rotation of the current frame are optimized under the constraints of the observation data of the current frame to obtain an original particle point cloud sequence with inter-frame correspondence.

[0029] For each frame of the original particle point cloud, the corresponding sparse key point cloud sequence is generated by sampling the original particle point cloud. Specifically, the sparse key point cloud sequence is generated from the original particle point cloud of each frame using the farthest point sampling method.

[0030] First, dynamic 3D Gaussian is used to analyze visual observation data. Reconstructing a traceable sequence of original particle positions ,in Represents the original particle coordinates. It is the index of the original particle. We first look at the first frame. Perform static 3D Gaussian splashing to obtain particle-based appearance descriptors (e.g., size, color, opacity) and an initial spatial representation, including position. and rotation .

[0031] Subsequently, the particle appearance descriptor is frozen, and a recursive optimization paradigm is employed, where the particle position and rotation are calculated for each frame. Based on the initialization of the previous frame, and in the observed data We optimize sequentially under constraints. This frame-by-frame optimization establishes a one-to-one correspondence between particles across frames, thus obtaining a temporal sequence of particle positions and rotations. To balance performance and computational cost, we employ farthest-point sampling (FPS) to extract keypoints from the original particle point cloud of each frame. Forming a key point cloud sequence .

[0032] In step S120, a particle graph sequence is constructed based on the spatial locations of key points at each time step in the sparse keypoint cloud sequence. Key points are treated as graph nodes, and edges are constructed according to the spatial proximity relationships between key points. Specifically, the steps include: defining each key point in the sparse keypoint cloud as a graph node; constructing connecting edges for each graph node based on spatial proximity relationships, wherein the connecting edges are constructed in one of the following ways: connecting each node to its corresponding node in Euclidean space. Find the nearest neighbor node; or connect all node pairs whose Euclidean distance is less than a preset distance threshold; generate edge features for each connection edge, which are used to encode the spatial relationship between the two connected nodes.

[0033] In step S130, keypoint-level spatial semantic completion is performed on the sparse keypoint cloud sequence to obtain a completed keypoint feature sequence containing geometric details of the original particle point cloud; object-level dynamic spatiotemporal aggregation is performed on the completed keypoint feature sequence to obtain aggregated keypoint features.

[0034] The process of performing keypoint-level spatial semantic completion on sparse keypoint cloud sequences includes: extracting coordinate features of keypoints; calculating the relative position information between keypoints and encoding it as a spatial bias, adding the spatial bias to the coordinate features to obtain enhanced coordinate features; extracting neighborhood features of each keypoint's neighborhood from the original particle point cloud; and using a multi-head position-aware attention mechanism, with enhanced coordinate features as the query, enhanced neighborhood features as the key, and neighborhood features as the value, and with Gaussian weighted Euclidean distance between keypoints as the attention prior, fusing enhanced coordinate features, enhanced neighborhood features, and neighborhood features to obtain the completed keypoint features.

[0035] Use a coordinate encoder Coordinates of key points Mapping to coordinate features These features serve as the initial embeddings for the keypoints:

[0036] ;

[0037] in, Indicates the first The sparse keypoint cloud coordinate matrix of the frame; Indicates the number of key points; This represents the learnable weight matrix of the first layer; This represents the learnable bias vector of the first layer; This represents the learnable weight matrix of the second layer; This represents the learnable bias vector of the second layer; This represents the output coordinate feature matrix; The dimension representing the coordinate feature.

[0038] To improve the model's perception of the spatial distribution of keypoints, a coordinate difference encoder is used to learn relative distances as spatial biases. Specifically, a two-layer line mapping with ReLU activation is used to encode the coordinate differences between keypoints. The encoded spatial biases are then added to the coordinate features. In this way, enhanced coordinate features are obtained. Furthermore, relying solely on the coordinate features of sparse keypoints is insufficient to capture the spatial semantics of moving objects. Therefore, PointNet is used to aggregate the coordinates of each keypoint in the original particle point cloud. Find the nearest neighbor points to obtain the neighborhood features. Add the encoded spatial bias to the neighborhood features. In this way, enhanced neighborhood characteristics are obtained. To address the inconsistency between coordinate features and neighborhood features in the feature space, a multi-head position-aware attention mechanism is adopted, which guides cross-spatial feature fusion through Gaussian weighted relationships.

[0039] Completed key features The calculation method is as follows:

[0040] ;

[0041] ;

[0042] in, Indicates the first The enhanced coordinate feature matrix of the frame; Indicates the first The enhanced neighborhood feature matrix of the frame; This represents the learnable weight matrix used to project input features into the query space; This represents the learnable weight matrix used to project input features into the key space; This represents the learnable weight matrix used to project input features onto the value space; This represents the calculated query tensor; This represents the calculated key tensor; This represents the calculated value tensor; This represents the feature dimension of the key vector; Indicates the sharpness coefficient; Represents the Gaussian weight matrix;

[0043] ;

[0044] It is the Euclidean distance between keypoints. This allows neighborhood features to be used in the prior art. Aggregation is carried out using spatial perception under the guidance of [the relevant authority]. and The key applications of the frame follow the same process to obtain the complete feature sequence. .

[0045] The completed keypoint feature sequence is dynamically aggregated at the object level to obtain aggregated keypoint features. Specifically, a reference frame is specified from the completed keypoint feature sequence; the motion offset of the non-reference frame relative to the reference frame is calculated through a motion alignment network; the motion offset is added to the completed keypoint features of the corresponding non-reference frame to obtain a motion-aligned keypoint feature sequence; and a temporal attention mechanism is used to perform a weighted summation of the motion-aligned keypoint feature sequence to obtain the aggregated keypoint features.

[0046] The object-level dynamic spatiotemporal aggregation module uses a motion alignment network to model and compensate for inter-frame motion offsets, and aggregates adjacent frames (i.e., arrive The motion of objects in (). We take the first Using the frame as a reference frame, calculate the first frame. Frame and the The relative motion offset of the frames. These offsets are added to the frame number respectively. Frame and the Frame feature vector In order to compensate for the motion space gap caused by cross-frame motion:

[0047]

[0048] in Indicates cross-frame features, Used to limit the magnitude of motion offset and prevent overcorrection. ":" and " "represents the learnable matrix" Block-level partitioning and mapping operations, bias The same partitioning operation is used. Indicates the first Frame alignment features The acquisition method is the same as above, and it serves as a reference frame. Zero correction is applied. Then, spatiotemporal attention is applied to selectively aggregate the feature sequences:

[0049] ;

[0050] ;

[0051] in, This represents the key point features after dynamic temporal aggregation.

[0052] In step S140, the aggregated keypoint features are input to the particle graph converter. The long-range interaction force between keypoints is modeled through the graph attention mechanism in the particle graph converter, and the updated keypoint features after long-range force propagation modeling are output.

[0053] The long-range interaction forces between keypoints are modeled using the graph attention mechanism in the particle graph converter, and the updated keypoint features after long-range force propagation modeling are output; specifically:

[0054] The aggregated keypoint features are projected into query tensors. Key tensors and at least one value tensor ;

[0055] Incorporating edge features representing relationships between nodes into the key tensor middle;

[0056] The attention score between nodes in the graph is calculated by scaling the dot product; the formula for calculating the attention score is:

[0057] ;

[0058] in, Indicates the first In the layer particle graph converter, nodes and nodes Normalized attention weights; and All are node indexes; Indicates the first Layer nodes The query tensor; Indicates the first Layer nodes The key tensor; It is a connection node and Edge features; It is a node The set of neighboring nodes; It is the layer index of the particle diagram converter; Indicates the first Layer nodes The key tensor; It is a node The neighboring nodes; Indicates the connection node and nodes The edge feature vectors.

[0059] Based on attention scores, the value tensor Sum of edge features Aggregation is performed to obtain updated keypoint features. :

[0060] .

[0061] In step S150, the key point displacements at future moments are predicted based on the updated key point feature decoding, and the future motion trajectory of the target object is generated based on the key point displacements; specifically:

[0062] Calculate the key point positions at future times based on the current key point positions and the predicted displacement vector;

[0063] Based on the key point locations in the future, the positions and rotations of the original particle point cloud in the future are obtained by interpolation using a linear skinning algorithm, so as to fully represent the motion trajectory of the target object.

[0064] Features encoded by particle diagram transformer Finally, the motion predictor decodes the keypoint displacement direction. Based on these vectors, predict the first... frame key point location Furthermore, interpolation prediction is performed with the help of hybrid linear skinning. The original particle point cloud of the frame, including position and rotation This enables the prediction of an object's future motion.

[0065] The object motion trajectory prediction method proposed in Embodiment 1 of this invention is based on spatiotemporal semantic completion and long-range force propagation modeling. It can simultaneously predict the fine appearance of the object and the stable and accurate motion trajectory, and shows stronger adaptability and robustness in motion prediction of objects made of complex or unknown materials.

[0066] The object motion trajectory prediction method proposed in Embodiment 1 of this invention effectively integrates the fine-grained geometric details of the original point cloud with cross-frame motion information through key point-level spatial semantic completion and object-level dynamic spatiotemporal aggregation, providing a more complete and spatiotemporally consistent key point representation basis for appearance prediction and motion decoding.

[0067] The method for predicting the trajectory of an object proposed in Embodiment 1 of this invention introduces a particle graph converter and uses its graph attention mechanism to establish a direct force transmission path between key points, which overcomes the limitations of traditional GNN message passing, can accurately express the force action over long distances, and significantly suppresses the drift phenomenon in trajectory prediction.

[0068] Example 2

[0069] The present invention also proposes a device, Figure 3 This is a schematic diagram of an object motion trajectory prediction device proposed in Embodiment 2 of the present invention, comprising:

[0070] Memory, used to store computer programs;

[0071] When a processor executes the computer program, the method steps are as follows:

[0072] In step S110, the multi-view temporal two-dimensional image sequence of the acquired target object is reconstructed to obtain an original particle point cloud sequence that can be tracked across frames; the original particle point cloud of each frame is sampled to generate a corresponding sparse key point cloud sequence.

[0073] In step S120, a particle graph sequence is constructed based on the spatial location of key points at each moment in the sparse key point cloud sequence, with key points as graph nodes, and edges are constructed according to the spatial proximity relationship between key points.

[0074] In step S130, keypoint-level spatial semantic completion is performed on the sparse keypoint cloud sequence to obtain a completed keypoint feature sequence containing geometric details of the original particle point cloud; object-level dynamic spatiotemporal aggregation is performed on the completed keypoint feature sequence to obtain aggregated keypoint features.

[0075] In step S140, the aggregated keypoint features are input to the particle graph converter. The long-range interaction forces between keypoints are modeled through the graph attention mechanism in the particle graph converter, and the updated keypoint features after long-range force propagation modeling are output.

[0076] In step S150, the key point displacements at future moments are predicted based on the updated key point feature decoding, and the future motion trajectory of the target object is generated based on the key point displacements.

[0077] The object motion trajectory prediction device proposed in Embodiment 2 of this invention is based on spatiotemporal semantic completion and long-range force propagation modeling. It can simultaneously predict the fine appearance of an object and its stable and accurate motion trajectory, and shows stronger adaptability and robustness in predicting the motion of objects made of complex or unknown materials.

[0078] It should be noted that the present invention also provides an electronic device, including: a communication interface capable of interacting with other devices such as network devices; and a processor connected to the communication interface to enable information interaction with other devices, used to execute a method for predicting the trajectory of an object provided by one or more of the above-mentioned technical solutions when running a computer program, wherein the computer program is stored in a memory. In practical applications, the various components of the electronic device are coupled together through a bus system. It is understood that the bus system is used to realize the connection and communication between these components. In addition to a data bus, the bus system also includes a power bus, a control bus, and a status signal bus. The memory in the embodiments of this application is used to store various types of data to support the operation of the electronic device. Examples of this data include any computer program used to operate on the electronic device. It is understood that the memory can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory, flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache.By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM). The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memory. The methods disclosed in the embodiments of this application can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The processor can be a general-purpose processor, a DSP (Digital Signal Processing, i.e., a chip capable of implementing digital signal processing technology), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules can be located in a storage medium, which is located in memory. The processor reads the program from the memory and, in conjunction with its hardware, completes the steps of the aforementioned method. When the processor executes the program, it implements the corresponding processes in the various methods of the embodiments of this application; for simplicity, these will not be elaborated further here.

[0079] The description of the relevant parts of the object motion trajectory prediction device provided in Embodiment 2 of this application can be found in the detailed description of the corresponding parts of the object motion trajectory prediction method provided in Embodiment 1 of this application, and will not be repeated here.

[0080] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that the elements inherent in a process, method, article, or apparatus that includes a list of elements are included. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Additionally, portions of the technical solutions provided in the embodiments of this application that are consistent with the implementation principles of corresponding technical solutions in the prior art have not been described in detail to avoid excessive elaboration.

[0081] While specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art can make other modifications or variations based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for predicting the trajectory of an object, characterized in that, Includes the following steps: The acquired multi-view temporal two-dimensional image sequence of the target object is reconstructed to obtain the original particle point cloud sequence that can be tracked across frames. For each frame of the original particle point cloud, sample the original particle point cloud to generate the corresponding sparse key point cloud sequence; Based on the spatial location of key points at each time step in the sparse key point cloud sequence, a particle graph sequence is constructed, with the key points as graph nodes, and edges are constructed according to the spatial proximity relationship between key points. The sparse keypoint cloud sequence is subjected to keypoint-level spatial semantic completion to obtain a completed keypoint feature sequence containing geometric details of the original particle point cloud; the completed keypoint feature sequence is subjected to object-level dynamic spatiotemporal aggregation to obtain aggregated keypoint features; the sparse keypoint cloud sequence is subjected to keypoint-level spatial semantic completion, specifically: extracting the coordinate features of keypoints. The relative position information between key points is calculated and encoded as a spatial offset, and the spatial offset is added to the coordinate features to obtain enhanced coordinate features; Extract neighborhood features of each key point from the original particle point cloud; By utilizing a multi-head position-aware attention mechanism, enhanced coordinate features are used as queries, enhanced neighborhood features as keys, and neighborhood features as values. The Gaussian weighted Euclidean distance between keypoints is used as the attention prior. Enhanced coordinate features, enhanced neighborhood features, and neighborhood features are fused to obtain the completed keypoint features. The completed keypoint feature sequence is subjected to object-level dynamic spatiotemporal aggregation to obtain aggregated keypoint features. Specifically, a reference frame is specified from the completed keypoint feature sequence; the motion offset of the non-reference frame relative to the reference frame is calculated using a motion alignment network; the motion offset is added to the completed keypoint features of the corresponding non-reference frame to obtain a motion-aligned keypoint feature sequence; and a temporal attention mechanism is used to perform a weighted summation of the motion-aligned keypoint feature sequence to obtain the aggregated keypoint features. The aggregated keypoint features are input into the particle graph converter, and the long-range interaction forces between keypoints are modeled through the graph attention mechanism in the particle graph converter. The updated keypoint features after long-range force propagation modeling are output. Based on the key point update feature decoding, predict the key point displacement at future moments, and generate the future motion trajectory of the target object based on the key point displacement.

2. The method according to claim 1, characterized in that, The acquired multi-view temporal two-dimensional image sequence of the target object is reconstructed to obtain the original particle point cloud sequence that can be tracked across frames, specifically: Based on the dynamic 3D Gaussian reconstruction method, static 3D Gaussian reconstruction is performed on the first frame image to obtain the particle appearance descriptor, initial position and rotation; The appearance descriptor is frozen, and the current frame is initialized based on the particle position and rotation of the previous frame in a recursive optimization manner. The particle position and rotation of the current frame are optimized under the constraints of the observation data of the current frame to obtain the original particle point cloud sequence with inter-frame correspondence.

3. The method according to claim 1, characterized in that, For each frame of the original particle point cloud, a corresponding sparse key point cloud sequence is generated by sampling, specifically as follows: The sparse key point cloud sequence is generated from the original particle point cloud of each frame using the farthest point sampling method.

4. The method according to claim 1, characterized in that, Completed key features The calculation method is as follows: ； ； in, Indicates the first The enhanced coordinate feature matrix of the frame; Indicates the first The enhanced neighborhood feature matrix of the frame; This represents the learnable weight matrix used to project input features into the query space; This represents the learnable weight matrix used to project input features into the key space; This represents the learnable weight matrix used to project input features onto the value space; This represents the calculated query tensor; This represents the calculated key tensor; This represents the calculated value tensor; This represents the feature dimension of the key vector; Indicates the sharpness coefficient; Represents the Gaussian weight matrix; ； It is the Euclidean distance between key points.

5. The method according to claim 1, characterized in that, The graph attention mechanism in the particle graph converter is used to model the long-range interaction forces between key points, and the updated key point features after long-range force propagation modeling are output. Specifically: The aggregated keypoint features are projected into query tensors. Key tensors and at least one value tensor ; The edge features representing the relationships between nodes are incorporated into the key tensor. middle; Calculate the attention score between nodes in the graph by scaling the dot product; Based on the attention score, the value tensor Sum of edge features Aggregation is performed to obtain the updated keypoint features.

6. The method according to claim 5, characterized in that, The formula for calculating the attention score is: ； in, Indicates the first In the layer particle graph converter, nodes and nodes Normalized attention weights; and All are node indexes; Indicates the first Layer nodes The query tensor; Indicates the first Layer nodes The key tensor; It is a connection node and Edge features; It is a node The set of neighboring nodes; It is the layer index of the particle diagram converter; Indicates the first Layer nodes The key tensor; It is a node The neighboring nodes; Indicates the connection node and nodes The edge feature vectors.

7. The method according to claim 1, characterized in that, And based on the displacement of the key points, the future trajectory of the target object is generated, specifically as follows: Calculate the key point positions at future times based on the current key point positions and the predicted displacement vector; Based on the key point positions at the future time, the positions and rotations of the original particle point cloud at the future time are obtained by interpolation using a linear skinning algorithm, so as to fully represent the motion trajectory of the target object.

8. A device for predicting the trajectory of an object, comprising at least one processor and a memory, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the at least one processor, it implements a method for predicting the trajectory of an object as described in any one of claims 1 to 7.

Images