Method and apparatus for predicting trajectory of object based on velocity field prediction network
By combining a deformation-assisted network with a divergence-free velocity field, and integrating multi-view video learning and motion state perception functions, the problem of insufficient prediction of sudden motion in existing technologies is solved. This improves the accuracy and robustness of dynamic scene dynamics modeling and is applicable to applications such as robot navigation, embodied interaction, and autonomous driving.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HARBIN INST OF TECH AT WEIHAI
- Filing Date
- 2026-03-18
- Publication Date
- 2026-06-09
AI Technical Summary
Existing technologies cannot effectively predict sudden movements such as explosions, bounces, and collisions, resulting in insufficient accuracy and robustness in dynamic scene dynamics modeling.
A combined architecture of deformation-assisted network and divergence-free velocity field is adopted. The physical laws of the scene are learned through multi-view video. The relative displacement, rotation and scaling of particles are predicted by combining deformation-assisted network and velocity field prediction network. The particle set is optimized by using motion state perception function and a loss function is constructed for accurate prediction.
It enables accurate prediction of sudden movements such as explosions, bounces, and collisions, improving the accuracy and robustness of dynamic scene dynamics modeling, and is applicable to fields such as robot navigation, embodied interaction, dynamic 3D generation, and autonomous driving.
Smart Images
Figure CN121904111B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and image processing technology, and specifically relates to a method and device for predicting the trajectory of an object. Background Technology
[0002] Obtaining accurate physical models of complex dynamic 3D environments is a core prerequisite for the practical application of technologies in many emerging fields such as robot planning, embodied interaction, and dynamic 3D / 4D generation. In robot planning, accurate dynamic 3D physical models are fundamental for autonomous robots to perform obstacle avoidance, path planning, and task execution in complex environments, ensuring the accuracy of the robot's prediction of the motion states of surrounding objects. In embodied interaction scenarios, this model supports a seamless blend of virtual and real-world interactive experiences, enabling virtual entities to accurately respond to dynamic changes in the real environment. Furthermore, in the field of dynamic 3D / 4D generation, it is a core support for applications such as film and television special effects production and the development of virtual simulation training systems, enabling the generation and simulation of high-fidelity dynamic scenes. With the rapid development of these fields, increasingly higher demands are being placed on the accuracy and generalization capabilities of physical modeling for complex dynamic 3D environments.
[0003] In recent years, three-dimensional learning techniques, represented by Neural Radiance Fields (NeRF), 3D Gaussian Splatting (3DGS) and its derivatives, have made progress in areas such as new perspective interpolation and three-dimensional shape reconstruction. However, these methods generally lack the ability to predict future motion. The core reason is that they have failed to effectively mine and learn the core physical properties of three-dimensional space. They can only fit and interpolate existing observation frame information and cannot infer the scene motion state during unobserved periods. To compensate for the lack of physical property learning, two types of improvement schemes have emerged in related fields: one is the method based on Physical-Informed Neural Networks (PINN) constraints, which models physical properties by incorporating partial differential equations and PINN loss terms. However, since the PINN loss term emphasizes the consistency constraints of global physical laws, it is easy to suppress the local fine features of the boundary region during the optimization process, resulting in inaccurate learning of physical laws in the boundary region and fuzzy prediction of future frames. The second is the method based on explicitly embedding neural networks into physical models (such as spring-mass models and graph network models). Although it has certain effects on physical property learning and simulation transfer, it has poor versatility and is only applicable to specific objects or motion scenes. It requires additional boundary conditions such as object masks and is difficult to adapt to complex dynamic scenes.
[0004] To address the aforementioned shortcomings, related research proposes a combined architecture of deformation-assisted networks and divergence-free velocity fields. This architecture requires no additional prior information and can learn the physical laws of a scene solely through multi-view videos, enabling the extrapolation of future motion in 3D scenes. While this approach solves some of the pain points of traditional methods, it still has significant limitations, namely, its inability to effectively predict sudden movements such as explosions, bounces, and collisions. The fundamental reason is that the physical rules governing such sudden movements are highly transient and discontinuous. Existing divergence-free velocity fields used for modeling object motion lack specific designs for addressing the abrupt changes in system state caused by sudden movements, making it difficult to model their evolutionary patterns and limiting their expansion and application in dynamic interactive scenarios. Summary of the Invention
[0005] To address the aforementioned technical problems, this invention proposes a method and device for predicting object motion trajectories. It can effectively predict sudden movements such as explosions, bounces, and collisions, improving the accuracy and robustness of dynamic scene dynamics modeling.
[0006] To achieve the above objectives, the present invention adopts the following technical solution:
[0007] In a first aspect, the present invention proposes a method for predicting the trajectory of an object, comprising the following steps:
[0008] Obtain a two-dimensional image sequence of the target object and construct an observation frame set; use the initial frame in the observation frame set to perform three-dimensional reconstruction of the target object to obtain an initial three-dimensional Gaussian particle set;
[0009] Using a deformation-assisted network, the relative displacement, relative rotation, and relative scaling of the particles from the initial frame to each observation frame are predicted based on the three-dimensional position, physical encoding, object geometric semantic encoding, and timestamp corresponding to the target observation frame of the initial three-dimensional Gaussian particle. The three-dimensional position, three-dimensional rotation, and three-dimensional scaling of the initial three-dimensional Gaussian particle are transformed using the relative displacement, relative rotation, and relative scaling, respectively, to obtain the particle set corresponding to each observation frame.
[0010] The three-dimensional positions of the particles in the particle set corresponding to each observation frame are input into the velocity field prediction network. The velocity field prediction network generates the velocity field of the particles in the current observation frame and updates the three-dimensional positions of the particles according to the velocity field to obtain the particle set corresponding to the future frame.
[0011] The particle set corresponding to the future frame is projected into a two-dimensional image space. A loss function is constructed based on the difference between the rendered image and the actual observed image. The particle set corresponding to the future frame is optimized based on the loss function, and the optimized particle set is used as the predicted object motion trajectory.
[0012] Secondly, the present invention proposes an object motion trajectory prediction device, comprising at least one processor and a memory, wherein the memory stores a computer program, characterized in that the computer program, when executed by the at least one processor, implements the object motion trajectory prediction method.
[0013] The effects described in the invention are merely those of the embodiments, and not all the effects of the invention. One of the above technical solutions has the following advantages or beneficial effects:
[0014] This invention proposes a method and device for predicting the trajectory of an object, belonging to the field of computer vision and image processing technology. The method includes the following steps: acquiring a two-dimensional image sequence of the target object and constructing an observation frame set; using the initial frame in the observation frame set to perform three-dimensional reconstruction of the target object to obtain an initial three-dimensional Gaussian particle set; using a deformation-assisted network, based on the three-dimensional position, physical encoding, object geometric semantic encoding, and timestamp corresponding to the target observation frame of the initial three-dimensional Gaussian particles, predicting the relative displacement, relative rotation, and relative scaling of the particles from the initial frame to each observation frame; and using the relative displacement, relative rotation, and relative scaling to respectively adjust the initial three-dimensional Gaussian particle set. The method transforms the particle's 3D position, 3D rotation, and 3D scaling to obtain the particle set corresponding to each observation frame. The 3D positions of the particles in the particle set corresponding to each observation frame are input into a velocity field prediction network. The velocity field prediction network generates the velocity field of the particles in the current observation frame and updates the 3D positions of the particles according to the velocity field to obtain the particle set corresponding to the future frame. The particle set corresponding to the future frame is projected onto a 2D image space. A loss function is constructed based on the difference between the rendered image and the actual observed image. The particle set corresponding to the future frame is optimized based on the loss function to obtain the optimized particle set as the predicted object motion trajectory. Based on this method, a device for predicting object motion trajectories is also proposed. This invention introduces a motion state perception function, integrates timestamps, spacing between interactive objects, current motion speed, and material features to construct a multi-dimensional input, decodes the system state representation, basic dissipation coefficient, and time-series decay factor, and modulates the divergence-free velocity field to obtain a Gaussian velocity field for motion state perception. This significantly improves the model's ability to perceive and adapt to sudden changes in system state, achieving accurate prediction of sudden motion.
[0015] The geometrically aware deformation-assisted field constructed in this invention utilizes the initial frame Gaussian particle parameters, timestamps, physical codes, and object geometric semantic codes obtained by encoding the initial frame Gaussian particle set through the PointNet++ encoder. These are input into the network to predict relative displacement, rotation, and scaling changes, providing an excellent initialization foundation for velocity field optimization, reducing the difficulty of direct optimization, and accelerating model convergence. At the same time, the integration of object geometric semantic codes enhances the accuracy of cross-frame deformation prediction, further improving the stability, adaptability, and deformation prediction accuracy of modeling in complex scenes.
[0016] The same-material constraint designed in this invention ensures that the basic dissipation coefficients of particles of the same material tend to be consistent, which conforms to the material properties of the real physical world. The temporal decay constraint guarantees the cumulative irreversibility of inelastic dissipation from collisions, making the temporal decay factor monotonically decrease with time, which matches the physical rule that the energy retention capacity of objects in the real world continuously decreases over time. Through the synergistic effect of multiple constraints, the accuracy and reliability of motion prediction are further improved. Attached Figure Description
[0017] Figure 1 This is a flowchart of a method for predicting the trajectory of an object according to Embodiment 1 of the present invention;
[0018] Figure 2 This is an architecture diagram of a method for predicting the trajectory of an object proposed in Embodiment 1 of the present invention;
[0019] Figure 3 This is a schematic diagram of an object motion trajectory prediction device proposed in Embodiment 2 of the present invention. Detailed Implementation
[0020] To clearly illustrate the technical features of this solution, the invention will be described in detail below through specific embodiments and in conjunction with the accompanying drawings. The following disclosure provides many different embodiments or examples for implementing different structures of the invention. To simplify the disclosure of the invention, components and arrangements of specific examples are described below. Furthermore, reference numerals and / or letters may be repeated in different examples. This repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed. It should be noted that the components illustrated in the drawings are not necessarily drawn to scale. Descriptions of well-known components, processing techniques, and processes are omitted in this invention to avoid unnecessarily limiting the invention.
[0021] Example 1
[0022] Embodiment 1 of this invention proposes a method for predicting the trajectory of an object, which solves the core technical problem in the prior art that it is impossible to accurately predict sudden movements such as explosions, bounces, and collisions.
[0023] The method for predicting the trajectory of an object proposed in Embodiment 1 of this invention can be used in the following technical fields: robot navigation and obstacle avoidance, embodied interaction and virtual reality, dynamic 3D generation and autonomous driving, etc.
[0024] In the field of robot navigation and obstacle avoidance: it helps robots accurately predict the movement trajectory of dynamic obstacles and achieve safe and efficient path planning;
[0025] In the fields of embodied interaction and virtual reality: it supports the accurate response of virtual objects to the real physical environment, enhancing the immersive interactive experience;
[0026] In the field of dynamic 3D generation: it provides high-fidelity dynamic scene generation capabilities for film and television special effects production, game development, and virtual simulation training;
[0027] In the field of autonomous driving: predicting the movement trajectories of dynamic targets such as pedestrians and vehicles to improve the safety of autonomous driving systems.
[0028] The scope of protection of Embodiment 1 of the present invention is not limited to the specific fields listed in Embodiment 1, and those skilled in the art can make reasonable selections according to the actual situation.
[0029] Figure 1 This is a flowchart of a method for predicting the trajectory of an object according to Embodiment 1 of the present invention; Figure 2 This is an architecture diagram of a method for predicting the trajectory of an object proposed in Embodiment 1 of the present invention; combined with Figure 1 and Figure 2 The process of implementing this invention will be explained together.
[0030] In step S1, a two-dimensional image sequence of the target object is obtained to construct an observation frame set; the initial frame in the observation frame set is used to perform three-dimensional reconstruction of the target object to obtain an initial three-dimensional Gaussian particle set.
[0031] For the target object, given a set of two-dimensional observation data These data are from a time step The images were collected from multiple perspectives, and our goal is to predict the object's subsequent movement through dynamic modeling. Motion within a time step. For clarity, we acquire one frame at each time step and will... The frame is called the observation frame, where Indicates the initial frame, subsequent frames The frame is called the prediction frame.
[0032] The process by which this application uses the initial frame in the observation frame set to perform three-dimensional reconstruction of the target object to obtain the initial three-dimensional Gaussian particle set is as follows: [The process is described in the original text, which is not directly related to the preceding sentence.] The original 3DGS algorithm is applied to obtain the initial three-dimensional Gaussian parameters. The 3DGS algorithm is a three-dimensional Gaussian sputtering algorithm.
[0033] The process of using the 3D Gaussian sputtering algorithm is as follows: the 2D image of the initial frame is input into the differentiable rendering framework, and the parameters of multiple Gaussian particles in 3D space are iteratively optimized through backpropagation and gradient descent; each Gaussian particle includes at least 3D position, 3D rotation, 3D scaling, spherical harmonic coefficient and opacity;
[0034] After optimization, a material feature vector and physical code are assigned to each Gaussian particle to obtain an initial three-dimensional Gaussian particle set.
[0035] The initial set of three-dimensional Gaussian particles includes: three-dimensional positions describing spatial properties. ;
[0036] 3D rotation ,in It is the real part of the quaternion. It is the imaginary part of the quaternion; a three-dimensional scaling factor describing appearance attributes. spherical harmonic coefficients describing particle color and opacity For convenience, we also refer to the parameterized three-dimensional Gaussian function as a particle. The spherical harmonic coefficients used to describe the particle's color... and opacity It does not change with the movement of the object, so it is obtained from the initial frame in subsequent dynamic modeling and prediction. and This remains unchanged. Furthermore, we assign a learnable vector to each particle. Used to characterize the material properties of particles, enhancing the model's ability to model the different material dynamic properties exhibited by heterogeneous objects in dynamic interactions. To describe the underlying physical information such as particle mass and forces, a latent physical code is shared across all frames. It characterizes the type of motion of a particle over the entire time period.
[0037] The physical encoding process involves inputting the three-dimensional position of the initial three-dimensional Gaussian particle into the first multilayer perceptron network for prediction.
[0038] ;
[0039] in, Represents physical encoding. ; This represents the first multilayer perceptron network; This represents the three-dimensional position of the initial three-dimensional Gaussian particle;
[0040] At this point, the parameter set corresponding to all Gaussian particles in the initial frame is represented as:
[0041] .
[0042] A Multilayer Perceptron (MLP) is a classic feedforward artificial neural network model consisting of an input layer, one or more hidden layers, and an output layer. Each layer contains several neurons, and neurons in adjacent layers are connected via fully connected connections. MLPs introduce nonlinear transformations through nonlinear activation functions, enabling them to learn complex mappings from input data to the output target.
[0043] In step S2, the deformation-assisted network predicts the relative displacement, relative rotation, and relative scaling of the particles from the initial frame to each observation frame based on the three-dimensional position, physical encoding, object geometric semantic encoding, and timestamp corresponding to the target observation frame of the initial three-dimensional Gaussian particle. The relative displacement, relative rotation, and relative scaling are then used to transform the three-dimensional position, three-dimensional rotation, and three-dimensional scaling of the initial three-dimensional Gaussian particle to obtain the particle set corresponding to each observation frame.
[0044] Because, without any prior knowledge and only with the supervision of sparse multi-view images, arbitrary observation frames are directly optimized from the initial frame. The velocity field of particles presents significant challenges, leading to considerable difficulty in model convergence. Therefore, we construct a material-aware deformation-assisted field to obtain a well-initialized velocity field.
[0045] Therefore, through a deformation-assisted network, this application predicts the relative displacement, relative rotation, and relative scaling of the particle from the initial frame to each observation frame based on the initial 3D Gaussian particle's 3D position, physical encoding, object geometric semantic encoding, and the timestamp corresponding to the target observation frame; specifically:
[0046] ;
[0047] in, This represents the three-dimensional position of the initial three-dimensional Gaussian particle; Indicates the timestamp corresponding to the target observation frame; Represents physical encoding; Represents the geometric semantic encoding of an object; This indicates a deformation-assisted network employing a second-layer perceptron network; This represents the predicted relative displacement; This indicates the predicted relative rotation; This indicates the relative scaling obtained from the prediction;
[0048] therefore The corresponding parameter set of Gaussian particles .
[0049] The initial 3D Gaussian particles are transformed in terms of their 3D position, 3D rotation, and 3D scaling to obtain the particle set corresponding to each observation frame as follows:
[0050] ;
[0051] in, Represents the observation frame obtained after transformation. The corresponding three-dimensional position of the particle; Represents the observation frame obtained after transformation. The corresponding three-dimensional rotation of the particle; Represents the observation frame obtained after transformation. The corresponding three-dimensional scaling of the particles; Represents quaternion multiplication; This indicates element-wise multiplication.
[0052] Object geometric semantic encoding The process is as follows: the initial frame is processed by the PointNet++ encoder. The three-dimensional position set of all three-dimensional Gaussian particles is encoded.
[0053] By adding semantic encoding related to the object's appearance in the deformation-assisted field Enhanced Accuracy in predicting cross-frame deformation.
[0054] In step S3, the three-dimensional positions of the particles in the particle set corresponding to each observation frame are input into the velocity field prediction network. The velocity field prediction network generates the velocity field of the particles in the current observation frame and updates the three-dimensional positions of the particles according to the velocity field to obtain the particle set corresponding to the future frame.
[0055] By constructing a Gaussian velocity field that is aware of motion state, we can learn the dynamic characteristics of particles in dynamic interactions.
[0056] by The corresponding parameter set of Gaussian particles Initially, the Gaussian velocity field of the motion state perception is used to solve for the observed frames. Corresponding divergence-free velocity field ; indicates as:
[0057] ;
[0058] in, Indicates the particle in the observation frame The three-dimensional spatial coordinates at that time; The fundamental velocity components include those along the path. , and linear velocity components of the axis , and and around , and angular velocity components of the axis , and ; Let the basis vector matrix be represented as:
[0059] ;
[0060] in, Indicates the particle in the observation frame The three-dimensional position coordinates;
[0061] divergence-free velocity field While not relying on inefficient physical information neural network loss, it ensures that the velocity field of each particle satisfies the divergence-free property to ensure that the extrapolated future frame has physical meaning, that is, the particle will not appear or disappear out of thin air in the velocity field, so as to satisfy the volume conservation.
[0062] The motion state perception module in the velocity field prediction network uses timestamps... Spacing between interactive objects Current speed of movement and material feature vector The basic velocity field is modulated to obtain a Gaussian velocity field for motion state perception. ;
[0063] Based on the Gaussian velocity field of motion state perception Multiplied by a very small time interval ,from The three-dimensional spatial coordinates of the particle at time t. Inferring future frames particle three-dimensional space coordinates Thus obtaining future frames The set of particles ;
[0064] in, Indicates particles in future frames Three-dimensional rotation at time; Indicates particles in future frames At that time , and 3D scaling along the axis; Spherical harmonic coefficients representing particle color; Indicates opacity.
[0065] Basic velocity components Obtained by multiplying the motion pattern decomposition vector and the weight vector, it is expressed as:
[0066]
[0067] in, Represents the motion mode decomposition vector Through a third-layer perceptron network For the physical encoding Decoded; The weight vector is passed through a fourth-layer perceptron network. timestamp Decoded.
[0068] Among them, the physical code describing the motion throughout the entire time period Using a third-layer perceptron network Decoded Various motion patterns, and observation frames The corresponding timestamps are passed through a weighted network. Get a set of weights Used to query a specific observation frame The movement pattern below.
[0069] Furthermore, in order to solve the divergence-free velocity field Due to the limitation of not being able to effectively predict sudden movements such as explosions, bounces, and collisions, we... Introducing motion state sensing function based on Used to extend the original divergence-free velocity field The ability to perceive sudden changes in system state caused by sudden motion. Among these, This indicates a motion state sensing encoder employing a fifth-layer perceptron network. Specifically, we designed a two-layer MLP with a GELU activation function as... The encoder is constructed, and three independent branches are built: , and These are three relatively independent branches of the fifth-layer perceptron network; specifically:
[0070] ;
[0071] ;
[0072] ;
[0073] in, This indicates a motion state sensing encoder using a fifth-layer perceptron network; Represents the encoded motion state features; Used for decoding system state characterization. ; Used for decoding the fundamental dissipation coefficient ; Used for decoding timing decay factor ; Indicates the system state modulation amount; The Gaussian velocity field represents the perception of motion state.
[0074] To compensate for the existence of three-dimensional real space and The dimensionality gap with other inputs is encoded using two independent multilayer perceptron networks. and For Learning from networks. Utilizing motion state sensing functions. Decoded system state modulation and Multiplication, that is We will use the divergence-free velocity field Extended to a Gaussian velocity field for motion state perception, capable of sensing sudden changes in system state. .
[0075] Gaussian velocity field based on motion state perception with the ability to sense sudden changes in system state We multiply it by a very small time interval ,from The three-dimensional spatial coordinates of the particle at time t. Inferring future frames particle three-dimensional space coordinates Thus obtaining future frames The set of particles .
[0076] In step S4, the particle set corresponding to the future frame is projected onto a two-dimensional image space. A loss function is constructed based on the difference between the rendered image and the actual observed image. The particle set corresponding to the future frame is optimized based on the loss function to obtain the optimized particle set as the predicted object motion trajectory.
[0077] The method of this invention further includes:
[0078] Future frames The corresponding set of particles Considered as a three-dimensional Gaussian kernel, the three-dimensional Gaussian sputtering algorithm is applied to project the three-dimensional Gaussian particles into the two-dimensional pixel space to obtain the rendered image;
[0079] With future frames corresponding Real images captured from various perspectives To supervise the process, a visual training loss constraint is constructed; the visual training loss constraint includes the L1 loss between the rendered image and the real observed image. And structural similarity index (SSIM) loss .
[0080] The loss function also includes the same material constraint and the time-series decay constraint;
[0081] The same material constraint is used to constrain the basic dissipation coefficient. To ensure that the basic dissipation coefficients of particles of the same material tend to be consistent, it is expressed as:
[0082] ;
[0083] in, Indicates a constraint based on the same material; Denotes the part of the same material in a heterogeneous object or any part of a homogeneous object. The fundamental dissipation coefficient of a Gaussian particle; Denotes the part of the same material in a heterogeneous object or any part of a homogeneous object. The fundamental dissipation coefficient of a Gaussian particle; Represents the expectation operator;
[0084] In the real world, inelastic dissipation from collisions is "cumulative and irreversible," meaning that the kinetic energy of each collision is converted into heat, plastic deformation energy, sound energy, etc. Therefore, an object's "energy retention capacity" continuously decreases over time. Decoded timing decay factor It should satisfy the requirement of time Monotonically decreasing monotonicity constraint. Correspondingly, we designed a time-series decay constraint.
[0085] The time-series decay constraint is used to constrain the time-series decay factor. To ensure that the time-series decay factor decreases monotonically with time, which conforms to the cumulative irreversibility of inelastic dissipation due to collisions, it is expressed as:
[0086] ;
[0087] in, Indicates timing decay constraints; Indicates future frames The corresponding time-series decay factor; Indicates future frames The corresponding time decay factor.
[0088] ,Right now This refers to the observation frame that is later in the temporal sequence. By combining visual training loss constraints, same material constraints, and temporal decay constraints, we have achieved constraints on the entire dynamic modeling framework:
[0089] .
[0090] Represents the initial three-dimensional Gaussian particle set; This represents the first multilayer perceptron network; This represents a third-layer perceptron network; This represents a third-layer perceptron network; This indicates a deformation-assisted network employing a second-layer perceptron network; This indicates a motion state sensing encoder using a fifth-layer perceptron network; These are the first branch of the fifth multilayer perceptron network.
[0091] In this invention, all multilayer perceptron networks are trained end-to-end. The gradient of the loss function with respect to the network parameters is calculated using the backpropagation algorithm, and the parameters are updated using a gradient descent optimizer (such as Adam). The loss function includes visual reconstruction loss (L1 loss and SSIM loss) and physical constraint loss (same material constraint and temporal decay constraint), which together guide the parameter optimization of all MLP networks.
[0092] This invention can effectively predict sudden movements such as explosions, bounces, and collisions, improving the accuracy and robustness of dynamic scene dynamics modeling.
[0093] Example 2
[0094] Embodiment 2 of the present invention also proposes a device for predicting the trajectory of an object. Figure 3 This is a schematic diagram of an object motion trajectory prediction device proposed in Embodiment 2 of the present invention.
[0095] At the hardware level, electronic device 300 includes a processor 310, and optionally, an internal bus 320, a network interface 330, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or it may also include non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for its functions.
[0096] The processor 310, network interface 330, and memory can be interconnected via an internal bus 320. This internal bus 320 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be categorized as an address bus, data bus, control bus, etc. For ease of illustration, only a single bidirectional arrow is used in this diagram, but this does not imply that there is only one bus or one type of bus. The memory is used to store programs. Specifically, the program can include program code, which includes computer operation instructions. The memory can include main memory 340 and non-volatile memory 350, and provides instructions and data to the processor 310.
[0097] Processor 310 reads the corresponding computer program from non-volatile memory 350 into memory 340 and then runs it, forming a device for locating the target user at the logical level. Processor 310 executes the program stored in memory and specifically performs the following:
[0098] In step S1, a two-dimensional image sequence of the target object is obtained to construct an observation frame set; the initial frame in the observation frame set is used to perform three-dimensional reconstruction of the target object to obtain an initial three-dimensional Gaussian particle set.
[0099] In step S2, the deformation-assisted network predicts the relative displacement, relative rotation, and relative scaling of the particles from the initial frame to each observation frame based on the three-dimensional position, physical encoding, object geometric semantic encoding, and timestamp corresponding to the target observation frame of the initial three-dimensional Gaussian particle. The relative displacement, relative rotation, and relative scaling are then used to transform the three-dimensional position, three-dimensional rotation, and three-dimensional scaling of the initial three-dimensional Gaussian particle to obtain the particle set corresponding to each observation frame.
[0100] In step S3, the three-dimensional positions of the particles in the particle set corresponding to each observation frame are input into the velocity field prediction network. The velocity field prediction network generates the velocity field of the particles in the current observation frame and updates the three-dimensional positions of the particles according to the velocity field to obtain the particle set corresponding to the future frame.
[0101] In step S4, the particle set corresponding to the future frame is projected onto a two-dimensional image space. A loss function is constructed based on the difference between the rendered image and the actual observed image. The particle set corresponding to the future frame is optimized based on the loss function to obtain the optimized particle set as the predicted object motion trajectory.
[0102] This device can be integrated into the onboard computer of a drone to achieve real-time target detection; it can also be used as a ground station to process images transmitted back by the drone offline.
[0103] Figure 1 It can be applied to processor 310, or implemented by processor 310. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit in the processor or by instructions in the form of software. The processor mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.
[0104] Of course, in addition to software implementation, the electronic device of this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0105] Example 3
[0106] The present invention also proposes a readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, implements the following method steps:
[0107] In step S1, a two-dimensional image sequence of the target object is obtained to construct an observation frame set; the initial frame in the observation frame set is used to perform three-dimensional reconstruction of the target object to obtain an initial three-dimensional Gaussian particle set.
[0108] In step S2, the deformation-assisted network predicts the relative displacement, relative rotation, and relative scaling of the particles from the initial frame to each observation frame based on the three-dimensional position, physical encoding, object geometric semantic encoding, and timestamp corresponding to the target observation frame of the initial three-dimensional Gaussian particle. The relative displacement, relative rotation, and relative scaling are then used to transform the three-dimensional position, three-dimensional rotation, and three-dimensional scaling of the initial three-dimensional Gaussian particle to obtain the particle set corresponding to each observation frame.
[0109] In step S3, the three-dimensional positions of the particles in the particle set corresponding to each observation frame are input into the velocity field prediction network. The velocity field prediction network generates the velocity field of the particles in the current observation frame and updates the three-dimensional positions of the particles according to the velocity field to obtain the particle set corresponding to the future frame.
[0110] In step S4, the particle set corresponding to the future frame is projected onto a two-dimensional image space. A loss function is constructed based on the difference between the rendered image and the actual observed image. The particle set corresponding to the future frame is optimized based on the loss function to obtain the optimized particle set as the predicted object motion trajectory.
[0111] Embodiment 3 of this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory that stores a computer program, which can be executed by a processor to complete the steps described in the aforementioned method. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.
[0112] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks. Alternatively, if the integrated units of this application are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.
[0113] The object motion trajectory prediction device provided in Embodiment 2 of this application and the object motion trajectory prediction storage medium proposed in Embodiment 3 of this application can be described in the detailed description of the corresponding part in the object motion trajectory prediction method provided in Embodiment 1 of this application, and will not be repeated here.
[0114] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that the elements inherent in a process, method, article, or apparatus that includes a list of elements are included. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Additionally, portions of the technical solutions provided in the embodiments of this application that are consistent with the implementation principles of corresponding technical solutions in the prior art have not been described in detail to avoid excessive elaboration.
[0115] While specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art can make other modifications or variations based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.
Claims
1. A method for predicting the trajectory of an object, characterized in that, Includes the following steps: Obtain a two-dimensional image sequence of the target object and construct an observation frame set; use the initial frame in the observation frame set to perform three-dimensional reconstruction of the target object to obtain an initial three-dimensional Gaussian particle set; Using a deformation-assisted network, the relative displacement, relative rotation, and relative scaling of the particle from the initial frame to each observation frame are predicted based on the initial 3D Gaussian particle's 3D position, physical encoding, object geometric semantic encoding, and timestamp corresponding to the observation frame. The relative displacement, relative rotation, and relative scaling are then used to transform the initial 3D Gaussian particle's 3D position, 3D rotation, and 3D scaling, respectively, to obtain the particle set corresponding to each observation frame. The three-dimensional positions of particles in the particle set corresponding to each observation frame are input into the velocity field prediction network. The velocity field prediction network generates the velocity field of the particles in the current observation frame, and updates the three-dimensional positions of the particles according to the velocity field to obtain the particle set corresponding to the future frame; specifically: observation frame The three-dimensional position of the particles in the corresponding particle set Input velocity field prediction network; The solution is obtained through the divergence-free velocity field decomposition module in the velocity field prediction network. Corresponding divergence-free velocity field ; indicates as: ； in, Indicates the particle in the observation frame The three-dimensional spatial coordinates at that time; The fundamental velocity components include those along the path. , and linear velocity components of the axis , and and around , and angular velocity components of the axis , and ; Let the basis vector matrix be represented as: ； in, Indicates the particle in the observation frame The three-dimensional position coordinates; The motion state perception module in the velocity field prediction network uses timestamps... Spacing between interactive objects Current speed of movement and material feature vector Modulating the fundamental velocity field yields a Gaussian velocity field for motion state perception. ; Based on the Gaussian velocity field of motion state perception Multiplied by a very small time interval ,from The three-dimensional spatial coordinates of the particle at time t. Inferring future frames particle three-dimensional space coordinates Thus obtaining future frames The set of particles ; in, Indicates particles in future frames Three-dimensional rotation at time; Indicates particles in future frames At that time , and 3D scaling along the axis; Spherical harmonic coefficients representing particle color; Indicates opacity; Represents physical encoding; The particle set corresponding to the future frame is projected into a two-dimensional image space. A loss function is constructed based on the difference between the rendered image and the actual observed image. The particle set corresponding to the future frame is optimized based on the loss function, and the optimized particle set is used as the predicted object motion trajectory.
2. The method for predicting the trajectory of an object according to claim 1, characterized in that, The initial three-dimensional Gaussian particle set is obtained by using the initial frame in the observation frame set to perform three-dimensional reconstruction of the target object. Specifically: The initial 2D image is input into a differentiable rendering framework, and the parameters of multiple Gaussian particles in 3D space are iteratively optimized through backpropagation and gradient descent; each Gaussian particle includes at least 3D position, 3D rotation, 3D scaling, spherical harmonic coefficients, and opacity. After optimization, a material feature vector and physical code are assigned to each Gaussian particle to obtain the initial three-dimensional Gaussian particle set.
3. The method for predicting the trajectory of an object according to claim 1, characterized in that, The physical encoding process is as follows: the three-dimensional position of the initial three-dimensional Gaussian particle is input into the first multilayer perceptron network for prediction; ； in, Represents physical encoding. ; This represents the first multilayer perceptron network; This represents the three-dimensional position of the initial three-dimensional Gaussian particle; The object's geometric semantic encoding process is as follows: the three-dimensional position set of all three-dimensional Gaussian particles in the initial frame is encoded by an encoder.
4. The method for predicting the trajectory of an object according to claim 3, characterized in that, Using a deformation-assisted network, based on the initial 3D Gaussian particle's 3D position, physical encoding, object geometric semantic encoding, and the timestamp corresponding to the observation frame, the relative displacement, relative rotation, and relative scaling of the particle from the initial frame to each observation frame are predicted; specifically: ； in, This represents the three-dimensional position of the initial three-dimensional Gaussian particle; Indicates the timestamp corresponding to the target observation frame; Represents physical encoding; Represents the geometric semantic encoding of an object; This indicates a deformation-assisted network employing a second-layer perceptron network; This represents the predicted relative displacement; This indicates the predicted relative rotation; This indicates the relative scaling obtained from the prediction; The initial 3D Gaussian particles are transformed in terms of their 3D position, 3D rotation, and 3D scaling to obtain the particle set corresponding to each observation frame as follows: ； in, Represents the observation frame obtained after transformation. The corresponding three-dimensional position of the particle; Represents the observation frame obtained after transformation. The corresponding three-dimensional rotation of the particle; Represents the observation frame obtained after transformation. The corresponding three-dimensional scaling of the particles; Represents quaternion multiplication; This indicates element-wise multiplication.
5. The method for predicting the trajectory of an object according to claim 1, characterized in that, The basic velocity components Obtained by multiplying the motion pattern decomposition vector and the weight vector, it is expressed as: ； in, Represents the motion pattern decomposition vector; through a third multilayer perceptron network. For the physical encoding Decoded; The weight vector is passed through a fourth-layer perceptron network. timestamp Decoded.
6. The method for predicting the trajectory of an object according to claim 5, characterized in that, The motion state perception module modulates the basic velocity field based on the timestamp, the distance between interactive objects, the current motion speed, and the material feature vector to obtain the Gaussian velocity field for motion state perception. The process is represented as follows: ； ； ； ； in, This indicates a motion state sensing encoder using a fifth-layer perceptron network; Represents the encoded motion state features; Used for decoding system state characterization ; Used for decoding the fundamental dissipation coefficient ; Used for decoding timing decay factor ; Indicates the system state modulation amount; The Gaussian velocity field represents the perception of motion state; where , and These are three relatively independent branches of the fifth-layer perceptron network.
7. The method for predicting the trajectory of an object according to claim 6, characterized in that, The method further includes: Future frames The corresponding set of particles Considered as a three-dimensional Gaussian kernel, the three-dimensional Gaussian sputtering algorithm is applied to project the three-dimensional Gaussian particles into the two-dimensional pixel space to obtain the rendered image; With future frames corresponding Real images captured from various perspectives To supervise the process, a visual training loss constraint is constructed; the visual training loss constraint includes L1 loss and structural similarity loss between the rendered image and the real observed image.
8. The method for predicting the trajectory of an object according to claim 7, characterized in that, The loss function also includes the same material constraint and the time-series decay constraint; The same material constraint is used to constrain the basic dissipation coefficient. To ensure that the basic dissipation coefficients of particles of the same material tend to be consistent, it is expressed as: ； in, Indicates a constraint based on the same material; The first part of a heterogeneous object is a portion of the same material, or any part of a homogeneous object is a portion of the same material. The fundamental dissipation coefficient of a Gaussian particle; The first part of a heterogeneous object is a portion of the same material, or any part of a homogeneous object is a portion of the same material. The fundamental dissipation coefficient of a Gaussian particle; Represents the expectation operator; The time-series decay constraint is used to constrain the time-series decay factor. To ensure that the time-series decay factor monotonically decreases with time, conforming to the cumulative irreversibility of inelastic dissipation due to collisions, it is expressed as: ； in, Indicates timing decay constraints; Indicates future frames The corresponding time-series decay factor; Indicates future frames The corresponding time decay factor.
9. A device for predicting the trajectory of an object, comprising at least one processor and a memory, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the at least one processor, it implements a method for predicting the trajectory of an object as described in any one of claims 1 to 8.