A method and system for fine efficient motion-guided four-dimensional content generation
By combining an interactive user interface and a multi-view image diffusion model with a four-dimensional Gaussian reconstruction model and a vector consistency loss function, the problems of long generation time and inaccurate motion trajectory control in existing technologies are solved, achieving efficient and realistic four-dimensional content generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TSINGHUA UNIVERSITY
- Filing Date
- 2026-01-26
- Publication Date
- 2026-06-09
AI Technical Summary
Existing technologies suffer from problems such as long processing time, reliance on professional manual operation, and inaccurate motion trajectory control when generating high-quality four-dimensional content. In particular, text- or image-based prompts cannot accurately define motion trajectories and amplitudes, making it difficult to fully capture user intent.
A sophisticated and efficient motion-guided four-dimensional content generation method is adopted. Motion prompts are received through an interactive user interface, and four-view static images are generated using a multi-view image diffusion model. The motion prompts are then converted into trajectory image tensors. Combined with a four-dimensional Gaussian reconstruction model and a vector consistency loss function, physically reasonable dynamic geometric modeling is achieved.
It achieves efficient four-dimensional content generation based on a single two-dimensional image and user motion cues. The generated animation is realistic, smooth, and accurately follows the expected trajectory. It has high operating efficiency and can complete three forward propagations within 1 second to generate highly realistic dynamic three-dimensional sequences.
Smart Images
Figure CN122176155A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of machine learning technology, and in particular to a sophisticated and efficient method and system for generating four-dimensional content guided by motion. Background Technology
[0002] Generating high-quality four-dimensional content (i.e., three-dimensional models with dynamic temporal evolution) is crucial for many fields, from animation to virtual reality. However, current processes suffer from bottlenecks such as being time-consuming and reliant on professional manual operation, which has driven academia to make extensive efforts to develop automated solutions.
[0003] One promising research direction is the generation of 3D assets for animation, which automates the 3D modeling process through automatic skeleton rigging [1-3] or parametric modeling. However, both of these methods have significant limitations in terms of motion degrees of freedom or object types. Most research on 4D generation without skeleton rigging and parametric modeling focuses on 4D reconstruction from 2D video. These works can effectively and accurately upgrade 2D sequences to 3D space, reducing the reliance on professional artists, but their limitation lies in the requirement to rely on existing video footage of the target motion.
[0004] Recent advancements have pioneered a new subfield: "cued 4D generation," which utilizes text or image input to automatically generate 2D videos for 4D reconstruction. These text / image-guided methods offer users greater flexibility in generating desired 4D content. While the field of 4D generation has evolved towards user-guided methods, current text- or image-based cues have fundamental limitations: they cannot precisely define motion trajectories, speeds, or amplitudes, making it difficult to fully capture the user's intent. To address this deficiency, this research aims to introduce novel control mechanisms.
[0005] Generating four-dimensional content based on motion cues requires addressing three core challenges: (1) transforming a single two-dimensional image into a coherent and temporally consistent three-dimensional representation; (2) efficiently fusing abstract or sparse motion cues; and (3) synthesizing high-quality, multi-view consistent, and physically plausible animations that conform to user intent. This invention, building upon the basic video-to-four-dimensional reconstruction model, proposes a motion cue injection mechanism that directly encodes the trajectory of arbitrary points (without requiring additional modules such as ControlNet), successfully addressing these challenges while avoiding extra parameters and computational overhead.
[0006] Given that appearance-based learning alone is insufficient for motion supervision.
[0007] Existing control methods all use text / images as cues, primarily relying on a two-stage process: video generation guided by text or images through a diffusion model, followed by an offline 4D reconstruction process. TC4D employs deformable 3D neural radiation fields (NeRF) to represent the scene and uses control points to constrain the global motion trajectory of objects. However, a key limitation of this method is its inability to achieve fine-grained object-level control. Although these methods can generate high-quality results, they suffer from two major drawbacks: 1) slow inference speed due to limitations imposed by the diffusion process, optimization-based reconstruction, or the iterative nature of NeRF representation; 2) inability to precisely control motion trajectory and amplitude using coarse-grained text / image cues. Summary of the Invention
[0008] The present invention aims to at least partially solve one of the technical problems in the related art.
[0009] To address this, this invention proposes a refined and efficient motion-guided four-dimensional content generation method. It presents a motion-conditional four-dimensional generation model that uses a feedforward approach to generate smooth and realistic three-dimensional Gaussian sequences from a single image and motion cues. The method employs point trajectories (defined as spatiotemporal point sequences) to represent motion, offering advantages such as adjustable point count, precise spatiotemporal control, and the ability to represent any type of motion, thereby comprehensively capturing user intent.
[0010] Another objective of this invention is to propose a precise and efficient motion-guided four-dimensional content generation system.
[0011] The third objective of this invention is to provide a computer device.
[0012] A fourth objective of this invention is to provide a non-transitory computer-readable storage medium.
[0013] To achieve the above objectives, this invention proposes a precise and efficient motion-guided four-dimensional content generation method, comprising:
[0014] Receive exercise prompts through an interactive user interface; Acquire a single input image of the target object, and use a multi-view image diffusion model to generate a four-view static image according to a fixed pitch angle and a preset orthogonal azimuth angle; The motion prompt information is converted into a trajectory image. A unique RGB value is assigned to each trajectory and filled into the spatiotemporal coordinate points to generate a trajectory image tensor. The trajectory image tensor is then stitched together with four-view static images along the time dimension to construct the model input features. The input features of the model are fed into the four-dimensional Gaussian reconstruction model to predict the complete three-dimensional Gaussian attribute set for the initial frame and only the geometric attributes of position and rotation for subsequent frames, so as to generate a temporally consistent three-dimensional Gaussian sequence. On the generated 3D Gaussian sequence, the appearance loss function, the rigid body loss function as much as possible, and the vector consistency loss function are calculated to jointly supervise the optimization of model parameters and achieve physically reasonable dynamic geometric modeling.
[0015] The precise and efficient motion-guided four-dimensional content generation method of this invention may also have the following additional technical features: In one embodiment of the present invention, receiving motion prompt information through an interactive user interface includes: Set the expected duration and use the brush tool to draw marks in the target area to select the placement area for the trajectory points; The system responds to user commands, automatically replaces the drawn markers with a uniformly distributed dot matrix, and precisely positions them on the object's surface; Choose either circular motion or translational motion mode, and repeat the process for multiple object groups to set animation trajectories for additional point sets.
[0016] In one embodiment of the present invention, the motion prompt information is converted into a trajectory image, a unique RGB value is assigned to each trajectory and filled into spatiotemporal coordinates to generate a trajectory image tensor, and the trajectory image tensor is concatenated with four-view static images along the time dimension to construct model input features, including: Obtain the motion prompt information specified by the user in the form of coordinates of each time step, assign a unique RGB value to each motion trajectory by random sampling, and fill the RGB values corresponding to each spatiotemporal coordinate point into the initially empty trajectory image. When multiple motion trajectories overlap in space, their RGB embedding values are superimposed and fused to generate a trajectory image tensor. The trajectory image tensor is stitched together with the four-view static images along the time dimension, wherein the main view uses the trajectory image tensor and the auxiliary view uses the initial static image copied and filled along the time dimension. The concatenated RGB tensor is fused with the Plück ray embedding in the channel dimension to construct model input features that include RGB values, ray origin, and ray direction.
[0017] In one embodiment of the present invention, the trajectory image tensor is stitched together with four-view static images along the time dimension, wherein the main viewpoint uses the trajectory image tensor, and the auxiliary viewpoints are filled by copying and pasting the initial static image along the time dimension, including: The trajectory image tensor is used as the temporal input of the main viewpoint, where the main viewpoint corresponds to the observation viewpoint from which the user draws motion cues. The initial static image is copied and filled along the time dimension for the auxiliary viewpoint to generate the temporal input for the auxiliary viewpoint; The trajectory image tensor from the main viewpoint and the copied fill image from the auxiliary viewpoint are stitched together along the time dimension to construct a complete four-view temporal input feature.
[0018] In one embodiment of the present invention, for each pixel, its input feature vector is formed by concatenating the RGB values, ray origin and ray direction in the channel dimension. The ray origin and ray direction are calculated based on the camera intrinsic and extrinsic parameters and are used to characterize the three-dimensional spatial geometric information corresponding to the pixel. The pixel is a unified index of each pixel in the four-view static image, and each pixel corresponds to a three-dimensional spatial ray emanating from the optical center of the camera.
[0019] In one embodiment of the present invention, the model input features are fed into a four-dimensional Gaussian reconstruction model to predict a complete three-dimensional Gaussian attribute set for the initial frame, and only the geometric attributes of position and rotation for subsequent frames, in order to generate a temporally consistent three-dimensional Gaussian sequence, including: The U-Net architecture equipped with spatial and temporal attention layers is used as the basic framework of the four-dimensional Gaussian reconstruction model, and the input features of the model are input into the four-dimensional Gaussian reconstruction model to output the initial four-dimensional Gaussian order. A constraint strategy is applied to the initial four-dimensional Gaussian sequence: the complete Gaussian attribute set is predicted in the initial frame, and only the geometric attributes of position and rotation are predicted in subsequent frames, resulting in a constrained four-dimensional Gaussian sequence. Based on the characteristic that point trajectory data cannot reasonably infer changes in appearance attributes, the constrained four-dimensional Gaussian sequence is used as the final output to generate a time-consistent three-dimensional Gaussian sequence.
[0020] In one embodiment of the present invention, an appearance loss function, a rigidity loss function (as rigid as possible), and a vector consistency loss function are calculated on the generated three-dimensional Gaussian sequence to jointly supervise the optimization of model parameters and achieve physically reasonable dynamic geometric modeling, including: Based on the physical property that deformable motion can be decomposed into rigidly connected sub-components, a rigid body loss constraint is introduced to constrain the relative distance between pairs of three-dimensional Gaussian points within the same sub-component, so that the Gaussian points maintain rigid motion over time. To eliminate the ambiguity of rotation and translation caused by only constraining relative distance in rigid body loss, a vector consistency loss function is introduced to explicitly specify the translational motion mode by constraining the relative position vector between Gaussian mean point pairs. By combining appearance loss function, rigidity loss function, and vector consistency loss function, temporal supervised optimization of 3D Gaussian sequences is performed to achieve accurate and physically reasonable dynamic geometric modeling.
[0021] To achieve the above objectives, another aspect of the present invention proposes a sophisticated and efficient motion-guided four-dimensional content generation system, comprising: The prompt information receiving module is used to receive motion prompt information through an interactive user interface; The static image generation module is used to acquire a single input image of the target object and generate a four-view static image using a multi-view image diffusion model according to a fixed pitch angle and a preset orthogonal azimuth angle. The model input feature construction module is used to convert the motion prompt information into a trajectory image, assign a unique RGB value to each trajectory and fill it into the spatiotemporal coordinate points to generate a trajectory image tensor, and then stitch the trajectory image tensor with the four-view static image along the time dimension to construct the model input features. The 3D Gaussian sequence generation module is used to feed the model input features into the 4D Gaussian reconstruction model, predict the complete 3D Gaussian attribute set for the initial frame, and predict only the geometric attributes of position and rotation for subsequent frames, so as to generate a temporally consistent 3D Gaussian sequence. The joint supervision module is used to calculate the appearance loss function, the rigid body loss function as much as possible, and the vector consistency loss function on the generated 3D Gaussian sequence, so as to jointly supervise the optimization of model parameters and achieve physically reasonable dynamic geometric modeling.
[0022] The present invention provides a sophisticated and efficient motion-guided four-dimensional content generation method and system that proposes Vector Consistency Loss (VCL) to complement Rigid Body Loss (ARAP). While ARAP can facilitate local rigid body motion, it cannot distinguish between rotation and translation. VCL eliminates this ambiguity by providing explicit directional supervision. The synergistic effect of the two enables more accurate and coherent motion learning, ultimately generating physically plausible four-dimensional content that faithfully reflects the user's intent.
[0023] To achieve the above objectives, a third aspect of this application provides a computer device comprising a processor and a memory; wherein the processor runs a program corresponding to the executable program code stored in the memory, for implementing the fine and efficient motion-guided four-dimensional content generation method as described in the first aspect embodiment.
[0024] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements the fine and efficient motion-guided four-dimensional content generation method as described in the first aspect embodiment.
[0025] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0026] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1This is a flowchart of a method for generating four-dimensional content with precise and efficient motion guidance according to an embodiment of the present invention; Figure 2 This is a schematic diagram of model reasoning according to an embodiment of the present invention; Figure 3 This is a schematic diagram of user interface input according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a precision and efficient motion-guided four-dimensional content generation system according to an embodiment of the present invention; Figure 5 It is a computer device according to an embodiment of the present invention. Detailed Implementation
[0027] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.
[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0029] The following description, with reference to the accompanying drawings, outlines a method, system, device, and storage medium for generating precise and efficient motion-guided four-dimensional content according to embodiments of the present invention.
[0030] Figure 1 This is a flowchart of a method for generating four-dimensional content with precise and efficient motion guidance according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes, but is not limited to, the following steps: S1 receives motion prompts via an interactive user interface; S2, acquire a single input image of the target object, and generate a four-view static image using a multi-view image diffusion model according to a fixed pitch angle and a preset orthogonal azimuth angle; S3, the motion prompt information is converted into a trajectory image, a unique RGB value is assigned to each trajectory and filled into the spatiotemporal coordinate points to generate a trajectory image tensor, and the trajectory image tensor is stitched with the four-view static image along the time dimension to construct the model input features; S4, the input features of the model are fed into the four-dimensional Gaussian reconstruction model to predict the complete three-dimensional Gaussian attribute set for the initial frame and only the geometric attributes of position and rotation for subsequent frames, so as to generate a temporally consistent three-dimensional Gaussian sequence. S5 calculates the appearance loss function, the rigid body loss function, and the vector consistency loss function on the generated 3D Gaussian sequence to jointly supervise the optimization of model parameters and achieve physically reasonable dynamic geometric modeling.
[0031] Understandably, this invention is the first to propose a motion-conditional four-dimensional generation framework, capable of generating four-dimensional assets based on two-dimensional images and user motion cues. It proposes a novel cue injection method that does not require adding a cue processing module to the basic reconstruction model, achieving plug-and-play functionality through a representation called "trajectory image" in this invention. It also proposes a physically inspired novel Vector Consistency Loss (VCL), which effectively eliminates the ambiguity of ARAP loss and achieves high-precision Gaussian trajectory learning.
[0032] Specifically, given a static image and motion cues, this model can generate four-dimensional assets with realistic and smooth motion effects that accurately follow the expected trajectory. For example... Figure 2 As shown, during the inference process, this invention first acquires four initial images using an existing multi-view image diffusion model. Then, it encodes user-defined motion cues into a standardized trajectory image format. Finally, it elevates these inputs into a three-dimensional dynamic sequence using a U-Net-based architecture. During the training phase, the model's prediction results are jointly optimized under supervised supervision using appearance loss and geometric loss. The specific steps are as follows: In one embodiment of the present invention, a predetermined duration is set and a brush tool is used to draw marks in the target area to select the trajectory point placement area; the system responds to the user's operation command, automatically replaces the drawn marks with a uniformly distributed dot matrix and accurately positions them on the object surface; a circular motion or translational motion mode is selected, and the process is repeated for multiple object groups to set the animation trajectory for additional point sets.
[0033] Specifically, to translate user intent into trajectory prompts, this invention designs a simple and intuitive user interface, enabling users to define motion trajectories through drawing and dragging operations. For example... Figure 3 As shown, the workflow is as follows: The user first sets the expected duration to input motion cues; then, they can use a blue pen to draw in the target area to select the region where trajectory points need to be placed; after clicking "Place Points," the system automatically replaces these drawn markers with a uniformly distributed green dot matrix, which will be precisely positioned on the surface of the target object. The user can then choose different motion modes—either moving the point set along an arc around a pivot point, or directly dragging the point set as a whole. This process can be repeated for multiple object groups: the user can set animation trajectories for additional point sets by drawing new areas.
[0034] In one embodiment of the present invention, during the inference phase, the present invention employs an existing multi-view image diffusion model to generate four images according to a fixed pitch angle and a preset orthogonal azimuth angle, such as... Figure 2 The leftmost image area is shown.
[0035] In one embodiment of the present invention, motion prompt information specified by the user in the form of coordinates at each time step is obtained, a unique RGB value is assigned to each motion trajectory by random sampling, and the RGB values corresponding to each spatiotemporal coordinate point are filled into the initially empty trajectory image. When multiple motion trajectories overlap in space, their RGB embedding values are superimposed and fused to generate a trajectory image tensor. The trajectory image tensor is stitched with four-view static images along the time dimension, wherein the main view uses the trajectory image tensor, and the auxiliary view uses the initial static image copied and filled along the time dimension. The stitched RGB tensor is fused with Plück ray embedding in the channel dimension to construct model input features including RGB values, ray origin, and ray direction.
[0036] Specifically, the present invention first obtains motion prompt information (represented as tensors) specified by the user in the form of coordinates at each time step. Then, a unique RGB value is assigned to each motion trajectory by random sampling. Next, for each spatiotemporal coordinate point in the nth trajectory... This invention fills the initially empty trajectory image with its corresponding unique RGB value. When multiple motion trajectories spatially overlap, their corresponding RGB embedded values are superimposed and fused. The final generated trajectory image The images will be stitched together along the time dimension with the initial four-view images. It should be noted that, considering actual user input, this invention only collects trajectory prompt information from the main viewpoint; for other auxiliary viewpoints, this invention directly uses the initial static image copied and filled along the time dimension.
[0037] in, , , and Represent the input RGB tensor, static object image, temporal stitching operation, and the first... One perspective. This represents the main input perspective. Subsequently, these RGB tensors are concatenated with Plück ray embeddings along the channel dimension. Therefore, the model input can be represented as:
[0038] in, , , and These represent the input features, RGB values, ray origin, and ray direction, respectively. Indicates the pixel number.
[0039] In one embodiment of the present invention, a U-Net architecture equipped with spatial and temporal attention layers is used as the basic framework of the four-dimensional Gaussian reconstruction model. The input features of the model are input into the four-dimensional Gaussian reconstruction model to output an initial four-dimensional Gaussian sequence. A constraint strategy is applied to the initial four-dimensional Gaussian sequence: the complete Gaussian attribute set is predicted in the initial frame, and only the geometric attributes of position and rotation are predicted in subsequent frames to obtain a constrained four-dimensional Gaussian sequence. Based on the characteristic that point trajectory data cannot reasonably infer changes in appearance attributes, the constrained four-dimensional Gaussian sequence is used as the final output to generate a temporally consistent three-dimensional Gaussian sequence.
[0040] Specifically, the L4GM model based on the U-Net architecture is used as the basic framework. This model is equipped with spatial and temporal attention layers, which can elevate the input data into a corresponding four-dimensional Gaussian sequence. The complete Gaussian attribute set includes, but is not limited to, the three-dimensional position, scale, rotation quaternion, opacity, and spherical harmonic coefficient color of Gaussian points. However, to avoid the "Gaussian decay problem"—that is, when the model cannot learn physically reasonable motion that matches the motion cues, it will actively reduce the opacity of Gaussian points and shrink the spatial scale of Gaussian points to minimize the appearance rendering loss, resulting in loss of detail and geometric degradation in the generated dynamic content—this invention constrains the model to predict the complete Gaussian attribute set only in the initial frame. For subsequent frames, the model only predicts selected geometric attributes, specifically including position. and rotation amount Obtain the Gaussian attribute set of frame t. ,in , , The scale, opacity, and color attributes of the initial frame are represented respectively and remain fixed in all subsequent frames. Based on the characteristic that the appearance attribute changes cannot be reasonably inferred from the point trajectory data, the constrained four-dimensional Gaussian sequence is used as the final output to generate a time-consistent three-dimensional Gaussian sequence.
[0041] In one embodiment of the present invention, based on the physical characteristic that deformable motion can be decomposed into rigidly connected sub-components, a rigid-body loss function is introduced to constrain the relative distance between pairs of three-dimensional Gaussian points within the same sub-component, so that the Gaussian points maintain rigid motion over time. To eliminate the ambiguity of rotation and translation caused by the rigid-body loss function constraining only the relative distance, a vector consistency loss function is introduced to explicitly specify the translational motion mode by constraining the relative position vector between pairs of Gaussian mean points. The appearance loss function, the rigid-body loss function, and the vector consistency loss function are combined to perform time-supervised optimization of the three-dimensional Gaussian sequence, thereby achieving accurate and physically reasonable dynamic geometric modeling.
[0042] Specifically, although the L4GM framework can effectively reconstruct 3D Gaussian sequences using only appearance loss functions such as mean squared error (MSE) and learned perceptual patch similarity (LPIPS), its inherent limitations in dynamic geometric modeling hinder its performance on motion cueing tasks: the model often fails to learn effective motion for challenging cues, instead opting for a degenerate solution that deceives the appearance loss. Based on this, this invention, combined with observations from relevant literature, finds that most deformable motions in the real world can be decomposed into several sub-components exhibiting rigid connections. Based on this finding, this invention constrains 3D Gaussian points within the same sub-component to maintain rigid motion over time. Specifically, this invention achieves this constraint by introducing a physically inspired, as-rigid-as-as-area (ARAP) loss:
[0043] in, , and These represent the number of input frames, the set of all point pair indices, and... Rigid affinity score between points.
[0044] However, since ARAP loss only constrains the relative distance between point pairs, it is inherently ambiguous. This allows for degeneracy: for example, Gaussian point pairs that should undergo pure translation may rotate together while minimizing the loss, thus limiting the effectiveness of this constraint for motion learning. To eliminate this ambiguity, this invention carefully designs a novel loss function to explicitly specify translation as the sole motion mode. Specifically, this invention proposes Vector Consistency Loss (VCL), defined as:
[0045] in express Vector consistency affinity score for point pairs. Unlike ARAP loss, which modulates scalar distance, VCL achieves precise motion control by constraining the relative position vectors between Gaussian-mean point pairs.
[0046] The refined and efficient motion-guided four-dimensional content generation method according to embodiments of the present invention employs a highly expressive and efficient motion cue mechanism. The entire four-dimensional generation process can be completed in a single forward propagation, and end-to-end training requires no additional cue processing module, ensuring high throughput while achieving fine-grained control. The model of this invention is the first to achieve four-dimensional content generation based on a single two-dimensional image and motion cue. This framework explicitly learns motion patterns through vector consistency loss, and its performance significantly outperforms existing methods based on appearance features. The system exhibits excellent operating efficiency, completing three forward propagations within one second, and only requires a single image and user motion cue input through the simple and intuitive user interface of this invention to generate highly realistic dynamic three-dimensional sequences.
[0047] To achieve the above embodiments, such as Figure 4 As shown, this embodiment also provides a precise and efficient motion-guided four-dimensional content generation system 10, which includes: The prompt information receiving module 100 is used to receive motion prompt information through an interactive user interface; The static image generation module 200 is used to acquire a single input image of the target object and generate a four-view static image using a multi-view image diffusion model according to a fixed pitch angle and a preset orthogonal azimuth angle. The model input feature construction module 300 is used to convert the motion prompt information into a trajectory image, assign a unique RGB value to each trajectory and fill it into the spatiotemporal coordinate points to generate a trajectory image tensor, and stitch the trajectory image tensor with the four-view static image along the time dimension to construct the model input features. The three-dimensional Gaussian sequence generation module 400 is used to feed the model input features into the four-dimensional Gaussian reconstruction model, predict the complete three-dimensional Gaussian attribute set for the initial frame, and predict only the geometric attributes of position and rotation for subsequent frames, so as to generate a temporally consistent three-dimensional Gaussian sequence. The joint supervision module 500 is used to calculate the appearance loss function, the rigid body loss function as much as possible, and the vector consistency loss function on the generated 3D Gaussian sequence, so as to jointly supervise the optimization of model parameters and achieve physically reasonable dynamic geometric modeling.
[0048] The refined and efficient motion-guided four-dimensional content generation system according to embodiments of the present invention employs a highly expressive and efficient motion cue mechanism. The entire four-dimensional generation process can be completed in a single forward propagation, and end-to-end training requires no additional cue processing module, ensuring high throughput while achieving fine-grained control. The model of this invention is the first to achieve four-dimensional content generation based on a single two-dimensional image and motion cue. This framework explicitly learns motion patterns through vector consistency loss, and its performance significantly outperforms existing methods based on appearance features. The system exhibits excellent operating efficiency, completing three forward propagations within one second, and only requires a single image and user motion cue input through the simple and intuitive user interface of this invention to generate highly realistic dynamic three-dimensional sequences.
[0049] To implement the methods of the above embodiments, the present invention also provides a computer device, such as... Figure 5 As shown, the computer device 600 includes a memory 601 and a processor 602; wherein, the processor 602 reads executable program code stored in the memory 601 to run a program corresponding to the executable program code, so as to implement the various steps of the method described above.
[0050] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in the foregoing embodiments.
[0051] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0052] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.
Claims
1. A precise and efficient motion-guided four-dimensional content generation method, characterized in that, include: Receive exercise prompts through an interactive user interface; Acquire a single input image of the target object, and use a multi-view image diffusion model to generate a four-view static image according to a fixed pitch angle and a preset orthogonal azimuth angle; The motion prompt information is converted into a trajectory image. A unique RGB value is assigned to each trajectory and filled into the spatiotemporal coordinate points to generate a trajectory image tensor. The trajectory image tensor is then stitched together with four-view static images along the time dimension to construct the model input features. The input features of the model are fed into the four-dimensional Gaussian reconstruction model to predict the complete three-dimensional Gaussian attribute set for the initial frame and only the geometric attributes of position and rotation for subsequent frames, so as to generate a temporally consistent three-dimensional Gaussian sequence. On the generated 3D Gaussian sequence, the appearance loss function, the rigid body loss function as much as possible, and the vector consistency loss function are calculated to jointly supervise the optimization of model parameters and achieve physically reasonable dynamic geometric modeling.
2. The method according to claim 1, characterized in that, Receive motion cues via an interactive user interface, including: Set the expected duration and use the brush tool to draw marks in the target area to select the placement area for the trajectory points; The system responds to user commands, automatically replaces the drawn markers with a uniformly distributed dot matrix, and precisely positions them on the object's surface; Choose either circular motion or translational motion mode, and repeat the process for multiple object groups to set animation trajectories for additional point sets.
3. The method according to claim 1, characterized in that, The motion cues are converted into trajectory images. A unique RGB value is assigned to each trajectory and filled into spatiotemporal coordinates to generate a trajectory image tensor. This trajectory image tensor is then concatenated with four-view static images along the time dimension to construct the model input features, including: Obtain the motion prompt information specified by the user in the form of coordinates of each time step, assign a unique RGB value to each motion trajectory by random sampling, and fill the RGB values corresponding to each spatiotemporal coordinate point into the initially empty trajectory image. When multiple motion trajectories overlap in space, their RGB embedding values are superimposed and fused to generate a trajectory image tensor. The trajectory image tensor is stitched together with the four-view static images along the time dimension, wherein the main view uses the trajectory image tensor and the auxiliary view uses the initial static image copied and filled along the time dimension. The concatenated RGB tensor is fused with the Plück ray embedding in the channel dimension to construct model input features that include RGB values, ray origin, and ray direction.
4. The method according to claim 3, characterized in that, The trajectory image tensor is concatenated with four-view static images along the time dimension, wherein the main viewpoint uses the trajectory image tensor, and the auxiliary viewpoints are filled by copying and pasting the initial static image along the time dimension, including: The trajectory image tensor is used as the temporal input of the main viewpoint, where the main viewpoint corresponds to the observation viewpoint from which the user draws motion cues. The initial static image is copied and filled along the time dimension for the auxiliary viewpoint to generate the temporal input for the auxiliary viewpoint; The trajectory image tensor from the main viewpoint and the copied fill image from the auxiliary viewpoint are stitched together along the time dimension to construct a complete four-view temporal input feature.
5. The method according to claim 3, characterized in that, For each pixel, its input feature vector is formed by concatenating the RGB values, ray origin, and ray direction in the channel dimension. The ray origin and ray direction are calculated based on the camera's intrinsic and extrinsic parameters and are used to characterize the three-dimensional spatial geometric information corresponding to the pixel. The pixel is a unified index of each pixel in the four-view static image, and each pixel corresponds to a three-dimensional spatial ray emanating from the camera's optical center.
6. The method according to claim 1, characterized in that, The input features of the model are fed into a four-dimensional Gaussian reconstruction model to predict a complete set of three-dimensional Gaussian attributes for the initial frame, and only predict the geometric attributes of position and rotation for subsequent frames, in order to generate a temporally consistent three-dimensional Gaussian sequence, including: The U-Net architecture equipped with spatial and temporal attention layers is used as the basic framework of the four-dimensional Gaussian reconstruction model, and the input features of the model are input into the four-dimensional Gaussian reconstruction model to output the initial four-dimensional Gaussian order. A constraint strategy is applied to the initial four-dimensional Gaussian sequence: the complete Gaussian attribute set is predicted in the initial frame, and only the geometric attributes of position and rotation are predicted in subsequent frames, resulting in a constrained four-dimensional Gaussian sequence. Based on the characteristic that point trajectory data cannot reasonably infer changes in appearance attributes, the constrained four-dimensional Gaussian sequence is used as the final output to generate a time-consistent three-dimensional Gaussian sequence.
7. The method according to claim 1, characterized in that, On the generated 3D Gaussian sequence, the appearance loss function, the rigidity loss function (as close as possible), and the vector consistency loss function are calculated to jointly supervise the optimization of model parameters and achieve physically reasonable dynamic geometric modeling, including: Based on the physical property that deformable motion can be decomposed into rigidly connected sub-components, a rigid body loss constraint is introduced to constrain the relative distance between pairs of three-dimensional Gaussian points within the same sub-component, so that the Gaussian points maintain rigid motion over time. To eliminate the ambiguity of rotation and translation caused by only constraining relative distance in rigid body loss, a vector consistency loss function is introduced to explicitly specify the translational motion mode by constraining the relative position vector between Gaussian mean point pairs. By combining appearance loss function, rigidity loss function, and vector consistency loss function, temporal supervised optimization of 3D Gaussian sequences is performed to achieve accurate and physically reasonable dynamic geometric modeling.
8. A precise and efficient motion-guided four-dimensional content generation system, characterized in that, include: The prompt information receiving module is used to receive motion prompt information through an interactive user interface; The static image generation module is used to acquire a single input image of the target object and generate a four-view static image using a multi-view image diffusion model according to a fixed pitch angle and a preset orthogonal azimuth angle. The model input feature construction module is used to convert the motion prompt information into a trajectory image, assign a unique RGB value to each trajectory and fill it into the spatiotemporal coordinate points to generate a trajectory image tensor, and then stitch the trajectory image tensor with the four-view static image along the time dimension to construct the model input features. The 3D Gaussian sequence generation module is used to feed the model input features into the 4D Gaussian reconstruction model, predict the complete 3D Gaussian attribute set for the initial frame, and predict only the geometric attributes of position and rotation for subsequent frames, so as to generate a temporally consistent 3D Gaussian sequence. The joint supervision module is used to calculate the appearance loss function, the rigid body loss function as much as possible, and the vector consistency loss function on the generated 3D Gaussian sequence, so as to jointly supervise the optimization of model parameters and achieve physically reasonable dynamic geometric modeling.
9. A computer device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the fine and efficient motion-guided four-dimensional content generation method as described in any one of claims 1-7.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the fine and efficient motion-guided four-dimensional content generation method as described in any one of claims 1-7.