Alignment-Based Multimodal 4D Content Generation Method and System

Abstract

This invention discloses an alignment-based multimodal 4D content generation method and system, belonging to the field of computer vision processing. In this invention, a single-modal input is first transformed into a video-3D model pair using a diffusion model, and this is used to initialize the 4D model. The video alignment focal length and multi-viewpoint alignment focal length are obtained, and accurate spatiotemporal registration is achieved through two-stage focal length alignment. An asynchronous optimization strategy of alternating odd and even steps to optimize the 3D Gaussian model and deformation mesh is employed to minimize the total loss function composed of known / unknown viewpoint-time alignment losses. The final generated 4D asset model can output rendered images or continuous videos from any viewpoint-time, achieving high-quality multimodal 4D content generation. This invention can flexibly handle various inputs and efficiently generate 4D asset models faithful to the inputs.

Description

Technical Field

[0001] This invention belongs to the field of computer vision processing, and in particular relates to a method and system for generating multimodal 4D content based on alignment. Background Technology

[0002] With the rapid development of artificial intelligence and computer graphics technologies, the generation of three-dimensional (3D) content has become a research hotspot. However, static 3D models are insufficient to fully express the dynamic changes of objects and scenes in the real world. Therefore, four-dimensional (4D) content (i.e., dynamic 3D content) capable of capturing and generating temporal information is becoming increasingly important. 4D content has broad application prospects in many fields such as virtual reality (VR), augmented reality (AR), digital humans, filmmaking, game development, metaverse construction, and robot simulation.

[0003] Currently, 3D / 4D content generation mainly faces the following challenges and limitations: 1) High complexity of data acquisition and modeling: Traditional 4D content generation typically relies on multi-sensor capture (such as multi-camera arrays, depth sensors, etc.) and complex reconstruction algorithms. These methods are costly, time-consuming, and have strict requirements on environment and equipment, making them difficult to popularize. 2) Text-based 3D / 4D generation: Although research on generating 3D content through text descriptions (such as text-to-3D) has made progress, generating high-fidelity, dynamic, and controllable 4D assets directly from text still faces significant challenges, mainly because the semantic richness of text information is limited, making it difficult to fully define complex dynamic scenes. 3) Image-based 3D / 4D generation: 3D reconstruction from single or small numbers of images (such as NeRF, 3DGaussian Splatting, etc.) has become increasingly mature, but extending it to dynamic 4D generation often requires multi-view or time-series images as input, and there are difficulties in maintaining temporal consistency and dynamic details. Furthermore, inferring dynamic information from static images is itself an underdetermined problem. 4) 3D / 4D Generation from Video: Generating high-quality, editable 4D content directly from a single video remains challenging. While video contains temporal information, its limited information dimension (2D projection) restricts the ability to directly reconstruct complex 3D geometry and dynamics, especially when dealing with occlusion, missing viewpoints, and motion blur. 5) 4D Generation from Existing 3D Models: Although 4D can be generated by adding deformations or animations to existing 3D models, this requires specialized modeling and animation knowledge, and the diversity of the generated content is limited by the quality and type of the initial 3D model. 6) Insufficient Diversity and Controllability of Generated Content: Existing methods often struggle to achieve high diversity and fine-grained control when generating 4D content. The generated results may lack variation in action, pose, appearance, or scene dynamics, making it difficult to meet the personalized needs of different application scenarios. 7) Inefficient Use of Multimodal Information: Many existing methods tend to focus on a single modal input, failing to effectively utilize the complementary information between different modal data (such as semantics of text, details of images, and dynamics of video). How to effectively fuse information from multiple modalities to drive more realistic and richer 4D generation is a problem that current technology urgently needs to solve. 8) The trade-off between quality and efficiency in generating 4D assets: While pursuing high-fidelity 4D generation, computational efficiency often becomes a bottleneck. For example, although 4D reconstruction based on neural radiation fields (NeRF) can generate high-quality views, the rendering speed and training time are usually long, and it is difficult to edit directly. 9) In recent years, diffusion models have shown powerful capabilities in the fields of image, video, and 3D content generation, and can generate high-fidelity and diverse samples. However, fully applying the advantages of diffusion models to generate dynamic 4D content from arbitrary multimodal inputs and overcoming the above limitations remains a challenging frontier research direction.

[0004] In summary, current multimodal 4D generation technologies still have significant shortcomings in data acquisition, input modality processing, generation diversity, multimodal information fusion, and generation efficiency. Therefore, the industry urgently needs a new method that can overcome these deficiencies and efficiently and effectively generate diverse and dynamic 4D assets from any modal input. Summary of the Invention

[0005] The purpose of this invention is to solve the problems existing in the prior art and to provide a method and system for generating multimodal 4D content based on alignment.

[0006] The inventive concept of this invention is to generate and transform input data of any modality into standard data pairs (video, 3D model). Based on these standard data pairs, this invention generates 4D asset models that faithfully reflect the input conditions. Specifically, this invention obtains accurate focal length parameters for training through focal length search, proposes action alignment loss and geometric alignment loss, and designs an unknown time-view alignment loss based on these. This unknown time-view alignment loss is weighted and summed with the known time-view alignment loss to obtain the total loss, which is then used to jointly optimize the 4D model, ultimately generating vivid and diverse 4D asset models.

[0007] To achieve the above-mentioned objectives, the present invention specifically adopts the following technical solution:

[0008] In a first aspect, the present invention provides an alignment-based multimodal 4D content generation method, which includes the following steps:

[0009] S1: Input data of any single modality is fed into the diffusion model to generate target data pairs consisting of video and 3D models;

[0010] S2: Initialize the 4D model from the 3D model in the target data pair, and use the first frame of the video in the target data pair as the alignment reference image. Render the 4D model within the preset focal length range to obtain the main view rendering image corresponding to each focal length at the initial moment. Calculate the mean square error between each main view rendering image and the alignment reference image, and take the focal length corresponding to the minimum mean square error as the video alignment focal length.

[0011] S3: Within the focal length range of S2, retain focal lengths smaller than the video alignment focal length to form a new focal length range. Within this new focal length range, render the initial 4D model from different viewpoints. Calculate the fractional distillation sampling loss value corresponding to the rendered image at each focal length from each viewpoint. Average the loss values ​​across all viewpoints at the same focal length, and use the focal length corresponding to the minimum average loss value as the multi-view alignment focal length. ;

[0012] S4: The 4D model is asynchronously optimized with the goal of minimizing the total loss function. In odd-numbered steps of the optimization process, only the 3D Gaussian model in the 4D model is optimized, while in even-numbered steps, only the deformation mesh in the 4D model is optimized. The optimized 4D model is used as the final high-quality 4D asset model. The total loss function is formed by the weighted sum of the known time-view alignment loss and the unknown time-view alignment loss. For known viewpoints and times, the 4D model is rendered in the main viewpoint based on the video alignment focal length to generate a multi-time main viewpoint rendering map. The known time-view alignment loss is calculated from the multi-time main viewpoint rendering map and the video in the target data pair. For unknown viewpoints and times, the 4D model is rendered in the non-main viewpoint based on the multi-view alignment focal length to generate a multi-time multi-viewpoint rendering map. By aligning the multi-time multi-viewpoint rendering map with the prior of the multi-view diffusion model, motion information and geometric information are transferred to the 4D model to construct motion alignment loss and geometric alignment loss. The motion alignment loss and geometric alignment loss are each weighted and fused with a time-related fusion parameter to obtain the unknown time-view alignment loss.

[0013] S5: Input the viewing angle and time parameters into the high-quality 4D asset model to obtain the corresponding rendered image; input the continuous viewing angle and time parameters into the high-quality 4D asset model to obtain multi-time and multi-view rendered video, and complete the generation of multimodal 4D content based on alignment.

[0014] Based on the above scheme, each step can be implemented in the following preferred manner.

[0015] As a preferred embodiment of the first aspect above, in step S1, the input data is text, image, video, or 3D model.

[0016] As a preferred embodiment of the first aspect, in step S1, the specific process of generating the video and 3D model in the target data pair is as follows: Input data is acquired; when the input data is text, it is input into the text-video diffusion model to generate a video corresponding to the text, and the first frame of the generated video is used as a control condition to input into the image-3D diffusion model to obtain a 3D model corresponding to the generated video; when the input data is an image, it is input into the image-video diffusion model to generate a video corresponding to the image, and the input image is used as a control condition to input into the image-3D diffusion model to obtain a 3D model corresponding to the input image; when the input data is a video, its first frame is used as a control condition to input into the image-3D diffusion model to obtain a 3D model corresponding to the input video; when the input data is a 3D model, its frontal view rendered image is used as a control condition to input into the image-video diffusion model to obtain a video corresponding to the input 3D model.

[0017] As a preferred embodiment of the first aspect, in step S2, the specific process of obtaining the video alignment focal length is as follows: Initialize the 4D model from the 3D model in the target data pair at the initial moment. For T-frame videos in the target data pair, use the video frames at the initial moment as the alignment reference image. Render the 4D model within a preset focal length range at the same focal length interval. Obtain the main view rendering image for each focal length at the initial moment. Calculate the mean square error between each main view rendering image and the alignment reference image. Use the focal length corresponding to the minimum mean square error as the video alignment focal length. .

[0018] As a preferred embodiment of the first aspect, in step S3, the specific process of calculating the fractional distillation sampling loss corresponding to the rendering image for each focal length is as follows: under each viewpoint, the rendering image corresponding to each focal length is input into the encoder to obtain the feature map corresponding to each rendering image. Randomly sampled Gaussian noise is superimposed on each feature map, with alignment with the reference image as the control condition. The fractional distillation sampling loss value is calculated based on the feature map after superimposing noise to obtain the loss value corresponding to the rendering image at that focal length.

[0019] As a preferred embodiment of the first aspect mentioned above, in step S4, a multi-moment main-view rendering map is generated by rendering a 4D model under the video alignment focal length and main-view perspective. The multi-moment main-view rendering map is composed of multiple single-moment main-view rendering maps, and each single-moment main-view rendering map corresponds one-to-one with each frame of the video in the target data pair. The mask of each frame and the single-moment main-view rendering map is obtained, the mean squared error loss between each frame and the single-moment main-view rendering map is calculated, the mean squared error loss between the mask of each frame and the mask of the single-moment main-view rendering map is calculated, and the two mean squared error losses are added together as the known moment-view alignment loss.

[0020] Furthermore, the known time-view alignment loss It is expressed as follows:

[0021]

[0022] in, This represents the total number of video frames in the target data pair, which is equal to the total number of time points. Indicates the main perspective; express Rendered image from the main viewpoint at any given moment; express The mask; This indicates the target data pair in the middle video. frame; express The mask; This represents the square of the L2 norm.

[0023] As a preferred embodiment of the first aspect mentioned above, in step S4, a multi-moment multi-view rendering image is generated by rendering the 4D model under multi-viewpoint focal length alignment and non-primary viewpoint. The multi-moment multi-view rendering image is composed of single-moment single-view rendering images from different moments and different perspectives. The single-view rendering image at any given time is multiplied by a first weighting coefficient related to the timestamp, and then randomly sampled Gaussian noise is multiplied by a second weighting coefficient related to the timestamp. The weighted result is then... The single-view rendering at any given time is formed by adding weighted Gaussian noise. Adding noise to the rendered image at any time The first step of adding noise to the rendering image and aligning the target data with the video. frame, The five parts of the single-view rendering image, the viewpoint corresponding to the time, the multi-view aligned focal length, and the timestamp of random sampling in the multi-view diffusion model are input into the U-Net network of the multi-view diffusion model, and the noise prediction result is output. The mean square error between the noise prediction results at all time points and the randomly sampled Gaussian noise is used as the action alignment loss.

[0024] Furthermore, action alignment loss It is expressed as follows:

[0025]

[0026] in, This represents the total number of randomly sampled viewpoints; The U-Net network representing the multi-view diffusion model; This represents the output noise prediction result of the U-Net network; This represents the first weighting coefficient associated with the timestamp; Indicates the first indivual Single-view rendering of the moment; express Corresponding perspective; This represents the second weighting coefficient associated with the timestamp; Gaussian noise representing random sampling; variable This represents the timestamp of random sampling in a multi-view diffusion model.

[0027] As a preferred embodiment of the first aspect mentioned above, in step S4, The viewpoint corresponding to the single-view rendering image at any given time is used as the reference viewpoint. The 3D model of the target data pair is then rendered under the reference viewpoint, generating... 3D rendering of time, Noise-added rendering images at all times 3D rendering of time The five parts—focal length alignment, timestamps of random sampling in the multi-view diffusion model, etc.—are input into the U-Net network of the multi-view diffusion model to output new noise prediction results; The mean square error between the new noise prediction results and the randomly sampled Gaussian noise at all time points is used as the geometric alignment loss.

[0028] Furthermore, geometric alignment loss It is expressed as follows:

[0029]

[0030] in, Indicates from the perspective Generated by rendering 3D models 3D rendering of the moment.

[0031] As a preferred embodiment of the first aspect mentioned above, in step S4, the unknown time-view alignment loss is formed by summing and averaging the action-geometric alignment losses at all times. The action-geometric alignment loss at each time is composed of two terms: the first term is the product of the first fusion parameter and the action alignment loss, and the second term is the product of the second fusion parameter and the geometric alignment loss. The first fusion parameter is the product of the first ratio and a preset hyperparameter, and the second fusion parameter is the product of the second ratio and the hyperparameter. The first ratio is the ratio of the current time to the total number of times, and the second ratio is the ratio of the time difference to the total number of times. The time difference is the difference between the current time and the total number of times.

[0032] Furthermore, unknown moment-view alignment loss It is expressed as follows:

[0033]

[0034] in, Indicates the preset hyperparameters; Indicates the first fusion parameter; Indicates the second fusion parameter; Indicates the first ratio; This indicates the second ratio.

[0035] Secondly, the present invention provides an alignment-based multimodal 4D content generation system, comprising:

[0036] The data acquisition module is used to feed input data of any single modality into the diffusion model to generate target data pairs consisting of video and 3D models;

[0037] The first focal length acquisition module is used to initialize a 4D model from the 3D model in the target data pair, and use the first frame of the video in the target data pair as the alignment reference image to render the 4D model within the preset focal length range to obtain the main view rendering image corresponding to each focal length at the initial moment, calculate the mean square error between each main view rendering image and the alignment reference image, and take the focal length corresponding to the minimum mean square error as the video alignment focal length.

[0038] The second focal length acquisition module is used to retain focal lengths smaller than the video alignment focal length within the focal length range of the first focal length acquisition module, forming a new focal length range. Within the new focal length range, the 4D model at the initial moment is rendered from different perspectives. The fractional distillation sampling loss value corresponding to the rendering map of each focal length is calculated at each perspective. The loss values ​​of all perspectives are averaged at the same focal length, and the focal length corresponding to the minimum average loss value is taken as the multi-view alignment focal length.

[0039] The model optimization module asynchronously optimizes the 4D model with the goal of minimizing the total loss function. In odd-numbered steps of the optimization process, only the 3D Gaussian model in the 4D model is optimized, while in even-numbered steps, only the deformation mesh in the 4D model is optimized. The optimized 4D model is used as the final high-quality 4D asset model. The total loss function is formed by a weighted sum of known time-view alignment loss and unknown time-view alignment loss: For known viewpoints and times, the 4D model is rendered in the main viewpoint based on the video alignment focal length to generate a multi-time main viewpoint rendering map. The known time-view alignment loss is calculated from the multi-time main viewpoint rendering map and the video in the target data pair. For unknown viewpoints and times, the 4D model is rendered in the non-main viewpoint based on the multi-view alignment focal length to generate a multi-time multi-viewpoint rendering map. By aligning the multi-time multi-viewpoint rendering map with the prior of the multi-view diffusion model, motion information and geometric information are transferred to the 4D model to construct motion alignment loss and geometric alignment loss. The motion alignment loss and geometric alignment loss are each weighted and fused with a time-related fusion parameter to obtain the unknown time-view alignment loss.

[0040] The result acquisition module is used to input observation viewpoint and time parameters into a high-quality 4D asset model to obtain the corresponding rendered image; it inputs continuous observation viewpoint and time parameters into a high-quality 4D asset model to obtain multi-time and multi-view rendered video, thus completing the generation of alignment-based multimodal 4D content.

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] This invention proposes an alignment-based multimodal 4D content generation method for input data of any single modality, which can efficiently generate 4D asset models faithful to the input data. First, this invention inputs data of any single modality (image / video / 3D model) into a diffusion model to generate video-3D model data pairs, breaking through the dependence of traditional 4D generation on multimodal joint input. Compared with traditional methods that require multi-view shooting or 3D scanning, this invention has lower data acquisition costs and wider applicability. Second, this invention uses the first frame of the video as a reference to calculate the MSE loss, determine the video alignment focal length, and achieve main viewpoint alignment. Then, within a reduced focal length range, it optimizes multi-view consistency based on SDS loss to determine the multi-view alignment focal length, achieving multi-view alignment. This two-stage focal length alignment optimization mechanism improves... This invention improves the geometric consistency of 4D models by enhancing viewpoint-action alignment accuracy and avoiding geometric distortion caused by focal length mismatch in traditional methods. During 4D model optimization, the invention uses ground truth video data for supervision at known time points and transfers action / geometric information through diffusion priors at unknown time points. It also designs fusion weights for action / geometric losses that adjust over time. Furthermore, the invention employs an alternating training strategy of optimizing the 3D Gaussian model in odd-steps and the deformation network in even-steps to accelerate training convergence and improve dynamic detail preservation. Finally, the trained 4D asset model supports parametric rendering and is suitable for industrial applications. From a practical application perspective, this invention achieves breakthrough results in 4D content generation through multimodal input adaptation, dual-focal length alignment optimization, asynchronous training strategies, and spatiotemporal decoupling loss design, enabling efficient and high-quality 4D content generation. Attached Figure Description

[0043] Figure 1 This is a flowchart of the steps of the present invention;

[0044] Figure 2 This is a schematic diagram of the first type of input data in the X4D dataset according to an embodiment of the present invention;

[0045] Figure 3 This is a schematic diagram illustrating the rendering results of different methods on the first type of input data in embodiments of the present invention;

[0046] Figure 4 This is a schematic diagram of the second type of input data in the X4D dataset according to an embodiment of the present invention;

[0047] Figure 5 This is a schematic diagram illustrating the rendering results of different methods on the second type of input data in an embodiment of the present invention;

[0048] Figure 6 This is a schematic diagram of one type of input data in the Consistent4D dataset according to an embodiment of the present invention;

[0049] Figure 7 This is a schematic diagram illustrating the rendering results of different methods on the Consistent4D dataset in the embodiments of the present invention;

[0050] Figure 8 This is a system block diagram of the present invention. Detailed Implementation

[0051] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0052] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features.

[0053] like Figure 1 As shown, in a preferred embodiment of the present invention, the above-mentioned alignment-based multimodal 4D content generation method includes the following steps S1 to S5. The specific implementation process of each step will be described in detail below.

[0054] S1: Generation and task transformation of multimodal data pairs. Input data of any single modality (e.g., text, image, video, or 3D model) is fed into the diffusion model to generate target data pairs consisting of video and 3D model.

[0055] It should be noted that in step S1 of this invention, the input data is text, image, video or 3D model.

[0056] It should be noted that, in step S1 of this invention, the specific process of generating the video and 3D model in the target data pair is as follows: Input data is acquired. When the input data is text, it is input into the text-video diffusion model to generate a video corresponding to the text. The first frame of the generated video is used as a control condition and input into the image-3D diffusion model to obtain a 3D model corresponding to the generated video. When the input data is an image, it is input into the image-video diffusion model to generate a video corresponding to the image. The input image is used as a control condition and input into the image-3D diffusion model to obtain a 3D model corresponding to the input image. When the input data is a video, its first frame is used as a control condition and input into the image-3D diffusion model to obtain a 3D model corresponding to the input video. When the input data is a 3D model, its frontal view rendered image is used as a control condition and input into the image-video diffusion model to obtain a video corresponding to the input 3D model.

[0057] In this embodiment, step S1 effectively transforms the multimodal 4D generation task into a (video, 3D model) to 4D generation task. In this step, after generating the video and 3D model in the target data pair according to the above process, the generated products are uniformly organized into (video, 3D model) data pairs for subsequent 4D generation.

[0058] S2: Obtain the video alignment focal length. Initialize the 4D model from the 3D model in the target data pair, and use the first frame of the video in the target data pair as the alignment reference image. Render the 4D model within the preset focal length range to obtain the main view rendering image corresponding to each focal length at the initial moment. Calculate the mean square error between each main view rendering image and the alignment reference image, and take the focal length corresponding to the minimum mean square error as the video alignment focal length.

[0059] It should be noted that in step S2 of this invention, the specific process of obtaining the video alignment focal length is as follows: the initial time (time 1) of initializing the 4D model from the 3D model in the target data pair, for the target data pair... The video alignment uses the initial video frame as the alignment reference image. Within a preset focal length range, a 4D model is rendered at the same focal length interval. The main view rendering image at each focal length at the initial moment is obtained. The mean square error between each main view rendering image and the alignment reference image is calculated. The focal length corresponding to the minimum mean square error is taken as the video alignment focal length. .

[0060] In step S2 of this embodiment, focal length values ​​for rendering 4D models are generated in intervals of 0.001 within the focal length range of 0 to 5, and the main view rendering image of each focal length value at the initial moment is recorded.

[0061] S3: Obtain the multi-view alignment focal length. Within the focal length range of S2, retain focal lengths smaller than the video alignment focal length to form a new focal length range. Within this new focal length range, render the initial 4D model from different viewpoints. Calculate the fractional distillation sampling (SDS) loss value corresponding to the rendered image at each focal length for each viewpoint. Average the loss values ​​across all viewpoints at the same focal length, and use the focal length corresponding to the minimum average loss value as the multi-view alignment focal length. .

[0062] It should be noted that in step S3 of the present invention, the specific process of calculating the fractional distillation sampling loss corresponding to the rendering image for each focal length is as follows: under each viewpoint, the rendering image corresponding to each focal length is input into the encoder to obtain the feature map corresponding to each rendering image. Randomly sampled Gaussian noise is superimposed on each feature map, with alignment with the reference image as the control condition. The fractional distillation sampling loss value is calculated based on the feature map after superimposing noise to obtain the loss value corresponding to the rendering image under that focal length.

[0063] In step S3 of this embodiment, four viewpoints [0, 90, 180, 270] are preset, and the 4D model at the initial moment is rendered under each viewpoint. Then, a rendering image will be obtained under each viewpoint and focal length. That is, under the same focal length, rendering images under four viewpoints will be obtained. , Each focal length represents a viewpoint. Then, the rendered images of each of the four viewpoints are input into the encoder to obtain the feature maps of each focal length under the four viewpoints. Randomly sample a Gaussian noise and superimpose this noise onto the feature map. Then, calculate the SDS loss value.

[0064] S4: Optimize the 4D model. The 4D model is asynchronously optimized with the goal of minimizing the total loss function. In odd-numbered steps of the optimization process, only the 3D Gaussian model (3DGS) in the 4D model is optimized, while in even-numbered steps, only the deformation mesh in the 4D model is optimized. The optimized 4D model is used as the final high-quality 4D asset model.

[0065] The total loss function is formed by a weighted sum of known time-view alignment loss and unknown time-view alignment loss: For known viewpoints and times, a 4D model is rendered in the main viewpoint based on the video alignment focal length to generate a multi-time main viewpoint rendering map, and the known time-view alignment loss is calculated from the multi-time main viewpoint rendering map and the video in the target data pair; For unknown viewpoints and times, a 4D model is rendered in the non-main viewpoint based on the multi-view alignment focal length to generate a multi-time multi-view rendering map, and motion information and geometric information are transferred to the 4D model by aligning the multi-time multi-view rendering map with the prior of the multi-view diffusion model to construct motion alignment loss and geometric alignment loss, and the motion alignment loss and geometric alignment loss are each weighted and fused with a time-related fusion parameter to obtain the unknown time-view alignment loss.

[0066] It should be noted that in step S4 of this invention, a multi-moment main-view rendering map is generated by rendering a 4D model under the video alignment focal length and main view perspective. The multi-moment main-view rendering map is composed of multiple single-moment main-view rendering maps, and each single-moment main-view rendering map corresponds one-to-one with each frame of the target data-aligned video. The mask of each frame and the single-moment main-view rendering map is obtained, the mean squared error loss between each frame and the single-moment main-view rendering map is calculated, the mean squared error loss between the mask of each frame and the mask of the single-moment main-view rendering map is calculated, and the two mean squared error losses are added together as the known moment-view alignment loss.

[0067] In this embodiment, for a known viewpoint and time, the known time-viewpoint alignment loss is obtained by calculating the mean squared error loss between the main viewpoint rendering image at a single time point and the corresponding video frame, and the mean squared error loss between the masks. :

[0068]

[0069] in, This represents the total number of video frames in the target data pair, which is also the total number of time points. Indicates the main perspective (frontal view); express Rendered image from the main viewpoint at any given moment; express The value on the alpha channel, i.e. The mask; This indicates the target data pair in the middle video. frame; express The mask; This represents the square of the L2 norm.

[0070] It should be noted that in step S4 of the present invention, a multi-moment multi-view rendering image is generated by aligning the focal length of the multi-viewpoint and rendering the 4D model under a non-primary viewpoint. The multi-moment multi-view rendering image is composed of single-moment single-view rendering images of different times and different views.

[0071] It should be noted that in step S4 of the present invention, the... A single-view rendering of a time stamp and a first weighting coefficient related to a timestamp. Multiply by a randomly sampled Gaussian noise and a second weighting coefficient associated with a timestamp. Multiply, then add the weighted products. The single-view rendering at any given time is formed by adding the weighted Gaussian noise. Adding noise to the rendered image at any time The first step of adding noise to the rendering image and aligning the target data with the video. frame, The five parts of the single-view rendering image, the viewpoint corresponding to the time, the multi-view aligned focal length, and the timestamp of random sampling in the multi-view diffusion model are input into the U-Net network of the multi-view diffusion model, and the noise prediction result is output. The mean square error between the noise prediction results at all time points and the randomly sampled Gaussian noise is used as the action alignment loss.

[0072] In this embodiment, for the first At any given moment, a viewpoint is randomly sampled within the non-primary viewpoint range. The 4D model is then rendered using the multi-viewpoint aligned focal length and the randomly sampled viewpoint, generating a rendering image for that specific moment and viewpoint. This is a single-moment, single-viewpoint rendering image. When random sampling... From the perspective of the first viewpoint, in the first At that moment, a corresponding generator will be generated. indivual Single-view renderings at different times. These renderings belong to the same time moment but not the same viewpoint. Further, after generating the corresponding noise prediction results based on the single-view renderings at each time moment, the noise prediction results for all views belonging to the same time moment are statistically analyzed. The mean square error between these noise prediction results and randomly sampled Gaussian noise is calculated to obtain the action alignment loss. :

[0073]

[0074] in, This represents the total number of randomly sampled viewpoints; The U-Net network representing the multi-view diffusion model; This represents the output noise prediction result of the U-Net network; This represents the first weighting coefficient associated with the timestamp; Indicates the first indivual Single-view rendering of the moment; express Corresponding perspective; This represents the second weighting coefficient associated with the timestamp; Gaussian noise representing random sampling; variable This represents the timestamp of random sampling in a multi-view diffusion model.

[0075] It should be noted that in step S4 of the present invention, the... The viewpoint corresponding to the single-view rendering image at any given time is used as the reference viewpoint. The 3D model of the target data pair is then rendered under the reference viewpoint, generating... 3D rendering of time, Noise-added rendering images at all times 3D rendering of time The five parts—focal length alignment, timestamps of random sampling in the multi-view diffusion model, etc.—are input into the U-Net network of the multi-view diffusion model to output new noise prediction results; The mean square error between the new noise prediction results and the randomly sampled Gaussian noise at all time points is used as the geometric alignment loss.

[0076] In this embodiment, for the first At a given moment, by changing two parts of the input in the multi-view diffusion model, we can obtain... The noise prediction result is calculated for a specific viewpoint at a given time. Then, the noise prediction results for all viewpoints at the same time are statistically analyzed. The mean square error between these noise prediction results and randomly sampled Gaussian noise is calculated to obtain the geometric alignment loss. :

[0077]

[0078] in, Indicates from the perspective Generated by rendering 3D models 3D rendering of the moment.

[0079] It should be noted that in step S4 of the present invention, the unknown time-view alignment loss is formed by adding and averaging the action-geometric alignment losses at all times. The action-geometric alignment loss at each time is composed of two terms added together. The first term is the first fusion parameter multiplied by the action alignment loss, and the second term is the second fusion parameter multiplied by the geometric alignment loss. The first fusion parameter is the first ratio multiplied by a preset hyperparameter, and the second fusion parameter is the second ratio multiplied by the hyperparameter. The first ratio is the ratio of the current time to the total number of times, and the second ratio is the ratio of the time difference to the total number of times. The time difference is the difference between the current time and the total number of times.

[0080] In this embodiment, due to the motion time Tend to When the time interval is reached, the geometric representation deviates significantly from the initial state. Therefore, this invention designs two time-dependent fusion parameters to obtain the action-geometric alignment loss at each time step. For the first... At time , the corresponding action-geometric alignment loss is:

[0081]

[0082] in, Indicates the preset hyperparameters; Indicates the first fusion parameter; Indicates the second fusion parameter; Indicates the first ratio; This indicates the second ratio.

[0083] Furthermore, the unknown time-view alignment loss can be obtained by summing and averaging the action-geometric alignment losses at all times. :

[0084]

[0085] Therefore, the total loss function is formed by weighted summation of the known time-view alignment loss and the unknown time-view alignment loss. as follows:

[0086]

[0087] In the total loss function, the action alignment loss ensures that the position and motion pose of the 4D object at each time step is consistent with the motion pose of the front view; by optimizing the known time-view alignment loss, the front view of the 4D model at multiple time steps is aligned with the video in the target data pair.

[0088] In this embodiment, gradient backpropagation is performed based on the obtained total loss function. When the number of training steps is odd, the weights of the deformation network are frozen, and only the 3D Gaussian model is optimized. When the number of training steps is even, the 3D Gaussian model is frozen, and only the deformation network is optimized. This asynchronous optimization method has the following two advantages: First, it uses multi-frame video and random viewpoint parameters as control conditions to optimize the 4D rendering at the corresponding time-viewpoint, thereby enhancing the model's generalization ability under random viewpoints; second, it uses the 4D rendering at time 0 with multiple viewpoints and the main viewpoint parameters as control conditions to optimize the 4D rendering at other timepoints with the same viewpoint, ensuring the consistency of the model's viewpoint at different times.

[0089] S5: Rendering Image or Video Generation. Inputting viewpoint and time parameters into a high-quality 4D asset model yields the corresponding rendered image; inputting continuous viewpoint and time parameters into a high-quality 4D asset model yields multi-time and multi-view rendered videos, completing the generation of alignment-based multimodal 4D content.

[0090] Therefore, the high-quality 4D asset model obtained through the S4 step training can use the input of any modality as a control condition to drive the generation of diverse 4D content.

[0091] To better demonstrate the specific implementation and technical effects of the present invention, the alignment-based multimodal 4D content generation method shown in steps S1 to S5 of the above preferred implementation is applied to a specific example.

[0092] Example

[0093] The steps in this embodiment are the same as those in the alignment-based multimodal 4D content generation method shown in steps S1 to S5 above, and will not be repeated here. The main focus is on demonstrating the specific dataset, some specific parameter settings, and implementation results of this embodiment.

[0094] To quantify the metrics, this embodiment conducted various quantization tests on the Consistent4D and X4D datasets. The test results of the method of this invention compared with four other methods—L4GM, SC4D, STAG4D, and DG4D—on the Consistent4D dataset are shown in Table 1, and the test results on the X4D dataset are shown in Table 2. One type of input data in the X4D dataset is shown in Table 2. Figure 2 As shown, the rendering results of different methods on this input data are as follows: Figure 3 As shown, another type of input data in the X4D dataset is as follows: Figure 4 As shown, the rendering results of different methods on this input data are as follows: Figure 5 As shown, one type of input data in the Consistent4D dataset is as follows: Figure 6As shown, the rendering results of different methods on this input data are as follows: Figure 7 As shown in Table 1. It should also be noted that L4GM, SC4D, STAG4D, and DG4D are all existing technologies, so their implementation processes will not be elaborated upon. In Table 1, PSNR (Peak Signal-to-Noise Ratio) is the peak signal-to-noise ratio, SSIM (Structural Similarity Index Measure) is the structural similarity index, LPIPS (Learned Perceptual Image Patch Similarity) is the learned perceptual image patch similarity, FVD (Fréchet Video Distance) is the Fréchet video distance, and CLIP (Contrastive Language–Image Pretraining) indicates the degree of matching between the model-generated rendering results and the text description or semantic intent. In Table 2, Appearance indicates the realism and temporal coherence of the rendering results in terms of visual appearance (such as color, texture, and lighting consistency); Structure indicates the geometric accuracy, temporal consistency, and physical plausibility of the rendering results; Motion measures the motion accuracy, naturalness, and physical plausibility of the dynamic model in the temporal dimension; and Fidelity is a multi-dimensional fidelity index.

[0095] Table 1 Evaluation Table for Consistent4D Dataset

[0096]

[0097] Table 2 Evaluation Table for X4D Dataset

[0098]

[0099] It should also be noted that the alignment-based multimodal 4D content generation method in the above embodiments can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides an alignment-based multimodal 4D content generation system corresponding to the alignment-based multimodal 4D content generation method provided in the above embodiments, such as... Figure 8 As shown, it includes:

[0100] The data acquisition module is used to feed input data of any single modality into the diffusion model to generate target data pairs consisting of video and 3D models;

[0101] The first focal length acquisition module is used to initialize a 4D model from the 3D model in the target data pair, and use the first frame of the video in the target data pair as the alignment reference image to render the 4D model within the preset focal length range to obtain the main view rendering image corresponding to each focal length at the initial moment, calculate the mean square error between each main view rendering image and the alignment reference image, and take the focal length corresponding to the minimum mean square error as the video alignment focal length.

[0102] The second focal length acquisition module is used to retain focal lengths smaller than the video alignment focal length within the focal length range of the first focal length acquisition module, forming a new focal length range. Within the new focal length range, the 4D model at the initial moment is rendered from different perspectives. The fractional distillation sampling loss value corresponding to the rendering map of each focal length is calculated at each perspective. The loss values ​​of all perspectives are averaged at the same focal length, and the focal length corresponding to the minimum average loss value is taken as the multi-view alignment focal length.

[0103] The model optimization module asynchronously optimizes the 4D model with the goal of minimizing the total loss function. In odd-numbered steps of the optimization process, only the 3D Gaussian model in the 4D model is optimized, while in even-numbered steps, only the deformation mesh in the 4D model is optimized. The optimized 4D model is used as the final high-quality 4D asset model. The total loss function is formed by a weighted sum of known time-view alignment loss and unknown time-view alignment loss: For known viewpoints and times, the 4D model is rendered in the main viewpoint based on the video alignment focal length to generate a multi-time main viewpoint rendering map. The known time-view alignment loss is calculated from the multi-time main viewpoint rendering map and the video in the target data pair. For unknown viewpoints and times, the 4D model is rendered in the non-main viewpoint based on the multi-view alignment focal length to generate a multi-time multi-viewpoint rendering map. By aligning the multi-time multi-viewpoint rendering map with the prior of the multi-view diffusion model, motion information and geometric information are transferred to the 4D model to construct motion alignment loss and geometric alignment loss. The motion alignment loss and geometric alignment loss are each weighted and fused with a time-related fusion parameter to obtain the unknown time-view alignment loss.

[0104] The result acquisition module is used to input observation viewpoint and time parameters into a high-quality 4D asset model to obtain the corresponding rendered image; it inputs continuous observation viewpoint and time parameters into a high-quality 4D asset model to obtain multi-time and multi-view rendered video, thus completing the generation of alignment-based multimodal 4D content.

[0105] It should also be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the embodiments provided in this application, the division of steps or modules in the system and method is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules or steps may be combined or integrated together, and a module or step may also be split.

[0106] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A method for generating multimodal 4D content based on alignment, characterized in that, Includes the following steps: S1: Input data of any single modality is fed into the diffusion model to generate target data pairs consisting of video and 3D models; S2: Initialize the 4D model from the 3D model in the target data pair, and use the first frame of the video in the target data pair as the alignment reference image. Render the 4D model within the preset focal length range to obtain the main view rendering image corresponding to each focal length at the initial moment. Calculate the mean square error between each main view rendering image and the alignment reference image, and take the focal length corresponding to the minimum mean square error as the video alignment focal length. S3: Within the focal length range of S2, retain the focal lengths smaller than the video alignment focal length to form a new focal length range. Render the 4D model at the initial moment from different perspectives within the new focal length range. Calculate the fractional distillation sampling loss value corresponding to the rendering map of each focal length under each perspective. Average the loss values ​​of all perspectives under the same focal length. Take the focal length corresponding to the minimum average loss value as the multi-view alignment focal length. S4: Asynchronously optimize the 4D model with the goal of minimizing the total loss function. In odd-numbered steps of the optimization process, only the 3D Gaussian model in the 4D model is optimized, while in even-numbered steps, only the deformation mesh in the 4D model is optimized. The optimized 4D model is used as the final high-quality 4D asset model. The total loss function is formed by the weighted sum of the known time-view alignment loss and the unknown time-view alignment loss. For known viewpoints and times, the 4D model is rendered in the main viewpoint based on the video alignment focal length to generate a multi-time main viewpoint rendering map. The known time-view alignment loss is calculated from the multi-time main viewpoint rendering map and the video in the target data pair. For unknown viewpoints and times, a multi-time multi-view rendering map is generated by rendering a 4D model under a non-primary viewpoint based on the multi-view alignment focal length. By aligning the multi-time multi-view rendering map with the prior of the multi-view diffusion model, action information and geometric information are transferred to the 4D model to construct action alignment loss and geometric alignment loss. The action alignment loss and geometric alignment loss are then weighted and fused with a time-related fusion parameter to obtain the unknown time-view alignment loss. S5: Input the viewing angle and time parameters into the high-quality 4D asset model to obtain the corresponding rendered image; input the continuous viewing angle and time parameters into the high-quality 4D asset model to obtain multi-time and multi-view rendered video, and complete the generation of multimodal 4D content based on alignment.

2. The alignment-based multimodal 4D content generation method as described in claim 1, characterized in that, In step S1, the input data can be text, images, videos, or 3D models.

3. The alignment-based multimodal 4D content generation method as described in claim 2, characterized in that, In step S1, the specific process of generating the video and 3D model in the target data pair is as follows: Input data is acquired. When the input data is text, it is input into the text-video diffusion model to generate a video corresponding to the text. The first frame of the generated video is used as a control condition and input into the image-3D diffusion model to obtain a 3D model corresponding to the generated video. When the input data is an image, it is input into the image-video diffusion model to generate a video corresponding to the image. The input image is used as a control condition and input into the image-3D diffusion model to obtain a 3D model corresponding to the input image. When the input data is a video, its first frame is used as a control condition and input into the image-3D diffusion model to obtain a 3D model corresponding to the input video. When the input data is a 3D model, its frontal view rendered image is used as a control condition and input into the image-video diffusion model to obtain a video corresponding to the input 3D model.

4. The alignment-based multimodal 4D content generation method as described in claim 1, characterized in that, In step S2, the specific process of obtaining the video alignment focal length is as follows: the initial moment of the 4D model is initialized by the 3D model in the target data pair. For the T-frame video in the target data pair, the video frame at the initial moment is used as the alignment reference image. The 4D model is rendered at the same focal length interval within the preset focal length range. The main view rendering image of each focal length at the initial moment is obtained. The mean square error between each main view rendering image and the alignment reference image is calculated. The focal length corresponding to the minimum mean square error is taken as the video alignment focal length.

5. The alignment-based multimodal 4D content generation method as described in claim 1, characterized in that, In step S3, the specific process of calculating the fractional distillation sampling loss corresponding to the rendering image for each focal length is as follows: Under each viewpoint, the rendering image corresponding to each focal length is input into the encoder to obtain the feature map corresponding to each rendering image. Randomly sampled Gaussian noise is superimposed on each feature map. With the alignment with the reference image as the control condition, the fractional distillation sampling loss value is calculated based on the feature map after superimposing noise to obtain the loss value corresponding to the rendering image under that focal length.

6. The alignment-based multimodal 4D content generation method as described in claim 1, characterized in that, In step S4, a multi-moment main-view rendering map is generated by rendering the 4D model under the video alignment focal length and main view. The multi-moment main-view rendering map is composed of multiple single-moment main-view rendering maps, and each single-moment main-view rendering map corresponds one-to-one with each frame of the target data-aligned video. The mask of each frame and the single-moment main-view rendering map is obtained, the mean squared error loss between each frame and the single-moment main-view rendering map is calculated, the mean squared error loss between the mask of each frame and the mask of the single-moment main-view rendering map is calculated, and the two mean squared error losses are added together as the known moment-view alignment loss.

7. The alignment-based multimodal 4D content generation method as described in claim 1, characterized in that, In step S4, a multi-time multi-view rendering map is generated by rendering the 4D model under multi-view alignment focal length and non-primary view. The multi-time multi-view rendering map is composed of single-time single-view rendering maps at different times and from different views. The single-view rendering map at time t is multiplied by a first weight coefficient related to a timestamp, and the randomly sampled Gaussian noise is multiplied by a second weight coefficient related to a timestamp. The weighted single-view rendering map at time t and the weighted Gaussian noise are added to form the noisy rendering map at time t. The noisy rendering map at time t, the t-th frame of the target data alignment video, the view corresponding to the single-view rendering map at time t, the multi-view alignment focal length, and the randomly sampled timestamp in the multi-view diffusion model are input into the U-Net network of the multi-view diffusion model, and the noise prediction result is output. The mean square error between the noise prediction results at all views at time t and the randomly sampled Gaussian noise is used as the action alignment loss.

8. The alignment-based multimodal 4D content generation method as described in claim 7, characterized in that, In step S4, the viewpoint corresponding to the single-view rendering image at time t is used as the reference viewpoint. The 3D model in the target data pair is rendered under the reference viewpoint to generate the 3D rendering image at time t. The five parts, namely the noisy rendering image at time t, the 3D rendering image at time t, 0, the multi-view alignment focal length, and the timestamp of random sampling in the multi-view diffusion model, are input into the U-Net network of the multi-view diffusion model to output a new noise prediction result. The mean square error between the new noise prediction results under all views at time t and the randomly sampled Gaussian noise is used as the geometric alignment loss.

9. The alignment-based multimodal 4D content generation method as described in claim 8, characterized in that, In step S4, the unknown time-view alignment loss is formed by adding and averaging the action-geometric alignment losses at all times. The action-geometric alignment loss at each time is composed of two terms: the first term is the first fusion parameter multiplied by the action alignment loss, and the second term is the second fusion parameter multiplied by the geometric alignment loss. The first fusion parameter is the first ratio multiplied by a preset hyperparameter, and the second fusion parameter is the second ratio multiplied by the hyperparameter. The first ratio is the ratio of the current time to the total number of times, and the second ratio is the ratio of the time difference to the total number of times. The time difference is the difference between the current time and the total number of times.

10. A multimodal 4D content generation system based on alignment, characterized in that, include: The data acquisition module is used to feed input data of any single modality into the diffusion model to generate target data pairs consisting of video and 3D models; The first focal length acquisition module is used to initialize a 4D model from the 3D model in the target data pair, and use the first frame of the video in the target data pair as the alignment reference image to render the 4D model within the preset focal length range to obtain the main view rendering image corresponding to each focal length at the initial moment, calculate the mean square error between each main view rendering image and the alignment reference image, and take the focal length corresponding to the minimum mean square error as the video alignment focal length. The second focal length acquisition module is used to retain focal lengths smaller than the video alignment focal length within the focal length range of the first focal length acquisition module, forming a new focal length range. Within the new focal length range, the 4D model at the initial moment is rendered from different perspectives. The fractional distillation sampling loss value corresponding to the rendering map of each focal length is calculated at each perspective. The loss values ​​of all perspectives are averaged at the same focal length, and the focal length corresponding to the minimum average loss value is taken as the multi-view alignment focal length. The model optimization module is used to asynchronously optimize the 4D model with the goal of minimizing the total loss function. In odd-numbered steps of the optimization process, only the 3D Gaussian model in the 4D model is optimized, while in even-numbered steps, only the deformation mesh in the 4D model is optimized. The optimized 4D model is used as the final high-quality 4D asset model. The total loss function is formed by a weighted sum of known time-view alignment loss and unknown time-view alignment loss: for known viewpoint and time, the 4D model is rendered in the main viewpoint based on the video alignment focal length to generate a multi-time main viewpoint rendering map. The known time-view alignment loss is calculated from the multi-time main viewpoint rendering map and the video in the target data pair. For unknown viewpoints and times, a multi-time multi-view rendering map is generated by rendering a 4D model under a non-primary viewpoint based on the multi-view alignment focal length. By aligning the multi-time multi-view rendering map with the prior of the multi-view diffusion model, action information and geometric information are transferred to the 4D model to construct action alignment loss and geometric alignment loss. The action alignment loss and geometric alignment loss are then weighted and fused with a time-related fusion parameter to obtain the unknown time-view alignment loss. The result acquisition module is used to input observation viewpoint and time parameters into a high-quality 4D asset model to obtain the corresponding rendered image; it inputs continuous observation viewpoint and time parameters into a high-quality 4D asset model to obtain multi-time and multi-view rendered video, thus completing the generation of alignment-based multimodal 4D content.

Images