Four-dimensional Gaussian model generation method, system and equipment based on Gaussian sputtering

Abstract

The invention discloses a four-dimensional Gaussian model generation method, system and equipment based on Gaussian sputtering, and belongs to the field of computer vision processing. The method comprises the following steps: firstly, generating a target video consistent with a given text prompt by using a pre-trained text-to-video diffusion model, then converting the target video into a three-dimensional grid model frame by frame by using an image-grid generation model, and rendering the three-dimensional grid model in a focal length range under a main view angle, the method comprises the following steps: firstly, calculating a rendering matching focal length, training a static Gaussian model at an initial moment by adopting a Gaussian sputtering method, then training a deformation network, and finally generating a four-dimensional Gaussian model under a given timestamp through the trained static Gaussian model and deformation network. By means of the method, the vivid high-dynamic 4D model which is fine in texture, accurate in geometrical relation and capable of accurately conforming to the semantics of the characters input by the user can be generated within obviously short time.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision processing, and in particular relates to a method, system and device for generating a four-dimensional Gaussian model based on Gaussian sputtering. Background Art

[0002] 3D Gaussian Splatting represents a 3D scene containing a set of multiple Gaussian points, each of which is defined by four attributes: position, covariance, color, and opacity. In order to draw a new view, 3D Gaussian Splatting reprojects the 3D Gaussian into the 2D image space to obtain the projection position and corresponding covariance of the Gaussian point in the 2D image space. Then, each Gaussian point is rendered along the light to obtain the color of each pixel in the rendered image. Throughout the training process, 3D Gaussian Splatting uses an adaptive control mechanism to improve the quality of 3D scene representation. The method involves regular gradient-based removal and addition of Gaussian points. It is worth noting that 3D Gaussian constitutes the basic 3D representation used in the model framework. The 4D Gaussian Splatting method is to set up an additional deformable network model on the basis of 3D Gaussian to predict the changes of 3D Gaussian points at each moment.

[0003] Score Distillation Sampling (SDS) is widely used in 3D generation methods. It aligns the 3D generation process with the training process of diffusion models (DMs). In the forward diffusion process of the 2D diffusion model, random noise is sampled from a normal distribution and added to the rendered image and rendered video, and the diffusion model is used to predict the sampled noise based on the noisy image and video. This method is used multiple times until the features of the rendered image or rendered video are migrated to the prior space of the diffusion model. The specific steps of SDS are: sampling a Gaussian random noise and adding it to the features of the rendered image to obtain the denoised noise, using the diffusion model to predict the noise to be removed, and calculating the mean squared error (MSE) between the noise predicted by the diffusion model and the sampled Gaussian random noise. The error is related to the timestamp of the denoising process, so a time-related weight is multiplied on the basis of the above mean squared error as the loss function of the diffusion model.

[0004] However, score distillation sampling leads to multifaceted problems and long-term cost issues, which have attracted the attention of many researchers. In the two-dimensional diffusion model, Classifier-Free Guided (CFG) is a commonly used technique. It trains the two-dimensional diffusion model in both conditional and unconditional modes, that is, the noise to be removed for the latent variable is predicted under given control conditions and uncontrolled conditions at the same time, and the weighted sum of the two noises is used as the noise to be removed in the current step.

[0005] However, the parameters of the diffusion model are crucial but often overlooked, and are the key factor that leads to the multi-faceted problem and a lot of time overhead. Specifically, the diffusion model is trained from images captured from several or even a single viewpoint, which is implicitly introduced into the parameters of the diffusion model and used as a prior to train the Gaussian model. This means that the parameters of the diffusion model for 3D scene representation learn the prior by aligning different random viewpoints to a fixed and limited number of viewpoints, which leads to the multi-faceted problem. Regarding the time cost, some researchers pointed out that SDS takes a lot of time to optimize. When training a Gaussian model using SDS, all parameters of the diffusion model are frozen, and only the parameters of the Gaussian model are learnable. This means that the training process requires the frozen diffusion model and predicts the correct denoising of each timestamp from any angle. Therefore, SDS takes a lot of time for 3D generation. Summary of the invention

[0006] The purpose of the present invention is to solve the problems existing in the prior art and provide a method, system and device for generating a four-dimensional Gaussian model based on Gaussian sputtering.

[0007] In order to achieve the above-mentioned invention object, the present invention specifically adopts the following technical solutions:

[0008] In a first aspect, the present invention provides a method for generating a four-dimensional Gaussian model based on Gaussian sputtering, characterized in that it comprises the following steps:

[0009] S1: Taking the given text prompt as the diffusion condition, the pre-trained text-to-video diffusion model is used to perform multi-step denoising operations on the random noise that follows the normal distribution to generate a target video consistent with the text prompt;

[0010] S2: Each target image frame in the target video is converted into a three-dimensional mesh model by the pre-trained image-mesh generation model, and the three-dimensional mesh model is rendered within the focal length range of the main view. Each focal length value corresponds to a first main view image, and the mean square error between a target image frame and each first main view image is calculated. The focal length value corresponding to the first main view image with the smallest mean square error is used as the rendering matching focal length of the target image frame;

[0011] S3: A static Gaussian model at the initial moment is trained using the Gaussian sputtering method and based on the rendering matching focal length of S2;

[0012] S4: After the static Gaussian model at the initial moment is trained, the deformation network is trained;

[0013] S5: The trained static Gaussian model at the initial moment is used as the initial static Gaussian model, the initial static Gaussian model parameters and a given timestamp are input into the trained deformation network, the static Gaussian model variable corresponding to the given timestamp is output and added to the initial static Gaussian model parameters, and the four-dimensional Gaussian model at the given timestamp is obtained, thereby completing the generation of the four-dimensional Gaussian model based on Gaussian sputtering.

[0014] Based on the above scheme, each step can be implemented in the following preferred specific manner.

[0015] As a preferred embodiment of the above-mentioned first aspect, in step S3, during the training process of the static Gaussian model, the static Gaussian model is first rendered from the main perspective camera pose by a Gaussian sputtering method to obtain a first rendered image, the mean square error between the target image and the corresponding first rendered image is calculated and used as the texture alignment loss, the static Gaussian model is then rendered from different random perspective camera poses by a Gaussian sputtering method, and a random perspective rendered image is obtained for each random perspective, a three-dimensional mesh model is rendered from each random perspective camera pose by a rasterization method, and a mesh rendered image is obtained for each random perspective, the geometric alignment loss is calculated based on the random perspective rendered image and the mesh rendered image at each random perspective, the weighted sum of the texture alignment loss and the geometric alignment loss is taken as the first total loss, and the static Gaussian model parameters are updated based on minimizing the first total loss.

[0016] As a preferred embodiment of the above-mentioned first aspect, the geometric alignment loss is formed by weighted summing up a first mean square error loss, a first mask mean square error loss and a first visual perception loss, wherein the first mean square error loss is the sum of the mean square errors between the random perspective rendered image and the mesh rendered image at each random perspective, the first mask mean square error loss is the sum of the mean square errors between the alpha channel output of the random perspective rendered image and the alpha channel output of the mesh rendered image at each random perspective, and the first visual perception loss is the sum of the mean square errors between the first feature and the second feature at each random perspective; wherein the first feature is a layer of features extracted from the random perspective rendered image after passing through a pre-trained VGG network, and the second feature is a layer of features extracted from the mesh rendered image after passing through a pre-trained VGG network.

[0017] As a preferred embodiment of the above-mentioned first aspect, in step S4, during the training process of the deformation network, the initial static Gaussian model parameters are fixed, each time point within the preset time range to be optimized is used as the moment to be optimized, the initial static Gaussian model parameters and the moment to be optimized are input into the deformation network, and the static Gaussian model variables corresponding to the moment to be optimized are output, and the static Gaussian model variables are added to the initial static Gaussian model parameters to obtain the Gaussian model to be optimized corresponding to the moment to be optimized, and the Gaussian sputtering method is used to render each Gaussian model to be optimized from the main perspective camera pose, and a frame of the second rendered image is generated for each moment to be optimized, and the static loss is calculated based on the target image of each frame and the corresponding second rendered image, and each frame is calculated. The mean square error between the target image and the corresponding second rendered image is used as the action alignment loss. The Gaussian sputtering method is used to render the Gaussian model to be optimized at each time to be optimized from each random view camera pose. A third rendered image is obtained corresponding to each time to be optimized under each random view. The target image of each frame, the corresponding third rendered image, the random view camera pose and a random timestamp are input into the multi-view diffusion model. The reference loss is calculated based on the noise predicted by the multi-view diffusion model and the randomly sampled Gaussian noise. The weighted sum of the action alignment loss and the reference loss is used as the dynamic loss. The weighted sum of the dynamic loss and the static loss is used as the second total loss. The deformation network parameters are updated based on minimizing the second total loss.

[0018] As a preferred embodiment of the above-mentioned first aspect, the static loss is formed by weighted summation of a second mean square error loss, a second mask mean square error loss and a second visual perception loss, the second mean square error loss is the sum of the mean square errors between the target image of each frame and the second rendered image of each frame, the second mask mean square error loss is the sum of the mean square errors between the alpha channel output of the target image of each frame and the alpha channel output of the corresponding second rendered image of each frame, and the second visual perception loss is the sum of the mean square errors between the third feature and the fourth feature; wherein the third feature is a layer of features extracted from a frame of target image after a pre-trained VGG network, and the fourth feature is a layer of features extracted from a frame of second rendered image after a pre-trained VGG network.

[0019] As a preferred embodiment of the first aspect, the reference loss The functional form is:

[0020]

[0021] Among them, C represents the number of random perspectives; T represents the number of moments to be optimized; is the random timestamp of the multi-view diffusion model; represents the coefficients associated with random timestamps; represents the noise predicted by the multi-view diffusion model; Represents random perspective A third rendering image is obtained corresponding to the next time t to be optimized; Represents the target image of the tth frame; is a randomly sampled Gaussian noise.

[0022] In a second aspect, the present invention provides a four-dimensional Gaussian model generation system based on Gaussian sputtering, comprising:

[0023] The Wensheng video module is used to take the given text prompt as the diffusion condition and use the pre-trained text-to-video diffusion model to perform multi-step denoising operations on the random noise that follows the normal distribution to generate a target video consistent with the text prompt;

[0024] A focal length alignment module is used to convert each target image frame in the target video into a three-dimensional mesh model by using a pre-trained image-mesh generation model, render the three-dimensional mesh model within the focal length range of the main viewing angle, obtain a first main viewing angle image for each focal length value, calculate the mean square error between a target image frame and each first main viewing angle image, and use the focal length value corresponding to the first main viewing angle image with the smallest mean square error as the rendering matching focal length of the target image frame;

[0025] The first model training module is used to train a static Gaussian model at an initial moment by using a Gaussian sputtering method and based on a rendering matching focal length;

[0026] The second model training module is used to train the deformable network after the static Gaussian model at the initial moment is trained;

[0027] The model generation module is used to use the trained static Gaussian model at the initial moment as the initial static Gaussian model, input the initial static Gaussian model parameters and a given timestamp into the trained deformation network, output the static Gaussian model variable corresponding to the given timestamp and add it to the initial static Gaussian model parameters to obtain the four-dimensional Gaussian model at the given timestamp, and complete the generation of the four-dimensional Gaussian model based on Gaussian sputtering.

[0028] In a third aspect, the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, can implement a four-dimensional Gaussian model generation method based on Gaussian sputtering as described in any one of the schemes of the first aspect above.

[0029] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for generating a four-dimensional Gaussian model based on Gaussian sputtering as described in any of the schemes of the first aspect above is implemented.

[0030] In a fifth aspect, the present invention provides a computer electronic device comprising a memory and a processor;

[0031] The memory is used to store computer programs;

[0032] The processor is used to implement the four-dimensional Gaussian model generation method based on Gaussian sputtering as described in any scheme of the first aspect above when executing the computer program.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] The present invention proposes a method for generating a 4D Gaussian model based on Gaussian sputtering, which is a novel text-to-4D generation framework with explicit pixel-level alignment features. The method of the present invention uses a Gaussian grid contrast learning method and a focal alignment process to learn geometric consistency from a 3D grid model and produce the same texture as the target video. In addition, the method of the present invention uses a novel motion alignment method and reference optimization techniques to optimize the 4D Gaussian model. The method of the present invention achieves excellent performance and generates a 4D Gaussian model with fine texture, accurate geometry, and coherent motion in significantly reduced time. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a flow chart of the steps of the method of the present invention;

[0036] Figure 2 A flow chart of the method of the present invention for generating a target video from a given text prompt;

[0037] Figure 3 A flowchart of the method for training a static Gaussian model according to the present invention;

[0038] Figure 4 A flowchart of the method for training a deformable network according to the present invention;

[0039] Figure 5 A schematic diagram of the result of generating a four-dimensional Gaussian model by the method of the present invention;

[0040] Figure 6 It is a system block diagram of the present invention. DETAILED DESCRIPTION

[0041] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0042] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for the purpose of distinguishing descriptions, and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0043] Due to the need for high-quality geometry and texture generation and coherent object animation, 4D content generation remains very challenging. A common point between the modules currently proposed is that they rely heavily on SDS to provide priors to guide the generation of geometry, texture, and motion. However, there are several noteworthy problems inherent in SDS, such as multi-faceted problems, unrealistic textures, and high computational costs. Based on this, the present invention provides a four-dimensional Gaussian model generation method based on Gaussian sputtering, which is achieved by Gaussian sputtering and score distillation sampling. It should be noted that Gaussian sputtering (Gaussian Splatting) and score distillation sampling (Score Distillation Sampling, SDS) in the present invention belong to the prior art, among which Gaussian sputtering technology is the core method for three-dimensional scene rendering, which achieves efficient and realistic real-time rendering effects by mapping point clouds or volume data in three-dimensional space to a set of Gaussian distributions. Score distillation sampling extracts and optimizes three-dimensional content from the text-to-video diffusion model by minimizing the difference between the pre-trained text-to-video diffusion model and the target distribution. The specific principles of these two technologies are not repeated here.

[0044] like Figure 1 As shown, in a preferred implementation of the present invention, the above-mentioned four-dimensional Gaussian model generation method based on Gaussian sputtering includes the following steps S1 to S5. The specific implementation process is described in detail below.

[0045] S1: Taking the given text prompt as the diffusion condition, we use the pre-trained text-to-video diffusion model to perform multi-step denoising operations on the random noise that follows the normal distribution to generate a target video that is consistent with the text prompt.

[0046] It should be noted that, in step S1 of the present invention, Figure 2 As shown, the generated target video with a length of T Can be used as pixel-aligned targets for geometry, texture, and motion. represents random noise in S1; v represents the text prompt in S1; Represents a text-to-video diffusion model; Indicates the first Frame target image. In addition, there are already pre-trained text-to-video diffusion models in the prior art. The present invention can also be further fine-tuned based on the pre-trained text-to-video diffusion model to meet the generation task of the present invention. Therefore, the method of the present invention is not limited to a certain text-to-video diffusion model. In this embodiment, the text-to-video diffusion model adopts Video Crafter, and the implementation method of the model belongs to the prior art and will not be repeated.

[0047] S2: Each target image frame in the target video is converted into a three-dimensional mesh model by a pre-trained image-mesh generation model, and the three-dimensional mesh model is rendered within the focal length range of the main perspective, and each focal length value corresponds to a first main perspective image; the mean square error between a target image frame and each first main perspective image is calculated, and the focal length value corresponding to the first main perspective image with the smallest mean square error is used as the rendering matching focal length of the target image frame.

[0048] It should be noted that in step S2 of the present invention, each frame of the target image in the target video is first converted into a three-dimensional mesh model with fine texture and precise geometry by a pre-trained image-mesh generation model. The image-mesh generation model here is also not limited to a specific model structure. In this embodiment, the image-to-3D mesh generation model (Convolutional Reconstruction Model, CRM) is used as the image-mesh generation model, and the three-dimensional mesh model is generated by the model. In order to obtain a suitable rendering matching focal length, the present invention performs a three-dimensional mesh model generation under the main viewing angle The focal length range ( ), and then a first main perspective image set consisting of the first main perspective images is obtained. .in, Respectively represent the minimum and maximum values ​​of the above focal length range; Represents a 3D mesh model At focal length In this embodiment, the minimum value of the focal length range is set to 0.01, and the corresponding maximum value is set to 5. Then, each first main perspective image in the first main perspective image set is calculated with a frame of target image in the target video, and the focal length corresponding to the first main perspective image with the lowest mean square difference is set as the rendering matching focal length. Taking the frame target image as an example, calculate the mean square error between it and each first main view image in the first main view image set, find the first main view image with the smallest mean square error and its corresponding focal length value, which is the rendering matching focal length :

[0049]

[0050] in, is the number of pixels of the target image in the height and width directions; represents the square of L2 norm; Indicates the focal length value corresponding to the minimum output mean square error.

[0051] S3: A static Gaussian model at the initial moment is trained using the Gaussian sputtering method and based on the rendering matching focal length of S2.

[0052] It should be noted that in step S3 of the present invention, due to the structural characteristics of the three-dimensional mesh model itself, its rendering naturally has geometric consistency under multiple perspectives. The multi-perspective rendering of the three-dimensional mesh model can be regarded as an approximation of the actual rendering result of the static Gaussian model under the corresponding perspective. Therefore, the rendering provided by the three-dimensional mesh model is compared with the rendering of the static Gaussian model at the same angle, thereby indirectly copying the geometric information of the three-dimensional mesh model to the static Gaussian model. From this perspective, the present invention proposes a Gaussian-mesh comparison learning method. Figure 3 As shown, each frame of the target image is used to align the focal length with the three-dimensional mesh model, and the static Gaussian model at each moment is trained by comparative learning between the static Gaussian model and the three-dimensional mesh model.

[0053] It should be noted that in step S3 of the present invention, during the training process of the static Gaussian model, the static Gaussian model is first rendered from the main perspective camera pose by the Gaussian sputtering method to obtain a first rendered image, the mean square error between the target image and the corresponding first rendered image is calculated and used as the texture alignment loss, and the static Gaussian model is rendered from different random perspective camera poses by the Gaussian sputtering method, and a random perspective rendered image is obtained for each random perspective. The three-dimensional mesh model is rendered from each random perspective camera pose by the rasterization method, and a mesh rendered image is obtained for each random perspective. The geometric alignment loss is calculated based on the random perspective rendered image and the mesh rendered image at each random perspective, the weighted sum of the texture alignment loss and the geometric alignment loss is taken as the first total loss, and the static Gaussian model parameters are updated based on minimizing the first total loss.

[0054] In step S3 of this embodiment, after obtaining the rendering matching focal length, the static Gaussian model is trained. In the training process of the static Gaussian model, the Gaussian sputtering method is used to obtain the main view camera pose. Render the static Gaussian model to obtain the first rendered image, calculate the mean square error between the target image and the corresponding first rendered image and use it as the texture alignment loss. Then use the Gaussian sputtering method to render the static Gaussian model at different random view camera poses, and obtain a random view rendering image for each random view. Use the rasterization method to render the three-dimensional mesh model at the same random view camera pose, and obtain a mesh rendering image. Under the same random view, calculate the loss based on the random view rendering image and the mesh rendering image, add the loss calculation results of each random view as the geometric alignment loss, and finally calculate the first total loss by weighted summing the geometric alignment loss and the texture alignment loss.

[0055] The above-mentioned geometric alignment loss is formed by weighted summation of a first mean square error loss, a first mask mean square error loss and a first visual perception loss, wherein the first mean square error loss is the sum of the mean square errors between the random perspective rendered image and the mesh rendered image at each random perspective, the first mask mean square error loss is the sum of the mean square errors between the alpha channel output of the random perspective rendered image and the alpha channel output of the mesh rendered image at each random perspective, and the first visual perception loss is the sum of the mean square errors between the first feature and the second feature at each random perspective; wherein the first feature is a layer of features extracted from the random perspective rendered image after the pre-trained VGG network, and the second feature is a layer of features extracted from the mesh rendered image after the pre-trained VGG network.

[0056] In this embodiment, the first total loss , texture alignment loss and geometric alignment loss The functional forms are:

[0057]

[0058]

[0059]

[0060]

[0061]

[0062]

[0063] in, All represent weight hyperparameters; Both represent the balance scaling factor; For the Frame first rendered image; Represents random perspective The first mean square error loss under is used to align pixel-level similarity, and the random view is from arrive Traversing, we get the first mean square error loss mentioned above; Represents random perspective The first mask mean square error loss under is used to reduce floating ghosts, and the random view angle is from arrive Traversing, we get the above first mask mean square error loss; Represents random perspective The first visual perception loss under is used to enhance visual perception, and the random view angle is arrive Traversing, we get the first visual perception loss mentioned above; Indicates the number of random perspectives; Represents random perspective Next Frame random perspective rendering image; Represents random perspective Next Frame grid rendered image; Respectively The corresponding alpha channel output; For the general The first Layer features, That is, corresponding to the first feature, Corresponding to the second feature; Respectively represent The height and width of the layer features; Respectively represent The height and width of a pixel in the layer feature.

[0064] S4: When the initial moment (i.e. After the static Gaussian model at time (time) is trained, the deformation network is trained.

[0065] It should be noted that, in step S4 of the present invention, Figure 4As shown, in the training process of the deformation network, the trained static Gaussian model at the initial moment is used as the initial static Gaussian model, the initial static Gaussian model parameters are fixed, each time point in the preset time range to be optimized is used as the moment to be optimized, the initial static Gaussian model parameters and the moment to be optimized are input into the deformation network, and the static Gaussian model variable corresponding to the moment to be optimized is output, and the static Gaussian model variable is added to the initial static Gaussian model parameters to obtain the Gaussian model to be optimized corresponding to the moment to be optimized. The Gaussian sputtering method is used to render each Gaussian model to be optimized from the main perspective camera pose, and a frame of the second rendered image is generated for each moment to be optimized. Finally, all the second rendered images constitute a set of rendered images within the time range to be optimized. Based on the target image of each frame and The static loss is calculated for the corresponding second rendered image, the mean square error between each frame of the target image and the corresponding second rendered image is calculated and used as the action alignment loss, the Gaussian sputtering method is used to render the Gaussian model to be optimized at each moment to be optimized from each random view camera pose, and a third rendered image is obtained for each moment to be optimized under each random view. The target image of each frame, the corresponding third rendered image, the random view camera pose and a random timestamp are input into the multi-view diffusion model, the reference loss is calculated based on the noise predicted by the multi-view diffusion model and the randomly sampled Gaussian noise, the weighted sum of the action alignment loss and the reference loss is used as the dynamic loss, the weighted sum of the dynamic loss and the static loss is used as the second total loss, and the deformation network parameters are updated based on minimizing the second total loss.

[0066] The above-mentioned static loss is a weighted sum of the second mean square error loss, the second mask mean square error loss and the second visual perception loss. The second mean square error loss is the sum of the mean square errors between the target image of each frame and the second rendered image of each frame. The second mask mean square error loss is the sum of the mean square errors between the alpha channel output of the target image of each frame and the alpha channel output of the corresponding second rendered image of each frame. The second visual perception loss is the sum of the mean square errors between the third feature and the fourth feature. Among them, the third feature is a layer of features extracted from a frame of target image after the pre-trained VGG network, and the fourth feature is a layer of features extracted from a frame of second rendered image after the pre-trained VGG network.

[0067] In this embodiment, the static loss The functional forms of the second mean square error loss, the second mask mean square error loss and the second visual perception loss are:

[0068]

[0069]

[0070]

[0071]

[0072] in, Both represent the balance scaling factor; For the Frame second rendered image; Indicates the main perspective Next Frame target image and The mean square error between the second rendered image of the frame, the frame index is from arrive Traversing, we get the second mean square error loss mentioned above; Indicates the main perspective Next The alpha channel of the frame target image is output and Mean squared difference between the alpha channel outputs of the second rendered image in the frame, with frame index starting from arrive Traversing, we get the second mask mean square error loss mentioned above; is the mean square error between the third feature and the fourth feature, and the frame index is from arrive Traversing, we get the above-mentioned second visual perception loss; Respectively The corresponding alpha channel output; For the general The first Layer features, That corresponds to the third feature, Corresponding to the fourth characteristic; Indicates the number of moments to be optimized.

[0073] Although a dynamic and geometrically reasonable dynamic Gaussian model can be obtained by following the above technical steps, surface splitting may still occur. Therefore, the present invention designs a reference loss to optimize the surface of the object and fill its surface holes. Furthermore, corresponding to the above static loss, the present invention also consists of two parts: motion alignment loss and reference loss. Among them, the motion alignment loss Used to inject motion information, its function form is:

[0074]

[0075] For reference loss, the present invention uses the open source Zero123 multi-view diffusion model, the implementation of which belongs to the prior art. For reference, optimization at random perspectives The performance of the dynamic Gaussian model under the above reference loss The functional form is:

[0076]

[0077] in, is the random timestamp of the multi-view diffusion model, ranging from (1,998); represents the coefficients associated with random timestamps; represents the noise predicted by the multi-view diffusion model; Represents random perspective Next time to be optimized A corresponding third rendering image is obtained; is a randomly sampled Gaussian noise.

[0078] Thus, the present invention can obtain dynamic loss and the second total loss , the two loss functions are:

[0079]

[0080]

[0081] in, Both represent the balance scaling factor; Both represent weight hyperparameters.

[0082] In the present invention, the weight hyperparameter and the balance scaling factor can be selected as appropriate values ​​according to actual conditions. In this embodiment, the weight hyperparameter and the balance scaling factor are both set to 1.

[0083] like Figure 4 As shown, the core of the present invention is to use focal length alignment and Gaussian-grid contrast learning methods to generate a geometrically consistent and texture-consistent static Gaussian model, and to design a reference optimization method based on a multi-view diffusion model based on score distillation sampling ( Figure 4 ), and then use a novel motion alignment method ( Figure 4 Motion alignment in ) and the above-mentioned reference optimization method are used to generate a dynamic Gaussian model (i.e., a four-dimensional Gaussian model), which can be used to represent 4D scenes with fine textures, accurate geometry, and coherent motion.

[0084] S5: The trained static Gaussian model at the initial moment is used as the initial static Gaussian model, the initial static Gaussian model parameters and a given timestamp are input into the trained deformation network, the static Gaussian model variable corresponding to the given timestamp is output and added to the initial static Gaussian model parameters, and the four-dimensional Gaussian model at the given timestamp is obtained, thereby completing the generation of the four-dimensional Gaussian model based on Gaussian sputtering.

[0085] The four-dimensional Gaussian model generation method based on Gaussian sputtering in the above embodiment is applied to a specific text prompt for a comparative experiment. The specific steps are as described in S1 to S5, which will not be repeated here, and the specific parameters and technical effects are mainly shown.

[0086] Example

[0087] In order to verify the effect of the present invention, a comprehensive user study was conducted according to the settings of AYG. The present invention uses the same control conditions as MAV3D and AYG to generate 4D dynamic models and obtain respective rendered videos. At the same time, users are required to vote in four aspects of texture, geometry, motion and text semantic consistency based on the visual perception of these videos, and CLIP is introduced to calculate the cosine similarity between text and rendered images to objectively measure the degree of semantic alignment. The time overhead of various 4D generation methods proposed for image to 4D and text to 4D tasks is also compared. The comparison results are shown in Tables 1 and 2, respectively. In Table 1, MAV3D is a text to 4D method based on dynamic NeRF, AYG is a text to 4D method based on a dynamic Gaussian model, and CLIP-S (↑) is a text-image semantic consistency indicator. In Table 2, Dream-in-4D is a text-to-4D method based on dynamic NeRF, Animate124 is an image-to-4D method based on dynamic NeRF, Consistent4D is an image-to-4D method based on dynamic NeRF, DreamGaussian4D is an image-to-4D method based on dynamic Gaussian model, and 4D-fy is a text-to-4D method based on dynamic NeRF.

[0088] Table 1. Texture, geometry, motion and text semantic evaluation table

[0089] Table 2. Generation efficiency evaluation table

[0090] like Figure 5 As shown, the present invention uses the text input by the user as the control condition, and it only takes about 15 minutes to generate a four-dimensional Gaussian model. From multiple novel perspectives, the target is mostly the motion state of time, and it can be found that the generated target not only has excellent geometric state, but also has a large motion state amplitude. In addition, the four-dimensional Gaussian model generated by the present invention strictly conforms to the semantics of the text input by the user.

[0091] It should also be noted that the four-dimensional Gaussian model generation method based on Gaussian sputtering in the above embodiment can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a four-dimensional Gaussian model generation system based on Gaussian sputtering corresponding to the four-dimensional Gaussian model generation method based on Gaussian sputtering provided in the above embodiment, such as Figure 6 As shown, it includes:

[0092] The Wensheng video module is used to take the given text prompt as the diffusion condition and use the pre-trained text-to-video diffusion model to perform multi-step denoising operations on the random noise that follows the normal distribution to generate a target video consistent with the text prompt;

[0093] A focal length alignment module is used to convert each target image frame in the target video into a three-dimensional mesh model by using a pre-trained image-mesh generation model, render the three-dimensional mesh model within the focal length range of the main viewing angle, obtain a first main viewing angle image for each focal length value, calculate the mean square error between a target image frame and each first main viewing angle image, and use the focal length value corresponding to the first main viewing angle image with the smallest mean square error as the rendering matching focal length of the target image frame;

[0094] The first model training module is used to train a static Gaussian model at an initial moment by using a Gaussian sputtering method and based on a rendering matching focal length;

[0095] The second model training module is used to train the deformable network after the static Gaussian model at the initial moment is trained;

[0096] The model generation module is used to use the trained static Gaussian model at the initial moment as the initial static Gaussian model, input the initial static Gaussian model parameters and a given timestamp into the trained deformation network, output the static Gaussian model variable corresponding to the given timestamp and add it to the initial static Gaussian model parameters to obtain the four-dimensional Gaussian model at the given timestamp, and complete the generation of the four-dimensional Gaussian model based on Gaussian sputtering.

[0097] It is understandable that the four-dimensional Gaussian model generation method based on Gaussian sputtering described in S1 to S5 above can be substantially implemented by a computer program. Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer program product corresponding to the four-dimensional Gaussian model generation method based on Gaussian sputtering provided in the above embodiment, which includes a computer program / instruction. When the computer program / instruction is executed by a processor, the four-dimensional Gaussian model generation method based on Gaussian sputtering as described in the above embodiment can be implemented.

[0098] Similarly, based on the same inventive concept, another preferred embodiment of the present invention further provides a computer electronic device corresponding to the four-dimensional Gaussian model generation method based on Gaussian sputtering provided in the above embodiment, which includes a memory and a processor;

[0099] The memory is used to store computer programs;

[0100] The processor is used to implement the four-dimensional Gaussian model generation method based on Gaussian sputtering in the above embodiment when executing the computer program.

[0101] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0102] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the four-dimensional Gaussian model generation method based on Gaussian sputtering provided in the above embodiment, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the four-dimensional Gaussian model generation method based on Gaussian sputtering in the above embodiment can be implemented.

[0103] It is understandable that the above storage medium may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. The storage medium may also be a U disk, a mobile hard disk, a magnetic disk or an optical disk, etc., which can store program codes.

[0104] It is understandable that the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0105] It should also be noted that those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the various embodiments provided in this application, the division of steps or modules in the system and method is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or steps can be combined or integrated together, and a module or step can also be split.

[0106] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A method for generating a four-dimensional Gaussian model based on Gaussian sputtering, characterized in that: The following steps are involved: S1: Taking the given text prompt as the diffusion condition, the pre-trained text-to-video diffusion model is used to perform multi-step denoising operations on the random noise that follows the normal distribution to generate a target video consistent with the text prompt; S2: Each target image frame in the target video is converted into a three-dimensional mesh model by the pre-trained image-mesh generation model, and the three-dimensional mesh model is rendered within the focal length range of the main view. Each focal length value corresponds to a first main view image, and the mean square error between a target image frame and each first main view image is calculated. The focal length value corresponding to the first main view image with the smallest mean square error is used as the rendering matching focal length of the target image frame; S3: A static Gaussian model at the initial moment is trained using the Gaussian sputtering method and based on the rendering matching focal length of S2; S4: After the static Gaussian model at the initial moment is trained, the deformation network is trained; S5: The trained static Gaussian model at the initial moment is used as the initial static Gaussian model, the initial static Gaussian model parameters and a given timestamp are input into the trained deformation network, the static Gaussian model variable corresponding to the given timestamp is output and added to the initial static Gaussian model parameters, and the four-dimensional Gaussian model at the given timestamp is obtained, thereby completing the generation of the four-dimensional Gaussian model based on Gaussian sputtering.

2. The method for generating a four-dimensional Gaussian model based on Gaussian sputtering according to claim 1, characterized in that: In step S3, during the training process of the static Gaussian model, the static Gaussian model is first rendered from the main perspective camera pose by the Gaussian sputtering method to obtain a first rendered image, the mean square error between the target image and the corresponding first rendered image is calculated and used as the texture alignment loss, the static Gaussian model is then rendered from different random perspective camera poses by the Gaussian sputtering method, and a random perspective rendered image is obtained for each random perspective, the three-dimensional mesh model is rendered from each random perspective camera pose by the rasterization method, and a mesh rendered image is obtained for each random perspective, the geometric alignment loss is calculated based on the random perspective rendered image and the mesh rendered image at each random perspective, the weighted sum of the texture alignment loss and the geometric alignment loss is taken as the first total loss, and the static Gaussian model parameters are updated based on minimizing the first total loss.

3. The method for generating a four-dimensional Gaussian model based on Gaussian sputtering according to claim 2, characterized in that: The geometric alignment loss is formed by weighted summation of a first mean square error loss, a first mask mean square error loss and a first visual perception loss, wherein the first mean square error loss is the sum of the mean square errors between the random perspective rendered image and the mesh rendered image at each random perspective, the first mask mean square error loss is the sum of the mean square errors between the alpha channel output of the random perspective rendered image and the alpha channel output of the mesh rendered image at each random perspective, and the first visual perception loss is the sum of the mean square errors between the first feature and the second feature at each random perspective; wherein the first feature is a layer of features extracted from the random perspective rendered image after the pre-trained VGG network, and the second feature is a layer of features extracted from the mesh rendered image after the pre-trained VGG network.

4. The method for generating a four-dimensional Gaussian model based on Gaussian sputtering according to claim 1, characterized in that: In step S4, during the training process of the deformation network, the initial static Gaussian model parameters are fixed, each time point within the preset time range to be optimized is used as the moment to be optimized, the initial static Gaussian model parameters and the moment to be optimized are input into the deformation network, the static Gaussian model variables corresponding to the moment to be optimized are output, the static Gaussian model variables are added to the initial static Gaussian model parameters, and the Gaussian model to be optimized corresponding to the moment to be optimized is obtained, and the Gaussian sputtering method is used to render each Gaussian model to be optimized from the main perspective camera pose, and a frame of the second rendered image is generated for each moment to be optimized, and the static loss is calculated based on each frame of the target image and the corresponding second rendered image, and the loss between each frame of the target image and the corresponding second rendered image is calculated. The mean square error between the corresponding second rendered images is used as the action alignment loss, and the Gaussian sputtering method is used to render the Gaussian model to be optimized at each time to be optimized from each random view camera pose. A third rendered image is obtained corresponding to each time to be optimized under each random view. The target image of each frame, the corresponding third rendered image, the random view camera pose and a random timestamp are input into the multi-view diffusion model. The reference loss is calculated based on the noise predicted by the multi-view diffusion model and the randomly sampled Gaussian noise. The weighted sum of the action alignment loss and the reference loss is used as the dynamic loss, and the weighted sum of the dynamic loss and the static loss is used as the second total loss. The deformation network parameters are updated based on minimizing the second total loss.

5. The method for generating a four-dimensional Gaussian model based on Gaussian sputtering according to claim 4, characterized in that: The static loss is a weighted sum of a second mean square error loss, a second mask mean square error loss and a second visual perception loss, the second mean square error loss is the sum of the mean square errors between the target image of each frame and the second rendered image of each frame, the second mask mean square error loss is the sum of the mean square errors between the alpha channel output of the target image of each frame and the alpha channel output of the corresponding second rendered image of each frame, and the second visual perception loss is the sum of the mean square errors between the third feature and the fourth feature; wherein the third feature is a layer of features extracted from a frame of target image after a pre-trained VGG network, and the fourth feature is a layer of features extracted from a frame of second rendered image after a pre-trained VGG network.

6. The method for generating a four-dimensional Gaussian model based on Gaussian sputtering according to claim 4, characterized in that: The reference loss The functional form is: ； in, Indicates the number of random perspectives; Indicates the number of moments to be optimized; is the random timestamp of the multi-view diffusion model; represents the coefficients associated with random timestamps; represents the noise predicted by the multi-view diffusion model; Represents random perspective Next time to be optimized A corresponding third rendering image is obtained; Indicates Frame target image; is a randomly sampled Gaussian noise.

7. A four-dimensional Gaussian model generation system based on Gaussian sputtering, characterized in that: include: The Wensheng video module is used to take the given text prompt as the diffusion condition and use the pre-trained text-to-video diffusion model to perform multi-step denoising operations on the random noise that follows the normal distribution to generate a target video consistent with the text prompt; A focal length alignment module is used to convert each target image frame in the target video into a three-dimensional mesh model by using a pre-trained image-mesh generation model, render the three-dimensional mesh model within the focal length range of the main viewing angle, obtain a first main viewing angle image for each focal length value, calculate the mean square error between a target image frame and each first main viewing angle image, and use the focal length value corresponding to the first main viewing angle image with the smallest mean square error as the rendering matching focal length of the target image frame; The first model training module is used to train a static Gaussian model at an initial moment by using a Gaussian sputtering method and based on a rendering matching focal length; The second model training module is used to train the deformable network after the static Gaussian model at the initial moment is trained; The model generation module is used to use the trained static Gaussian model at the initial moment as the initial static Gaussian model, input the initial static Gaussian model parameters and a given timestamp into the trained deformation network, output the static Gaussian model variable corresponding to the given timestamp and add it to the initial static Gaussian model parameters to obtain the four-dimensional Gaussian model at the given timestamp, and complete the generation of the four-dimensional Gaussian model based on Gaussian sputtering.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the four-dimensional Gaussian model generation method based on Gaussian sputtering as described in any one of claims 1 to 6 can be implemented.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the four-dimensional Gaussian model generation method based on Gaussian sputtering as described in any one of claims 1 to 6 is implemented.

10. A computer electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the four-dimensional Gaussian model generation method based on Gaussian sputtering as described in any one of claims 1 to 6 when executing the computer program.

Images