Method for enhancing three-dimensional digital human generation through two-dimensional editing

Abstract

The invention relates to the technical field of deep learning, and discloses a method for enhancing three-dimensional digital human generation through two-dimensional editing, and the method comprises the steps: carrying out the rendering of a three-dimensional digital human under a given camera pose through an alpha-blowing technology, and obtaining a rendered image; defining a noise adding track and a corresponding reverse denoising track for the rendered image; a denoising diffusion implicit model process guided by an ordinary differential equation is adopted in the noise adding process, and guide information calculated by a classifier-free guide formula is used in the denoising process; aligning corresponding latent variables on the noise adding track and the reverse de-noising track by using an L2 loss function, and constructing a loss function; a plurality of Gaussian functions representing coefficients applied to the loss function in each iteration are introduced to obtain a final loss function, and the three-dimensional representation is updated by minimizing the final loss function. According to the method, by introducing a multi-step denoising mechanism, details and information are recovered step by step, the precision of generated details is improved, and the excessive smoothness phenomenon is reduced.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a method for generating a three-dimensional digital human by enhancing the three-dimensional editing. Background Art

[0002] 3D Gaussian is an emerging three-dimensional representation method that uses 3D Gaussian distribution in space to express scenes. Each 3D Gaussian contains the following properties: mean, covariance matrix, opacity, and spherical harmonics. Among them, the mean represents the center position of the 3D Gaussian; the covariance matrix describes its size and shape. In order to ensure that the covariance matrix has physical meaning, it is usually decomposed into the product of the rotation matrix and the scaling matrix; the spherical harmonics represent the color information of the 3D Gaussian itself, as well as the color information related to the viewing direction. The degree (k) of the spherical harmonics determines the richness of the color expression. The higher the degree, the more complex the color represented, and the value is usually 3.

[0003] When rendering, the scene represented by 3D Gaussian is rendered onto a two-dimensional plane using a point-based rendering method (α-blending). Specifically, the color of a pixel on the rendering plane is the result of a weighted blend of the colors and opacity of multiple 3D Gaussians:

[0004]

[0005] Among them, C out is the color of a pixel on the rendering plane, N is the ordered 3D Gaussian sequence projected onto the current pixel under a given camera pose, and c i is the color of the i-th 3D Gaussian. α is obtained by multiplying the opacity of the center point of the 3D Gaussian with a two-dimensional Gaussian distribution with covariance Σ′. i , where Σ′=JWΣW T J T , the affine approximation of the projection transformation is completed by the Jacobian matrix J, W is the perspective transformation matrix, and Σ is the covariance matrix of the 3D Gaussian.

[0006] The diffusion model generates data by estimating the denoised gradient direction of the current state distribution. During training, the model first adds Gaussian noise ε of time step t to the input data x and obtains x t :

[0007]

[0008] The model then minimizes the prediction noise The loss between the actual noise ε is used to learn the noise prediction network

[0009]

[0010] During inference, it completes data generation by making predictions step by step and removing noise.

[0011] DDIM sampling is used to estimate the relationship between any two noisy latent variables and is therefore often used to accelerate the inference process. m Convert to x k (regardless of whether m>k or m <k):

[0012]

[0013] Classifier-free guidance (CFG) is a technique used in the generation process to achieve text-guided synthesis without relying on explicit classifiers. It introduces a mechanism that enables unconditional generation and conditional generation to collaboratively control image generation. Its formula is as follows:

[0014]

[0015] where y represents the conditional text embedding obtained from the input prompt, Represents the unconditional empty text embedding, and ω represents the guidance coefficient. The larger the value of ω, the more the generated image conforms to the text y, but the image quality may be reduced.

[0016] Fractional Distilled Sampling (SDS) aims to distill information from a pre-trained 2D text-to-image diffusion model. First, a camera pose c is randomly selected to render a differentiable 3D representation θ, using a rendering function g(θ,c) to obtain an image x. Then, random t timestep noise is added to x and the image is obtained by Denoising. The 3D representation is updated by minimizing the difference between these two noises:

[0017]

[0018] Where f(t) is a weight function that is adjusted according to the time step t. In SDS, the CFG coefficient is set to 100.

[0019] In the traditional 3D digital human modeling process, the creation and editing of digital human models usually requires complex 3D design tools and professional skills. With the advancement of technology, extracting 3D digital human representations from pre-trained 2D diffusion models has attracted widespread attention in the AIGC field. This method can generate a 3D representation of a digital human from scratch based on the text prompts provided by the user, greatly reducing the workload of 3D modelers.

[0020] Current research methods mainly follow SDS, which relies on single-step editing and has limitations in efficiently extracting information from diffusion models. It often leads to obvious directional errors, and the generated images are over-saturated and over-smoothed, resulting in the inability to generate realistic 3D digital human content. A series of improvements have been introduced to the SDS-based 3D digital human generation method, including more advanced three-dimensional representation, optimized diffusion time step for sampling, and so on. Despite these advances, problems such as over-saturation, over-smoothing, and poor content still exist.

[0021] In contrast, we find that traditional diffusion models often require multi-step denoising, which involves adding information of different granularities, to correct directional errors and generate high-quality images, whereas in a single step, directly predicting x 0 The direction of the image usually leads to significant deviations from the real data trajectory. Existing 3D digital human generation technologies often ignore the inherent information and editing functions in 2D images, resulting in low efficiency in the creation of 3D digital humans and lack of sufficient flexibility and precision. Therefore, how to effectively enhance the effect of 3D digital human generation through simple 2D editing methods is a key application problem that needs to be solved urgently. Summary of the invention

[0022] In order to solve the above technical problems, the present invention provides a method for generating a three-dimensional digital human by enhancing the three-dimensional editing.

[0023] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0024] A method for generating a three-dimensional digital human by enhancing two-dimensional editing comprises the following steps:

[0025] a) Rendering the 3D digital human under a given camera pose by using α-blending technology to obtain a rendered image;

[0026] b) for the rendered image, defining an n-step denoising trajectory and a corresponding reverse denoising trajectory;

[0027] c) The denoising process uses an ordinary differential equation guided denoising diffusion implicit model process and uses empty text embedding to retain the characteristics of the input image; the denoising process uses the guidance information calculated by the classifier-free guidance formula to obtain semantic information of different granularities through a step-by-step denoising process;

[0028] d) Use the L2 loss function to align the corresponding latent variables on the noisy trajectory and the reverse denoised trajectory, and construct the loss function L i , to learn semantic information of different granularities from the pre-trained 2D diffusion model from the current view;

[0029] e) Introduce the loss function L applied to each iteration i Multiple Gaussian functions of the coefficients are used to simulate the characteristic that the coefficients first increase and then decrease, and the loss function L is calculated according to the coefficients. i Perform weighted summation to obtain the final loss function, and update the three-dimensional representation by minimizing the final loss function;

[0030] g) Repeat steps a) to e) until the set stop condition is reached to obtain a three-dimensional digital human representation.

[0031] Furthermore, step b) specifically includes: for rendering the image Define an n-step noise trajectory and a corresponding inverse denoising trajectory t n >t n-1 >…>t 1 >t 0 , is the tth diffusion trajectory i latent variables, is the tth in the reverse denoising trajectory i A latent variable.

[0032] Furthermore, step c) specifically includes:

[0033] The predicted noise of the noise adding process is obtained through the trained U-Net network

[0034]

[0035] According to the DDIM sampling formula, the noise image of the next time step is obtained:

[0036]

[0037] Among them, the intermediate variable

[0038] The predicted denoising process noise is obtained through the trained U-Net network U-Net(·)

[0039]

[0040] According to the DDIM sampling formula, the predicted image of the previous time step is obtained:

[0041]

[0042] Where y represents the conditional text embedding in the input prompt, Represents an unconditional empty text embedding.

[0043] Further, step d) specifically includes:

[0044] Use L2 loss to align the corresponding latent variables on the noisy trajectory and the reverse denoised trajectory, and construct the loss function L i :

[0045]

[0046] Furthermore, step e) specifically includes:

[0047] Use multiple Gaussian functions to simulate the characteristic that the coefficients increase first and then decrease:

[0048]

[0049] Among them, Δ i represents the number of iterations at which the peak of the i-th Gaussian function appears, σ represents the covariance; for the endpoint Δ n-1 =0 and Δ 0 = k, k represents the total number of iterations, and Δ i Set to equal spacing:

[0050]

[0051] Normalize all coefficients ω(i,k) under the same iteration k to obtain the normalized coefficients

[0052]

[0053] The final loss function is as follows:

[0054]

[0055] The 3D representation is updated by minimizing the final loss function.

[0056] Compared with the prior art, the beneficial technical effects of the present invention are:

[0057] The existing 3D digital human generation method based on diffusion model often has the phenomenon of missing details or over-smoothing during the generation process, especially in the details of facial features, clothing texture, hairstyle, etc. In view of the above problems, the present invention provides a method for enhancing the generation of 3D digital humans through 2D editing, which gradually restores details and information by introducing a multi-step denoising mechanism, improves the accuracy of generated details, and reduces the over-smoothing phenomenon. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is a flow chart of a method for enhancing three-dimensional digital human generation through two-dimensional editing in an embodiment of the present invention;

[0059] Figure 2 This is a comparison of the effects of generating a three-dimensional digital human in an embodiment of the present invention;

[0060] Figure 3 It is a schematic diagram of the effect of generating a three-dimensional digital human in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] A preferred embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0062] Current methods based on fractional distillation sampling (SDS) have limitations in efficiently extracting information from diffusion models, resulting in the inability to generate realistic 3D content. This paper revisits the fractional distillation sampling (SDS) method and analyzes its essence as a basic image editing process, which often suffers from oversaturation, oversmoothing, and lack of rich content due to low-quality single-step denoising. By fully leveraging the multi-step denoising capabilities of the diffusion model in each optimization iteration, high-quality image generation can be achieved, thereby facilitating a comprehensive image editing process. Therefore, this paper proposes a method for enhancing 3D digital human generation through 2D editing. In each iteration, the present invention establishes a denoising trajectory that projects the input image into a high-dimensional diffusion space, accompanied by a denoising editing trajectory that gradually incorporates information of different granularities from the diffusion model; wherein larger time steps provide coarse-grained information, while smaller time steps provide fine-grained information. By minimizing the differences in latent variables at corresponding time steps along these trajectories, multi-granular information can be effectively refined into a 3D representation.

[0063] The text-based 2D editing task is essentially an image-to-image task. The original image is input into a pre-trained image generation model based on the mapping relationship between text and image. Subsequently, the image is mapped to a high-dimensional latent space through a denoising process. During the denoising process, the edited text information is gradually introduced to achieve the editing of the image. The edited image needs to retain most of the content of the original image. Similarly, the text-to-3D method based on fractional distillation sampling (SDS) exploits the prior knowledge of the pre-trained text-to-image model and integrates this information into a differentiable 3D representation. In each optimization step, a camera pose is randomly selected to obtain the rendered image, which is then input into a pre-trained diffusion model, which is similar to the input process in the editing task. By minimizing the noise difference between single-step denoising and denoising, the text information is gradually introduced. This process is similar to the editing task, where the editing information is gradually introduced through denoising. Therefore, text-to-3D can be regarded as invoking a text-guided editing task in each optimization step.

[0064] In text-based 2D editing tasks, the input image x is usually first inverted through an alignment process. The present invention defines a diffusion trajectory for the noise addition process and a denoising track Need to gradually align the corresponding latent variables and To achieve the inversion. Methods such as NTI and PTI use differentiable embeddings to represent unconditional empty text in the denoising process. or conditional text t , thereby achieving alignment optimization. This process is expressed as:

[0065]

[0066] where i = n,…,1, or After this alignment process, the new text embedding y edit Can be used with optimized embedding sequence Combine and edit.

[0067] For fractional distillation sampling (SDS), first n The time step is directly Add noise, and then perform a denoising step to get The following formula can be derived:

[0068]

[0069] By eliminating You can get:

[0070]

[0071] in, It is the core of the Fractional Distillation Sampling (SDS) formula.

[0072] LucidDreamer is the current state-of-the-art (SOTA) method. Its Interval Score Matching (ISM) method combines the Replace with where x t is by using DDIM to reverse x s Add noise to get. Set t n-1 =s,t n = t, and by Add noise to get Then, the DDIM denoising formula is used to Denoise to restore The equation in this process can be expressed as:

[0073]

[0074] in By adding these two equations and eliminating You can get:

[0075]

[0076] Based on the above derivation, it can be observed that fractional distillation sampling (SDS) and interval score matching (ISM) essentially perform the inversion of diffusion trajectories in the two-dimensional editing task on latent variables at different time steps. (i=0 or n-1) are aligned.

[0077] As mentioned above, it can be seen that current methods based on fractional distillation sampling (SDS) essentially only provide single-step denoising information. In contrast, traditional diffusion models rely on multi-step denoising to integrate information at different granularities to achieve high-quality generation. In the early denoising stages, the model captures coarse-grained details (e.g., the location and overall shape of objects), while in the later stages, fine-grained details (e.g., the edges of objects) are obtained. Therefore, in 2D editing tasks, aligning information across multiple denoising steps is crucial to achieve accurate image inversion and ensure high-quality image editing.

[0078] Therefore, the present invention fully integrates the 2D editing task with the 3D generation task. In each 3D generation iteration, a multi-step denoising editing process is applied to extract rich semantic features (both coarse-grained and fine-grained). Then, a reconstruction method similar to the above alignment optimization formula is used to align the latent variables of the noise nodes and the corresponding denoised nodes to achieve high-quality 3D representation generation. The specific steps are as follows:

[0079] First, randomly select a camera pose c to render a differentiable 3D representation θ and use the rendering function g(θ,c) to obtain the rendered image

[0080]

[0081] For rendered images Define an n-step noise trajectory and a corresponding inverse denoising trajectory The denoising and de-noising process is as follows:

[0082]

[0083] in It represents the predicted noise obtained by the U-Net network. The predicted noise during the noise addition process is The denoising process uses the guidance information calculated by the classifier-free guidance formula, which is as follows:

[0084]

[0085] where y represents the conditional text embedding in the input hint, Represents the unconditional empty text embedding, ω represents the guidance coefficient, the larger the value, the more consistent the generated image is with the text y, but the image quality may be reduced.

[0086] The L2 loss is used to align the corresponding latent variables on the two trajectories, as follows:

[0087]

[0088] This enables the 3D representation to learn rich semantic information at different granularities from the pre-trained 2D diffusion model from the current viewpoint, thereby generating high-quality 3D content.

[0089] Introduce ω(i,k) to represent the loss function L applied to the kth iteration i The coefficient of, when k is small, comes from the coarse-grained loss L i The contribution of ω(i,k) (larger i) is larger. As the optimization proceeds, more fine-grained information (smaller i) becomes more important. The present invention uses multiple Gaussian functions to simulate the characteristic that the coefficient increases first and then decreases. The Gaussian functions are as follows:

[0090]

[0091] Among them, Δ i represents the number of iterations at which the peak of the i-th Gaussian function appears, for the endpoint Δ n-1 =0 and Δ 0 = k (k represents the total number of iterations), and Δ i Set to equal intervals, as follows:

[0092]

[0093] Then all coefficients ω(i,k) under the same iteration k are normalized, and the formula is as follows:

[0094]

[0095] The final loss function is as follows:

[0096]

[0097] The 3D representation is updated by minimizing this loss function.

[0098] The current open source SOTA method is used as a benchmark to compare and analyze the text-generated 3D digital human method of the present invention, including DreamFusion, Magic3D, ProlificDreamer, GaussianDreamer and LucidDreamer; DreamFusion, ProlificDreamer and Magic3D have been re-implemented by Three-studio. CLIP similarity, Frechet Inception Distance (FID), and the image quality score predicted by the reference-free spatial domain image quality assessment algorithm BRISQUE are used as evaluation indicators, where CLIP similarity measures the alignment between the generated 3D representation rendering image and the corresponding text description, FID is used to evaluate the difference between the generated model and the real data distribution, and BRISQUE evaluates the image space quality, the smaller the value, the higher the quality. The results of the indicator calculation are shown in Table 1. The results of CLIP similarity show that the 3D digital human generated by the present invention is more aligned with the input text prompt. This advantage comes from the multi-step editing process in each optimization, which can incorporate richer text information. In addition, the quantitative evaluation of FID and BRISQUE scores further confirms that the method of the present invention can generate 3D content with better generation quality.

[0099] Table 1:

[0100]

[0101] At the same time, the quality of the 3D digital human generated by this embodiment is significantly better than that of other methods. Figure 2 .

[0102] Figure 3 3D digital human generation effect diagram according to an embodiment of the present invention. The first two rows generate 3D human faces, and the last two rows generate 3D digital human avatars, and the results are almost indistinguishable from real images.

[0103] Example

[0104] This embodiment selects 3D Gaussian distribution for three-dimensional representation and uses Point-E to generate the initial sparse Gaussian point cloud of the object. This embodiment selects Stable Diffusion 2.1 as the diffusion model from 2D text to image. In the diffusion trajectory, the time node range is defined from 5 to 10, and the step size between nodes is set to 40 to 60, 60 to 80, and 80 to 100. The classifier-free guidance (CFG) coefficient is set to 7.5.

[0105] like Figure 1As shown, the method for enhancing the generation of a three-dimensional digital human through two-dimensional editing in this embodiment includes operations S110 to S160.

[0106] Operation S110, rendering a differentiable 3D representation θ according to the selected camera view c, and obtaining a rendered image using a rendering function g(θ, c)

[0107]

[0108] Operation S120, for rendering the image Define an n-step noise trajectory and a corresponding reverse denoising trajectory In this embodiment, n=10 is set.

[0109] In operation S130, the predicted noise is first obtained through the trained U-Net network:

[0110]

[0111] According to the DDIM sampling formula, the noise image of the next time step is obtained:

[0112]

[0113] in,

[0114] Operation S140, obtaining predicted denoising noise through the trained U-Net network:

[0115]

[0116] According to the DDIM sampling formula, the predicted image of the previous time step is obtained:

[0117]

[0118] in Same as the noise adding process, y represents the conditional text embedding in the input prompt, Represents an unconditional empty text embedding.

[0119] In operations S150 and S160, L2 loss is used to align corresponding latent variables on two tracks, which enables the 3D representation to learn rich semantic information of different granularities from the pre-trained 2D diffusion model from the current viewpoint, thereby generating high-quality 3D content.

[0120] Introduce ω(i,k) to represent the loss function L applied to the kth iteration iThe present embodiment uses multiple Gaussian functions to simulate the characteristic that the coefficient increases first and then decreases. The Gaussian functions are as follows:

[0121]

[0122] Among them, Δ i represents the number of iterations at which the peak of the i-th Gaussian function appears, for the endpoint Δ n-1 =0,Δ 0 = k (k represents the total number of iterations, and in this embodiment, k is set to 5000), and Δ i Set to equal intervals, as follows:

[0123]

[0124] At the same time, a larger σ value needs to be set to ensure that information of different granularities can participate in the optimization process in each iteration. In this embodiment, it is set to 1000. Then, all coefficients ω(i, k) under the same iteration k are normalized. The formula is as follows:

[0125]

[0126] The final loss function is as follows:

[0127]

[0128] The 3D representation is updated by minimizing this loss function.

[0129] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.

[0130] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A method for generating a three-dimensional digital human by enhancing two-dimensional editing, characterized in that: The following steps are involved: a) Rendering the 3D digital human under a given camera pose by using α-blending technology to obtain a rendered image; b) for the rendered image, defining an n-step denoising trajectory and a corresponding reverse denoising trajectory; c) The denoising process uses an ordinary differential equation guided denoising diffusion implicit model process and uses empty text embedding to preserve the characteristics of the input image; The denoising process uses the guidance information calculated by the classifier-free guidance formula to obtain semantic information of different granularities through a step-by-step denoising process; d) Use the L2 loss function to align the corresponding latent variables on the noisy trajectory and the reverse denoised trajectory, and construct the loss function L i , to learn semantic information of different granularities from the pre-trained 2D diffusion model from the current view; e) Introduce the loss function L applied to each iteration i Multiple Gaussian functions of the coefficients are used to simulate the characteristic that the coefficients first increase and then decrease, and the loss function L is calculated according to the coefficients. i Perform weighted summation to obtain the final loss function, and update the three-dimensional representation by minimizing the final loss function; g) Repeat steps a) to e) until the set stop condition is reached to obtain a three-dimensional digital human representation.

2. The method for generating a three-dimensional digital human by enhancing two-dimensional editing according to claim 1, characterized in that: Step b) specifically includes: for rendering the image Define an n-step noise trajectory and a corresponding inverse denoising trajectory t n >t n-1 >…>t1>t0, is the tth diffusion trajectory i latent variables, is the tth in the reverse denoising trajectory i A latent variable.

3. The method for generating a three-dimensional digital human by enhancing two-dimensional editing according to claim 2, characterized in that: Step c) specifically comprises: The predicted noise of the noise adding process is obtained through the trained U-Net network According to the DDIM sampling formula, the noise image of the next time step is obtained: Among them, the intermediate variable The predicted denoising process noise is obtained through the trained U-Net network U-Net(·) According to the DDIM sampling formula, the predicted image of the previous time step is obtained: where y represents the conditional text embedding in the input prompt, Represents an unconditional empty text embedding.

4. The method for generating a three-dimensional digital human by enhancing two-dimensional editing according to claim 2, characterized in that: Step d) specifically comprises: Use L2 loss to align the corresponding latent variables on the noisy trajectory and the reverse denoised trajectory, and construct the loss function L i :

5. The method for generating a three-dimensional digital human by enhancing two-dimensional editing according to claim 2, characterized in that: Step e) specifically comprises: Use multiple Gaussian functions to simulate the characteristic that the coefficients increase first and then decrease: Among them, Δ i represents the number of iterations at which the peak of the i-th Gaussian function appears, σ represents the covariance; for the endpoint Δ n-1 =0 and Δ0=k, where k represents the total number of iterations. i Set to equal spacing: Normalize all coefficients ω(i,k) under the same iteration k to obtain the normalized coefficients The final loss function is as follows: The 3D representation is updated by minimizing the final loss function.

Images