Three-dimensional reconstruction method and device for reconstructing three-dimensional human body from natural language description text

Abstract

The invention relates to a three-dimensional reconstruction method and device for reconstructing a three-dimensional human body from a natural language description text, and relates to the technical field of computer vision, and the method comprises the steps: carrying out the image rendering through an initial three-dimensional Gaussian field constructed through a human body template, and obtaining a human body image and a depth image, the method comprises the following steps of: adding noise to an image by using Gaussian noise at a target moment to obtain a noise image set, performing diffusion estimation on the noise image set in combination with a cue word of which the weight is improved to obtain Gaussian noise at a previous moment, performing gradient return in combination with the Gaussian noise at two moments, and performing loss updating by using a hand depth map in a three-dimensional Gaussian field to obtain a hand depth image. And obtaining the updated three-dimensional human body after three-dimensional reconstruction. According to the method, the depth loss is utilized, and the depth of the human body model and the depth of the local body part are aligned, so that the generated 3D human body can accurately capture the complex features of the human body part, the fine human body part is recovered, and the problem that an existing three-dimensional human body construction technology cannot generate the accurate body part is solved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a three-dimensional reconstruction method and device for reconstructing a three-dimensional human body from a natural language description text. Background Art

[0002] Currently, creating personalized 3D (three-dimensional, or 3D) virtual avatars is crucial for content generation in various applications, such as virtual try-on, gaming, and virtual reality. Users want to be able to customize their characters, from changing their specific identities (such as clothes, hair, etc.) to applying a specific artistic style. However, traditional methods of creating digital avatars usually require skilled artists to perform tedious and time-consuming work such as multi-view capture, texturing, and animation. Recent advances in avatar technology have demonstrated more user-friendly solutions, namely automatically creating high-quality 3D avatars through natural language descriptions alone.

[0003] Existing 3D human body construction methods tend to generate 3D humans with blurred local parts, such as hands and feet. They rely heavily on text-to-image models, such as diffusion models, and are prone to generating unreasonable human body parts. The reason is that text-to-image (T2I) models usually do not explicitly simulate the fine-grained structure of the human body and are often plagued by the "Janus (polyhedron)" problem. For example, the hand image generated by the diffusion model may have errors in multiple parts, such as six fingers on the hand, etc. The existing technology tends to ignore the impact of these erroneous object information on 3D human body generation, resulting in insufficient accurate local limbs in 3D human body generation. It can be seen that the existing technology is affected by inaccurate information and cannot generate accurate body parts. Summary of the invention

[0004] The present application provides a 3D reconstruction method and device for reconstructing a 3D human body from a natural language description text. Using prompt weights on key prompt words helps to match the generated content and text prompts. Using depth loss, the human body model and the depth of local body parts are aligned, so that the generated 3D human body can accurately capture the complex features of human body parts and restore fine human body parts, solving the problem that existing 3D human body construction technology cannot generate accurate body parts.

[0005] In a first aspect, the present application provides a 3D reconstruction method for reconstructing a 3D human body from a natural language description text, comprising:

[0006] Determine the target prompt word after the weight is increased, and construct an initial three-dimensional Gaussian field based on the initial human body template;

[0007] Performing image rendering through the initial three-dimensional Gaussian field to obtain a human body image and a depth image;

[0008] According to the human body image and the depth image, a first Gaussian noise at a target moment is used to perform noise addition to obtain a noise image set, wherein the noise image set includes a human body noisy image corresponding to the human body image and a depth noisy image corresponding to the depth image;

[0009] Based on the target prompt word, the diffusion estimation is performed in combination with the noise image set to obtain the second Gaussian noise of the previous moment of the target moment;

[0010] Gradient feedback is performed based on the first Gaussian noise and the second Gaussian noise, and the loss is updated using the hand depth map in the three-dimensional Gaussian field to obtain an updated target three-dimensional Gaussian field, where the target three-dimensional Gaussian field includes a three-dimensionally reconstructed three-dimensional human body.

[0011] Optionally, determining the target prompt word after the weight is increased includes:

[0012] Get the preset basic prompt words and camera posture information;

[0013] Determine direction information according to the camera posture information, and obtain detail prompt words and weight information of body parts;

[0014] On the basis of the basic prompt word, the direction information, the detail prompt word and the weight information are used to perform weight enhancement to obtain the target prompt word.

[0015] Optionally, an initial three-dimensional Gaussian field is constructed based on an initial human body template, including:

[0016] Get the initial human body template of the initialization posture;

[0017] Initial construction is performed according to the initial human body template to obtain a three-dimensional Gaussian field.

[0018] Optionally, the step of adding noise using the first Gaussian noise at a target time to obtain a noise image set includes:

[0019] Obtain the first Gaussian noise through noise sampling and determine the target time;

[0020] The human body image and the depth image are denoised by using the first Gaussian noise and the target moment through a preset sampler to obtain a noise image set at the target moment.

[0021] Optionally, performing gradient backpropagation according to the first Gaussian noise and the second Gaussian noise, and performing loss update using the hand depth map in the three-dimensional Gaussian field to obtain an updated target three-dimensional Gaussian field includes:

[0022] Performing mean square error calculation based on the first Gaussian noise and the second Gaussian noise to obtain a depth loss function, and performing gradient backpropagation update based on the depth loss function to obtain an updated three-dimensional Gaussian field;

[0023] Aligning the hand depth map in the updated three-dimensional Gaussian field with the hand depth map in the initial human body template to obtain a correlation loss function;

[0024] The updated three-dimensional Gaussian field is updated twice using the correlation loss function to obtain a target three-dimensional Gaussian field.

[0025] Optionally, performing image rendering through the initial three-dimensional Gaussian field to obtain a human body image and a depth image includes:

[0026] Using the preset renderer, according to the formula Rendering a human body image and a depth image of a human body from the camera pose information and the initial three-dimensional Gaussian field;

[0027] Among them, I gs is the rendered human body image, d gs is the rendered depth image, pyrenderer is the renderer, is the initial three-dimensional Gaussian field, and camera_pose is the camera pose information.

[0028] Optionally, the step of performing diffusion estimation based on the target prompt word in combination with the noise image set to obtain a second Gaussian noise at a previous moment of the target moment includes:

[0029] Using the formula Taking the target prompt word as a condition, the human body noisy image and the depth noisy image are respectively input into the UNet network for processing to obtain the second Gaussian noise of the previous moment of the target moment;

[0030] in, is the second Gaussian noise corresponding to the human body noisy image, is the second Gaussian noise corresponding to the depth-noised image, Provide the target cue word.

[0031] Optionally, performing mean square error calculation according to the first Gaussian noise and the second Gaussian noise to obtain a depth loss function, and performing gradient backpropagation update based on the depth loss function to obtain an updated three-dimensional Gaussian field, including:

[0032] according to Calculate the mean square error and get the depth loss function;

[0033] according to Perform gradient backpropagation to update the three-dimensional Gaussian field;

[0034] in, and are loss functions, η is the learning rate, and it is continuously updated during the training update process of the three-dimensional Gaussian field.

[0035] Optionally, the aligning the hand depth map in the updated three-dimensional Gaussian field with the hand depth map in the initial human body template to obtain a correlation loss function includes:

[0036] according to Align the two hand depth maps and use the Pearson correlation loss;

[0037] Among them, d gs,hand To update the hand depth map in the 3D Gaussian field, d SMPL-X,hand It is the hand depth map in the initial human template.

[0038] In a second aspect, the present application provides a 3D reconstruction device for reconstructing a 3D human body from a natural language description text, comprising:

[0039] The prompt word acquisition and model construction module is used to determine the target prompt word after the weight is increased, and to construct an initial three-dimensional Gaussian field based on the initial human body template;

[0040] An image rendering module, used for performing image rendering through the initial three-dimensional Gaussian field to obtain a human body image and a depth image;

[0041] An image denoising module, configured to perform denoising using a first Gaussian noise at a target time according to the human body image and the depth image, to obtain a noisy image set, wherein the noisy image set includes a human body denoised image corresponding to the human body image and a depth denoised image corresponding to the depth image;

[0042] A diffusion estimation module, used for performing diffusion estimation based on the target prompt word and the noise image set to obtain a second Gaussian noise at a previous moment of the target moment;

[0043] The three-dimensional Gaussian field update module is used to perform gradient backpropagation according to the first Gaussian noise and the second Gaussian noise, and use the hand depth map in the three-dimensional Gaussian field to perform loss update to obtain an updated target three-dimensional Gaussian field, wherein the target three-dimensional Gaussian field includes a three-dimensional human body after three-dimensional reconstruction.

[0044] In summary, the embodiment of the present application uses the initial human body template to construct an initial three-dimensional Gaussian field, and obtains the target prompt word after the weight is increased. Then, the image rendering is performed through the initial three-dimensional Gaussian field to obtain a human body image and a depth image. According to the human body image and the depth image, the first Gaussian noise at the target moment is used for noise addition to obtain a noise image set. Then, based on the target prompt word, the diffusion estimation is performed in combination with the noise image set to obtain the second Gaussian noise at the previous moment of the target moment. The first Gaussian noise and the second Gaussian noise are combined for gradient backpropagation, and the hand depth map in the three-dimensional Gaussian field is used for loss update to obtain an updated target three-dimensional Gaussian field. The target three-dimensional Gaussian field contains a three-dimensional human body after three-dimensional reconstruction. Therefore, the embodiment of the present application first uses the prompt weight based on the original prompt word, which helps to match the generated content and text prompt, and then uses the depth loss to align the human body model with the depth of the local body part, and restore the fine human body parts, so that the generated 3D human body can accurately capture the complex features of the human body parts, and solve the problem that the existing three-dimensional human body construction technology cannot generate accurate body parts. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0047] Figure 1 A schematic flow chart of a 3D reconstruction method for reconstructing a 3D human body from a natural language description text provided in an embodiment of the present application;

[0048] Figure 2 It is a schematic diagram of the steps of a three-dimensional reconstruction method for reconstructing a three-dimensional human body from a natural language description text provided by an optional embodiment of the present application;

[0049] Figure 3 An example of this application provides a comparison diagram of a model generated by using text in the prior art and a model generated by using text in this embodiment;

[0050] Figure 4 It is a schematic diagram of a text-based three-dimensional human body generation algorithm provided by an optional example of the present application;

[0051] Figure 5A structural block diagram of a three-dimensional reconstruction device for reconstructing a three-dimensional human body from a natural language description text provided in an embodiment of the present application;

[0052] Figure 6 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0054] To facilitate the understanding of the embodiments of the present application, further explanation will be given below in conjunction with the drawings and specific embodiments. The embodiments do not constitute a limitation on the embodiments of the present application.

[0055] Figure 1 A flowchart of a method for reconstructing a three-dimensional human body from a natural language description text provided in an embodiment of the present application is provided. Figure 1 As shown, the 3D reconstruction method for reconstructing a 3D human body from a natural language description text provided in the embodiment of the present application may specifically include the following steps:

[0056] Step 110, determining the target prompt word after the weight is increased, and constructing an initial three-dimensional Gaussian field based on the initial human body template.

[0057] In this embodiment, the target prompt word is a prompt word obtained by adding weight and direction information to the original prompt word; the initial human body template refers to the initial human body template model, which usually includes an initialized human body posture, such as a human body posture similar to the letter A. The initial human body template can be initialized using the SMPL-X human body template model, and of course it can also be initialized and constructed using other human body template models, and this embodiment does not limit this.

[0058] In a specific implementation, this embodiment can initialize the human body template constructed by SMPL-X and construct a 3D Gaussian field, which can be used as a three-dimensional expression of a three-dimensional human body for subsequent processing.

[0059] Step 120: performing image rendering using the initial three-dimensional Gaussian field to obtain a human body image and a depth image.

[0060] Step 130 , adding noise using the first Gaussian noise at the target moment based on the human body image and the depth image to obtain a noise image set.

[0061] The noise image set includes a human body noisy image corresponding to the human body image and a depth noisy image corresponding to the depth image.

[0062] A unified description of steps 120 to 130 is given as follows:

[0063] In this embodiment, the human body image refers to an RGB (Red, Green, Blue, three primary colors) image of the human body. The first Gaussian noise may include two Gaussian noises, the first Gaussian noise is the Gaussian noise at the target moment, and the two Gaussian noises correspond to the human body image and the depth image respectively, and this embodiment does not limit this.

[0064] In a specific implementation, by using a 3D Gaussian field for image rendering, an RGB image and a depth image of a human body rendered in different postures can be obtained, and then both acquired images can be subjected to Gaussian noise denoising. To implement Gaussian noise denoising, this embodiment can first sample Gaussian noise to obtain Gaussian noise corresponding to the two images, and then use the Gaussian noise corresponding to the image to denoise the image at the time step corresponding to the target moment, thereby obtaining a human body denoised image corresponding to the RGB image and a depth denoised image corresponding to the depth image.

[0065] It should be noted that, in this embodiment, the target time can be understood as a time preset according to actual needs, or can be understood as the time corresponding to the sampling of Gaussian noise. The target time can be represented by t, and this embodiment does not impose any restrictions on this.

[0066] Step 140 , based on the target prompt word and in combination with the noise image set, diffusion estimation is performed to obtain a second Gaussian noise at a moment before the target moment.

[0067] In this embodiment, the second Gaussian noise may also include two Gaussian noises, and the second Gaussian noise is the Gaussian noise of the previous moment of the target moment. These two Gaussian noises may correspond one-to-one to the human body noisy image and the depth noisy image, respectively; the previous moment of the target moment may be understood as the moment of the previous time step based on the target moment. For example, if the target moment is t, the previous moment of the target moment may be t-1, and this embodiment does not impose any limitation on this.

[0068] In a specific implementation, this embodiment uses the target prompt word as a condition, uses two noisy images to perform diffusion estimation, and estimates the noise at time t-1 as the second Gaussian noise.

[0069] Step 150, performing gradient backpropagation based on the first Gaussian noise and the second Gaussian noise, and performing loss update using the hand depth map in the three-dimensional Gaussian field to obtain an updated target three-dimensional Gaussian field, wherein the target three-dimensional Gaussian field includes a three-dimensionally reconstructed three-dimensional human body.

[0070] In a specific implementation, this embodiment can update the 3D Gaussian field twice to generate fine human body parts. Specifically, the loss calculation is performed on the sampled first Gaussian noise and the second Gaussian noise respectively, and the gradient is returned, thereby performing the initial 3D Gaussian field update. Subsequently, the visibility mechanism is used to introduce correlation loss, such as Pearson correlation loss, to solve the problem of depth value mismatch, and the hand depth map of the 3D Gaussian field after the initial update is aligned. The 3D Gaussian field is further updated for a second time using the loss update to restore the focal depth loss of fine human body parts, thereby generating fine human body parts with precise topology, such as hands.

[0071] It should be noted that, in this embodiment, according to actual needs, the depth maps corresponding to the human body parts can be selected for alignment processing, so as to generate fine human body parts, and the alignment of the depth maps is not limited to the hand depth maps.

[0072] It can be seen that the embodiment of the present application performs image rendering through the initial three-dimensional Gaussian field constructed by the human body template to obtain a human body image and a depth image, and uses the Gaussian noise of the target moment to add noise to the image to obtain a noise image set, and the noise image set is combined with the weighted prompt words for diffusion estimation to obtain the Gaussian noise of the previous moment, and the Gaussian noise of the two moments is combined for gradient return, and the hand depth map in the three-dimensional Gaussian field is used for loss update to obtain the updated three-dimensional human body after three-dimensional reconstruction. This embodiment uses depth loss to align the human body model with the depth of local body parts, so that the generated 3D human body can accurately capture the complex features of human body parts, restore fine human body parts, and solve the problem that the existing three-dimensional human body construction technology cannot generate accurate body parts.

[0073] Reference Figure 2 , shows a schematic flow chart of the steps of a 3D reconstruction method for reconstructing a 3D human body from a natural language description text provided by an optional embodiment of the present application. The method may specifically include the following steps:

[0074] Step 210, determining the target prompt word after the weight is increased, and constructing an initial three-dimensional Gaussian field based on the initial human body template.

[0075] For the description of step 210, reference may be made to the above step 110, which will not be described in detail in this embodiment.

[0076] In an optional embodiment, the embodiment of the present application determines the target prompt word after the weight is increased, which can specifically include: obtaining preset basic prompt words and camera posture information; determining direction information based on the camera posture information, and obtaining detail prompt words and weight information of body parts; based on the basic prompt words, using the direction information, the detail prompt words and the weight information to increase the weight to obtain the target prompt word.

[0077] In the related art, existing 3D human body construction schemes usually utilize text-to-image (T2I) generation tools and combine 3D human prior knowledge to create a 3D human from a text prompt. Among them, CLIP-based methods tend to use human semantic information in the CLIP model to promote 3D human generation. Since the CLIP model mainly focuses on the alignment of high-level semantic information between text and image, it often struggles in accurately generating fine-grained details, especially in terms of human features and clothing specified in the text prompt. Considering the strong ability of diffusion models in text-to-image synthesis, most existing schemes seek to use powerful diffusion models to promote 3D human generation, which usually use score distillation sampling (SDS) loss to extract knowledge from diffusion models to optimize 3D models, such as 3D Gaussian scattering (3DGS) or neural radiance field (NeRF). However, these methods still have a variety of problems, including but not limited to: ① The existing technology mainly tends to generate 3D humans with local partial blur, that is, it tends to generate blurred or inconsistent human body parts such as hands and heads, mainly due to ignoring the focus on local geometry and precise perspective control. As Figure 3 As shown in Figure (a), the hand image generated by the diffusion model has six fingers. For this kind of error, existing methods tend to ignore the impact of these erroneous object information on 3D human generation, resulting in insufficient accurate local limbs in 3D human generation. Although the mesh-based method retains the hand structure of SMPL-X, it has difficulties in simulating the detailed topology of hair and accessories. It can be seen that the existing schemes cannot generate accurate body parts; ② Without carefully designed text prompts, it is easy to fail to control the T2I model to generate high-quality content that is aligned with the text. Reference Figure 3 ,from Figure 3 As can be seen in the figure (b2), although the text prompt specifies "two legs", the image generated by StableDiffusion2.0 depicts the entire body. Content misalignment causes local body parts to be blurred in the 3D human model. In addition, misalignment often causes the image perspective generated by the T2I model to not match the camera angle used for actual rendering. This multi-view information misalignment of body parts can lead to the Janus (multi-faceted) problem (see Figure 3As shown in (c2)). It can be seen that how to control the T2I model to output the information required for enhancing the rationality of the 3D human body is still an important issue, that is, the existing technology cannot reasonably control the model output to enhance the 3D human body.

[0078] To solve the above problems, this embodiment proposes a three-dimensional reconstruction method for reconstructing a three-dimensional human body from a natural language description text, mainly from three aspects to solve the problems existing in the prior art. First, this embodiment proposes a new focus paradigm for text-driven 3D human body generation. Using this focus paradigm, focus on the key moments and text prompts of the 3D human body to standardize the local geometry and viewing direction. Specifically, this embodiment presets a fine basic prompt word, and on the basis of the basic prompt word, direction information is added according to the camera posture, and then combined with detailed body part prompt words, and weight information is introduced, and the prompt word weight enhancement technology is used to obtain the target prompt word. The embodiment of the present application introduces a focus perspective-dependent prompt with fine-grained perspective control on the basis of the original prompt word to promote the 3D consistency of local body parts.

[0079] Furthermore, the present embodiment proposes a focal depth loss for recovering fine human body parts. Unlike existing methods that use coarse controls such as the human skeleton to guide the diffusion model, the present embodiment focuses on aligning the depth of the target 3D human body model (i.e., 3D Gaussian) and local body parts in SMPL-X, which contains prior information of fine human limbs. A visibility mechanism is designed and the Pearson correlation loss is introduced to solve the problem of depth value mismatch. The focal depth loss enables the present embodiment to generate fine human body parts, such as hands, with precise topology. Regarding the depth loss part, please refer to the subsequent steps 260-280, which will not be described in detail in this embodiment.

[0080] Furthermore, the present application proposes a focus perspective-dependent SDS for solving the 3D inconsistency problem of local body parts. To ensure that the generated content matches the text prompt, this embodiment designs a focus perspective-dependent prompt with fine-grained perspective control, utilizes the focus perspective-dependent prompt, emphasizes the keywords about the human body in the text prompt, and uses a prompt weighting mechanism. In addition, in order to achieve 3D consistent generation without introducing multiple faces, by adding perspective-related text about local body parts and the global body to the original text prompt, the subsequent SDS loss is subjected to more precise perspective control, thereby utilizing the focus perspective-dependent SDS to solve the 3D inconsistency problem of local body parts.

[0081] For example, the basic prompt word may be: "a boy wearing a hoodie", and the basic prompt word is recorded as On this basis, direction information is added according to the camera posture. For example, the direction information may include but is not limited to: front, side, and back, etc., to obtain a new prompt word. For example, the new prompt word may be "a boy in a hoodie, front / side / back", expressed as: Direction information. Then add detailed body part prompts and use the prompt weighting technology (using "() ++ ”), and the weight of specific body parts is increased through the cue word weighting technique. For example, when the model needs to pay more attention to the details of the hands, the cue word weighting technique uses “(hand) ++ ” means, as in “(the hand) of a boy wearing a hoodie ++ , front / side / rear", expressed as The target prompt word is obtained, and the target prompt word can be used express.

[0082] In an optional embodiment, the embodiment of the present application constructs an initial three-dimensional Gaussian field based on an initial human body template, which may specifically include: obtaining an initial human body template with an initial posture; and performing initial construction according to the initial human body template to obtain a three-dimensional Gaussian field.

[0083] In a specific implementation, this embodiment can use the SMPL-X human body template model to initialize a human body template model with a posture similar to the letter A. The template is used to initialize a 3D Gaussian field, which can be represented by G. The 3D Gaussian field can be used as the three-dimensional expression that needs to be trained in this embodiment:

[0084]

[0085] This embodiment can render multi-perspective human body images through 3D Gaussian field.

[0086] Step 220: performing image rendering using the initial three-dimensional Gaussian field to obtain a human body image and a depth image.

[0087] For the description of step 220, reference may be made to the above step 120, which will not be described in detail in this embodiment.

[0088] Optionally, the above-mentioned performing image rendering through the initial three-dimensional Gaussian field to obtain the human body image and the depth image may specifically include: using a preset renderer, according to the formula A human body image and a depth image of a human body are rendered from the camera posture information and the initial three-dimensional Gaussian field; wherein, I gs is the rendered human body image, d gs is the rendered depth image, pyrenderer is the renderer, is the initial three-dimensional Gaussian field, and camera_pose is the camera pose information.

[0090] In a specific implementation, this embodiment can preset a renderer, such as a pyrenderer renderer, through which the RGB image of the human body is rendered from different sampling camera postures. gs and the depth image d gs Considering that it is difficult for existing text graph models to synthesize delicate body parts such as hands, this embodiment provides more precise view control by adding view directions while generating local body parts. To ensure that different view directions can be rendered, this embodiment uses the formula:

[0091] Implement image rendering.

[0092] Step 230, obtaining a first Gaussian noise through noise sampling, and determining a target time.

[0093] Step 240: Noise the human body image and the depth image using the first Gaussian noise and the target moment through a preset sampler to obtain a noise image set at the target moment.

[0094] The noise image set includes a human body noisy image corresponding to the human body image and a depth noisy image corresponding to the depth image.

[0095] A unified description of steps 230 to 240 is given as follows:

[0096] In a specific implementation, this embodiment presets a sampler (DDIM sampler for short), and the sampler is used to add noise to the human image and the depth image. Specifically, at the target moment, the DDIM sampler is used to add noise to the human image and the depth image respectively in combination with the sampled first Gaussian noise, to obtain a human noisy image corresponding to the human image and a depth noisy image corresponding to the depth image.

[0097] In order to implement the noise addition processing of the human body image and the depth image, this embodiment sets a formula to implement the noise addition. Among them, the formula for image noise addition processing can be expressed as: In this embodiment, the RGB image is represented by I gs , the depth image is represented as d gs , the sampled Gaussian noise (i.e. the first Gaussian noise) contains and Add noise at time step (i.e. target moment) t, where For RGB image I gs Add noise to get the human body noisy image For the depth image dgs Add noise to get a deep noise image

[0098] Step 250 , based on the target prompt word and in combination with the noise image set, diffusion estimation is performed to obtain a second Gaussian noise at a moment before the target moment.

[0099] For the description of step 250, reference may be made to the above step 140, which will not be described in detail in this embodiment.

[0100] Optionally, the above-mentioned diffusion estimation based on the target prompt word and the noise image set to obtain the second Gaussian noise at the previous moment of the target moment may specifically include: using the formula Taking the target prompt word as a condition, the human body noisy image and the depth noisy image are respectively input into the UNet network for processing to obtain the second Gaussian noise of the previous moment of the target moment; wherein, is the second Gaussian noise corresponding to the human body noisy image, is the second Gaussian noise corresponding to the depth-noised image, Provide the target cue word.

[0101] In this embodiment, the UNet of StableDiffusion can be used to realize the noise estimation of the second Gaussian noise, with T lv As a condition, the human body noisy image and the depth noisy image are taken as input respectively, and the Gaussian noise at time t-1 is estimated using UNet of StableDiffusion.

[0102] Step 260: Calculate the mean square error based on the first Gaussian noise and the second Gaussian noise to obtain a depth loss function, and perform gradient backpropagation update based on the depth loss function to obtain an updated three-dimensional Gaussian field.

[0103] In a specific implementation, the update of the 3D Gaussian field may include two update processes, and the first update may be updated using gradient backpropagation. Specifically, in this embodiment, the first Gaussian noise and the second Gaussian noise may be subjected to MSE loss (depth loss) respectively, the mean square error may be calculated, and gradient backpropagation may be performed to update the 3D Gaussian field until the image rendered by the 3D Gaussian field is sufficiently close to the image distribution generated by stable diffusion.

[0104] In an optional implementation, the embodiment of the present application calculates the mean square error according to the first Gaussian noise and the second Gaussian noise to obtain a depth loss function, and performs gradient backpropagation update based on the depth loss function to obtain an updated three-dimensional Gaussian field, which may specifically include: according to Calculate the mean square error and get the depth loss function; according to Perform gradient backpropagation to update the three-dimensional Gaussian field; and are loss functions, η is the learning rate, and is continuously updated during the training update process of the 3D Gaussian field. The training process continues until the distribution of images rendered by the 3D Gaussian field is close enough to the distribution of images generated by Stable Diffusion.

[0105] Step 270: Align the hand depth map in the updated three-dimensional Gaussian field with the hand depth map in the initial human body template to obtain a correlation loss function.

[0106] In the specific implementation, the second update of the 3D Gaussian field can mainly involve the alignment of the depth map and the update of the correlation loss function. Specifically, when aligning the hand depth map, this example extracts the hand depth map from the 3D Gaussian field, aligns it with the hand depth map of SMPL-X, and calculates the Pearson correlation loss as the correlation loss function.

[0107] In the specific implementation, in order to achieve the alignment of the hand depth map, so that the distribution of the image rendered by the 3D Gaussian field is close enough to the image generated by stable diffusion, this embodiment uses the formula Implement depth map alignment.

[0108] In an optional embodiment, the embodiment of the present application uses the hand depth map in the updated three-dimensional Gaussian field and the hand depth map in the initial human body template to align to obtain a correlation loss function, which may specifically include: The two hand depth maps are aligned and the Pearson correlation loss is used; where d gs,hand To update the hand depth map in the 3D Gaussian field, d SMPL-X,hand It is the hand depth map in the initial human template.

[0109] Step 280: perform a second update on the updated three-dimensional Gaussian field using the correlation loss function to obtain a target three-dimensional Gaussian field.

[0110] In this embodiment, after the hand depth maps of the two models are aligned, the Pearson correlation loss obtained by the alignment can be used to further update the 3D Gaussian field.

[0111] In actual implementation, in order to further update the 3D Gaussian field, this embodiment uses the formula: The 3D Gaussian field is further updated based on the updated 3D Gaussian field to obtain a target three-dimensional Gaussian field.

[0112] Exemplary, combined Figure 3 Comparison diagrams: (b1) and (b2), and (c1) and (c2). It can be seen that this embodiment can generate human body parts more accurately than the existing T2I solution by adding weights and performing weight enhancement. Figure 4 As shown in the schematic diagram of the text-based 3D human body generation algorithm, it can be seen that extensive experiments and ablation studies have demonstrated that the 3D reconstruction algorithm provided in this application (or: a 3D clothed person reconstruction algorithm that generates fine and consistent limb details from text descriptions) exhibits superiority under various prompt words, and the generated 3D human body accurately captures the complex features of human body parts and is effective in text-driven 3D human body generation.

[0113] In summary, the embodiment of the present application determines the target prompt word after the weight is increased, and constructs an initial three-dimensional Gaussian field based on the initial human body template, performs image rendering through the initial three-dimensional Gaussian field to obtain a human body image and a depth image, and then uses a preset sampler to add noise to the human body image and the depth image at the target time using the sampled first Gaussian noise to obtain a noise image set, and performs diffusion estimation based on the target prompt word in combination with the noise image set to obtain a second Gaussian noise at the previous moment of the target moment, calculates the mean square error based on the first Gaussian noise and the second Gaussian noise to obtain a depth loss function, and performs gradient backpropagation update based on the depth loss function to obtain an updated three-dimensional Gaussian field, thereby realizing the first update of the three-dimensional Gaussian field, and then uses the updated three-dimensional Gaussian field to align with the hand depth map in the human body template model to obtain a correlation loss function, and uses the correlation loss function to perform a second update on the updated three-dimensional Gaussian field to obtain a target three-dimensional Gaussian field. Therefore, in order to exert more precise control when optimizing the 3D model, the embodiment of the present application proposes Focal Gaussian, which focuses on the key parts of the human body when optimizing the 3DGS-based human body model using a fixed diffusion model, that is, focusing on the key parts of the human body. Specifically, in order to generate fine human body parts, this embodiment uses the geometric prior of SMPL-X to align the depth of the SMPL-X hand with the corresponding 3DGS (three-dimensional Gaussian field) depth through focal depth loss. Taking into account the mismatch in the values ​​of the two depth maps, this embodiment introduces the Pearson correlation loss for standardization, and uses the focal depth loss to restore fine human body parts by aligning the 3D human body model with the local body part depth of SMPL-X, thereby solving the problem that existing methods are difficult to generate fine body parts. In addition, in order to solve the Janus problem, this embodiment proposes a focal perspective-dependent SDS loss, focusing on key components, and using fine-grained perspective control on local body parts. Specifically, in order to achieve 3D consistent generation without introducing multiple faces, this embodiment performs more precise perspective control on the SDS loss by adding perspective-related text about local body parts and the global body in the original text prompt, and in order to ensure that the generated content matches the text prompt, this embodiment designs a focal perspective-dependent prompt that emphasizes keywords about the human body in the text prompt, and uses a prompt weighting mechanism to utilize focal perspective-related prompt words to emphasize and exert perspective control on keywords of local body parts, so as to solve the problem that the prior art is limited to "Janus".

[0114] It should be noted that, for the purpose of simple description, the method embodiments are expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited to the described order of actions, because according to the embodiments of the present application, certain steps may be performed in other orders or simultaneously.

[0115] like Figure 5 As shown, the embodiment of the present application further provides a 3D reconstruction device 500 for reconstructing a 3D human body from a natural language description text, comprising:

[0116] The prompt word acquisition and model construction module 510 is used to determine the target prompt word after the weight is increased, and to construct an initial three-dimensional Gaussian field based on the initial human body template;

[0117] An image rendering module 520 is used to perform image rendering using the initial three-dimensional Gaussian field to obtain a human body image and a depth image;

[0118] An image denoising module 530 is configured to perform denoising on the human body image and the depth image using a first Gaussian noise at a target time to obtain a noisy image set, wherein the noisy image set includes a human body noisy image corresponding to the human body image and a depth noisy image corresponding to the depth image;

[0119] A diffusion estimation module 540 is used to perform diffusion estimation based on the target prompt word in combination with the noise image set to obtain a second Gaussian noise at a previous moment of the target moment;

[0120] The three-dimensional Gaussian field update module 550 is used to perform gradient feedback according to the first Gaussian noise and the second Gaussian noise, and use the hand depth map in the three-dimensional Gaussian field to perform loss update to obtain an updated target three-dimensional Gaussian field, wherein the target three-dimensional Gaussian field includes a three-dimensional human body after three-dimensional reconstruction.

[0121] Optionally, the prompt word acquisition and model building module 510 includes:

[0122] The prompt word and posture acquisition submodule is used to obtain the preset basic prompt words and camera posture information;

[0123] A direction information acquisition submodule, used to determine direction information according to the camera posture information, and to acquire detail prompt words and weight information of body parts;

[0124] A weight enhancement submodule, configured to enhance the weight of the basic prompt word by using the direction information, the detail prompt word and the weight information to obtain a target prompt word;

[0125] The human body template acquisition submodule is used to obtain the initial human body template of the initialization posture;

[0126] The initialization construction submodule is used to perform initialization construction according to the initial human body template to obtain a three-dimensional Gaussian field.

[0127] Optionally, the image rendering module 520 is specifically used to: use a preset renderer according to the formula A human body image and a depth image of a human body are rendered from the camera posture information and the initial three-dimensional Gaussian field; wherein, I gs is the rendered human body image, d gs is the rendered depth image, pyrenderer is the renderer, is the initial three-dimensional Gaussian field, and camera_pose is the camera pose information.

[0128] Optionally, the image noise adding module 530 includes:

[0129] A sampling submodule, used for acquiring the first Gaussian noise through noise sampling and determining the target time;

[0130] The submodule is used to add noise to the human body image and the depth image by using the first Gaussian noise and the target moment through a preset sampler to obtain a noise image set at the target moment.

[0131] Optionally, the diffusion estimation module 540 is specifically used to: use the formula Taking the target prompt word as a condition, the human body noisy image and the depth noisy image are respectively input into the UNet network for processing to obtain the second Gaussian noise of the previous moment of the target moment; wherein, is the second Gaussian noise corresponding to the human body noisy image, is the second Gaussian noise corresponding to the depth-noised image, Provide the target cue word.

[0132] Optionally, the three-dimensional Gaussian field updating module 550 includes:

[0133] A first updating submodule, configured to calculate a mean square error according to the first Gaussian noise and the second Gaussian noise to obtain a depth loss function, and to perform a gradient backpropagation update based on the depth loss function to obtain an updated three-dimensional Gaussian field;

[0134] An alignment submodule, configured to align the hand depth map in the updated three-dimensional Gaussian field with the hand depth map in the initial human body template to obtain a correlation loss function;

[0135] The second updating submodule is used to perform a second update on the updated three-dimensional Gaussian field using the correlation loss function to obtain a target three-dimensional Gaussian field.

[0136] Optionally, the first updating submodule is specifically used to: Calculate the mean square error and get the depth loss function; according to Perform gradient backpropagation to update the three-dimensional Gaussian field; and are loss functions, η is the learning rate, and it is continuously updated during the training update process of the three-dimensional Gaussian field.

[0137] Optionally, the alignment submodule is specifically used to: The two hand depth maps are aligned and the Pearson correlation loss is used; where d gs,hand To update the hand depth map in the 3D Gaussian field, d SMPL-X,hand It is the hand depth map in the initial human template.

[0138] It should be noted that the three-dimensional reconstruction device for reconstructing a three-dimensional human body from a natural language description text provided in an embodiment of the present application can execute the three-dimensional reconstruction method for reconstructing a three-dimensional human body from a natural language description text provided in any embodiment of the present application, and has the corresponding functions and beneficial effects of the execution method.

[0139] In a specific implementation, the above-mentioned 3D reconstruction device for reconstructing a 3D human body from a natural language description text can be integrated into a device, so that the device can use the weighted prompt words to apply viewing angle control to the local body parts of the 3D human body, accurately capture the complex features of the human body parts, and as an electronic device, realize the generation of accurate body parts. The electronic device can be composed of two or more physical entities, or it can be composed of one physical entity, such as a personal computer (PC), a computer, a server, etc., and the embodiments of the present application do not impose specific restrictions on this.

[0140] like Figure 6As shown, an embodiment of the present application provides an electronic device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114; the memory 113 is used to store computer programs; the processor 111 is used to implement the steps of the three-dimensional reconstruction method for reconstructing a three-dimensional human body from a natural language description text provided by any of the aforementioned method embodiments when executing the program stored in the memory 113. Exemplarily, the steps of a three-dimensional reconstruction method for reconstructing a three-dimensional human body from a natural language description text may include the following steps: determining a target prompt word after weighting is increased, and constructing an initial three-dimensional Gaussian field based on an initial human body template; performing image rendering through the initial three-dimensional Gaussian field to obtain a human body image and a depth image; based on the human body image and the depth image, using the first Gaussian noise at the target moment to perform noise addition to obtain a noise image set, the noise image set including a human body noisy image corresponding to the human body image and a depth noisy image corresponding to the depth image; based on the target prompt word, combining the noise image set to perform diffusion estimation to obtain a second Gaussian noise at a moment before the target moment; performing gradient backpropagation based on the first Gaussian noise and the second Gaussian noise, and using the hand depth map in the three-dimensional Gaussian field to perform loss update to obtain an updated target three-dimensional Gaussian field, the target three-dimensional Gaussian field including the three-dimensionally reconstructed three-dimensional human body.

[0141] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of a three-dimensional reconstruction method for reconstructing a three-dimensional human body from a natural language description text as provided in any of the aforementioned method embodiments are implemented.

[0142] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0143] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A 3D reconstruction method for reconstructing a 3D human body from a natural language description text, characterized in that: include: Determine the target prompt word after the weight is increased, and construct an initial three-dimensional Gaussian field based on the initial human body template; Performing image rendering through the initial three-dimensional Gaussian field to obtain a human body image and a depth image; According to the human body image and the depth image, a first Gaussian noise at a target moment is used to perform noise addition to obtain a noise image set, wherein the noise image set includes a human body noisy image corresponding to the human body image and a depth noisy image corresponding to the depth image; Based on the target prompt word, the diffusion estimation is performed in combination with the noise image set to obtain the second Gaussian noise of the previous moment of the target moment; Gradient feedback is performed based on the first Gaussian noise and the second Gaussian noise, and the loss is updated using the hand depth map in the three-dimensional Gaussian field to obtain an updated target three-dimensional Gaussian field, where the target three-dimensional Gaussian field includes a three-dimensionally reconstructed three-dimensional human body.

2. The method according to claim 1, characterized in that The step of determining the target prompt word after the weight is increased includes: Get the preset basic prompt words and camera posture information; Determine direction information according to the camera posture information, and obtain detail prompt words and weight information of body parts; On the basis of the basic prompt word, the direction information, the detail prompt word and the weight information are used to perform weight enhancement to obtain the target prompt word.

3. The method according to claim 2, characterized in that Construct an initial three-dimensional Gaussian field based on the initial human body template, including: Get the initial human body template of the initialization posture; Initial construction is performed according to the initial human body template to obtain a three-dimensional Gaussian field.

4. The method according to claim 3, characterized in that The method of adding noise using the first Gaussian noise at a target time based on the human body image and the depth image to obtain a noise image set includes: Obtain the first Gaussian noise through noise sampling and determine the target time; The human body image and the depth image are denoised by using the first Gaussian noise and the target moment through a preset sampler to obtain a noise image set at the target moment.

5. The method according to claim 4, characterized in that Gradients are returned according to the first Gaussian noise and the second Gaussian noise, and the loss is updated using the hand depth map in the three-dimensional Gaussian field to obtain an updated target three-dimensional Gaussian field, including: Performing mean square error calculation based on the first Gaussian noise and the second Gaussian noise to obtain a depth loss function, and performing gradient backpropagation update based on the depth loss function to obtain an updated three-dimensional Gaussian field; Aligning the hand depth map in the updated three-dimensional Gaussian field with the hand depth map in the initial human body template to obtain a correlation loss function; The updated three-dimensional Gaussian field is updated twice using the correlation loss function to obtain a target three-dimensional Gaussian field.

6. The method according to claim 5, characterized in that The method of performing image rendering through the initial three-dimensional Gaussian field to obtain a human body image and a depth image includes: Using the preset renderer, according to the formula Rendering a human body image and a depth image of a human body from the camera pose information and the initial three-dimensional Gaussian field; Among them, I gs is the rendered human body image, d gs is the rendered depth image, pyrenderer is the renderer, is the initial three-dimensional Gaussian field, and camera_pose is the camera pose information.

7. The method according to claim 6, characterized in that The step of performing diffusion estimation based on the target prompt word and in combination with the noise image set to obtain the second Gaussian noise at a previous moment of the target moment includes: Using the formula Taking the target prompt word as a condition, the human body noisy image and the depth noisy image are respectively input into the UNet network for processing to obtain the second Gaussian noise of the previous moment of the target moment; in, is the second Gaussian noise corresponding to the human body noisy image, is the second Gaussian noise corresponding to the depth-noised image, Provide the target cue word.

8. The method according to claim 7, characterized in that A mean square error is calculated according to the first Gaussian noise and the second Gaussian noise to obtain a depth loss function, and a gradient backpropagation update is performed based on the depth loss function to obtain an updated three-dimensional Gaussian field, including: according to Calculate the mean square error and get the depth loss function; according to Perform gradient backpropagation to update the three-dimensional Gaussian field; in, and are loss functions, η is the learning rate, and it is continuously updated during the training update process of the three-dimensional Gaussian field.

9. The method according to claim 7, characterized in that: The method of aligning the hand depth map in the updated three-dimensional Gaussian field with the hand depth map in the initial human body template to obtain a correlation loss function includes: according to Align the two hand depth maps and use the Pearson correlation loss; Among them, d gs,hand To update the hand depth map in the 3D Gaussian field, d SMPL-X,hand It is the hand depth map in the initial human template.

10. A 3D reconstruction device for reconstructing a 3D human body from a natural language description text, characterized in that: include: The prompt word acquisition and model construction module is used to determine the target prompt word after the weight is increased, and to construct an initial three-dimensional Gaussian field based on the initial human body template; An image rendering module, used for performing image rendering through the initial three-dimensional Gaussian field to obtain a human body image and a depth image; An image denoising module, configured to perform denoising using a first Gaussian noise at a target time according to the human body image and the depth image, to obtain a noisy image set, wherein the noisy image set includes a human body denoised image corresponding to the human body image and a depth denoised image corresponding to the depth image; A diffusion estimation module, used for performing diffusion estimation based on the target prompt word and the noise image set to obtain a second Gaussian noise at a previous moment of the target moment; The three-dimensional Gaussian field update module is used to perform gradient backpropagation according to the first Gaussian noise and the second Gaussian noise, and use the hand depth map in the three-dimensional Gaussian field to perform loss update to obtain an updated target three-dimensional Gaussian field, wherein the target three-dimensional Gaussian field includes a three-dimensional human body after three-dimensional reconstruction.

Images