3D Digital Human Generation Method and System

Abstract

The present application discloses a three-dimensional digital human body generation method and system, belonging to the field of computer vision technology. The method includes: obtaining a human body front image dataset and performing preprocessing to obtain a target dataset; constructing a back generation normal fusion network model, and training and testing the back generation normal fusion network model according to the target dataset; inputting the front image of the three-dimensional digital human body to be generated into the tested back generation normal fusion network model, and the back generation normal fusion network model outputs the three-dimensional digital human body corresponding to the front image of the three-dimensional digital human body to be generated. The back image of the human body is generated by combining the latent diffusion model with the joint point information, and the missing back geometry and texture information of the single view are converted into data consistent with the dual perspectives, improving the accuracy of the back image prediction.

Description

Technical Field

[0001] This application belongs to the technical field of computer vision, and specifically relates to a three-dimensional digital human body generation method and system. Background Art

[0002] With the development of technologies such as virtual reality and digital twin, single-view Figure 3 dimensional digital human body generation technology has become a research hotspot. Existing methods mainly rely on parametric mesh representation (such as the SMPL model) or implicit functions to reconstruct three-dimensional human body models from a single frontal image.

[0003] However, such methods have the following key problems: Difficult to reconstruct back information: A single view can only provide frontal visual data, and the back geometric structure (such as the back and hind legs) needs to be inferred, which is likely to lead to geometric distortion or texture inconsistency. Uneven transition between the human body and clothing: Directly inputting the nude normal map and the clothed normal map into the implicit function independently may result in discontinuities or abruptness in the area where the human body surface meets the clothing. In the prior art, methods based on generative adversarial networks (GANs) or image translation (such as Pix2PixHD, Zero-1-to-3) attempt to predict the back view, but there are problems such as overfitting, inconsistent poses, or appearance mismatches. In addition, traditional normal map fusion methods do not consider the complex relationship between the human body and clothing, resulting in insufficient robustness of the reconstruction results. Summary of the Invention

[0004] The purpose of the embodiments of this application is to provide a three-dimensional digital human body generation method and system, which can solve the technical problem of difficult back information reconstruction in the process of three-dimensional digital human body generation in the prior art.

[0005] To solve the above technical problems, this application is implemented as follows:

[0006] In a first aspect, the embodiments of this application provide a three-dimensional digital human body generation method, and the method includes:

[0007] Obtain a dataset of human frontal images, preprocess the dataset of human frontal images to obtain a target dataset;

[0008] Construct a back-generated normal fusion network model, where the back-generated normal fusion network model includes a back generation module, a frontal normal reconstruction and fusion module, a back normal prediction and fusion module, and a human body reconstruction implicit function module;

[0009] Train the back-generated normal fusion network model according to part of the data in the target dataset, and test the trained back-generated normal fusion network model with the other part of the data;

[0010] Input the front image of the 3D digital human to be generated into the tested back-generated normal fusion network model, and the back-generated normal fusion network model outputs the 3D digital human corresponding to the front image of the 3D digital human to be generated.

[0011] As an alternative implementation of the first aspect of the present application, the back-generated normal fusion network model is trained according to part of the data in the target dataset, and the trained back-generated normal fusion network model is tested with the other part of the data; specifically:

[0012] Divide the target dataset into a target training set and a target test set according to a preset ratio;

[0013] Construct a loss function, and train the back-generated normal fusion network model according to the target dataset and the loss function;

[0014] Test the back-generated normal fusion network model according to the target test set to complete the training and testing of the back-generated normal fusion network model.

[0015] As an alternative implementation of the first aspect of the present application, input the front image of the 3D digital human to be generated into the tested back-generated normal fusion network model, and the back-generated normal fusion network model outputs the 3D digital human corresponding to the front image of the 3D digital human to be generated; specifically:

[0016] After the front image of the 3D digital human to be generated is input into the back-generated normal fusion network model, the back generation module processes the front image to generate a back image corresponding to the front image;

[0017] Input the front image into the front normal reconstruction fusion module, and the front normal reconstruction fusion module processes the front image to obtain a front fusion normal map;

[0018] Input the front image and the back image into the back normal prediction fusion module, and the back normal prediction fusion module processes the front image and the back image to obtain a back fusion normal map;

[0019] Input the front image, the back image, the front normal fusion map, and the back fusion normal map into the human body reconstruction implicit function module, and the human body reconstruction implicit function module processes the front image, the back image, the front normal fusion map, and the back fusion normal map to obtain the 3D digital human.

[0020] As an alternative implementation of the first aspect of the present application, the back generation module includes a CLIP image encoder, a VAE encoder, an OPENPOSE network layer, a ControlNet network layer, and a latent diffusion network layer. The back generation module processes the front image to generate a back image corresponding to the front image. Specifically:

[0021] Extract features from the front image according to the CLIP image encoder and the VAE encoder respectively to obtain a CLIP feature encoding and a VAE feature encoding.

[0022] Extract key points from the front image according to the OPENPOSE network layer to obtain a key point map.

[0023] Extract features from the key point map according to the ControlNet network layer to obtain key point features.

[0024] Process the CLIP feature encoding, the VAE feature encoding, and the key point features according to the latent diffusion network layer to obtain the back image.

[0025] As an alternative implementation of the first aspect of the present application, the front normal reconstruction and fusion module includes a PIXIE network layer, a first nude fusion layer, an A-CE2P network layer, and a first dressed fusion layer. The front normal reconstruction and fusion module processes the front image to obtain a front fusion normal map. Specifically:

[0026] Process the front image according to the PIXIE network layer to obtain an initial digital human model, and perform front rendering on the initial digital human model to obtain a front nude normal map.

[0027] Fuse the front image and the front nude normal map according to the first nude fusion layer to obtain a front dressed normal map.

[0028] Parse and segment the front image according to the A-CE2P network layer to obtain a front mask, and perform Gaussian blur processing on the front mask to obtain a front confidence weight.

[0029] The first dressed fusion layer fuses the front dressed normal map and the front nude normal map according to the front confidence weight to obtain the front fusion normal map.

[0030] As an alternative implementation of the first aspect of the present application, the back normal prediction fusion module includes a PIXIE network layer, a second nude fusion layer, an A-CE2P network layer, and a second dressed fusion layer. The back normal prediction fusion module processes the front image and the back image to obtain a back fusion normal map. Specifically:

[0031] Process the front image according to the PIXIE network layer to obtain an initial digital human model, and perform back rendering on the initial digital human model to obtain a back nude normal map;

[0032] Fuse the back image and the front nude normal map according to the second nude fusion layer to obtain a back dressed normal map;

[0033] Parse and segment the back image according to the A-CE2P network layer to obtain a back mask, and perform Gaussian blur processing on the back mask to obtain a back confidence weight;

[0034] The second dressed fusion layer fuses the back dressed normal map and the back nude normal map according to the back confidence weight to obtain the back fusion normal map.

[0035] As an alternative implementation of the first aspect of the present application, the human body reconstruction implicit function module processes the front image, the back image, the front normal fusion map, and the back fusion normal map to obtain the three-dimensional digital human body. Specifically:

[0036] Perform query point sampling on the initial digital human model to obtain a plurality of front query points and a plurality of back query points;

[0037] Calculate the signed distance from each front query point to each front surface of the initial digital human model, and the signed distance from each back query point to each back surface of the initial digital human model;

[0038] According to the front image, the front normal fusion map, and each front query point, obtain each front image feature, each front normal vector feature, and each front segmentation feature corresponding to each front query point;

[0039] According to the back image, the back normal fusion map, and each back query point, obtain each back image feature, each back normal vector feature, and each back segmentation feature corresponding to each front query point;

[0040] Construct each front input feature corresponding to each front query point according to each signed distance, each front image feature, each front normal vector feature, and each front segmentation feature corresponding to each front query point;

[0041] Construct an input feature for each back query point based on each signed distance on the back, each back image feature, each back normal vector feature, and each back segmentation feature corresponding to each back query point.

[0042] Input each front input feature and each back input feature into the implicit function to obtain each front output feature corresponding to each front input feature and each back output feature corresponding to each back input feature.

[0043] The marching cubes algorithm generates the three-dimensional digital human body based on each front output feature and each back output feature.

[0044] In a second aspect, an embodiment of the present application provides a three-dimensional digital human body generation system, and the system includes:

[0045] An acquisition module: acquire a dataset of human front images, and preprocess the dataset of human front images to obtain a target dataset.

[0046] A construction module: construct a back-generated normal fusion network model, and the normal fusion network model includes a back generation module, a front normal reconstruction and fusion module, a back normal prediction and fusion module, and a human body reconstruction implicit function module.

[0047] A training and testing module: train the back-generated normal fusion network model according to a part of the data in the target dataset, and test the trained back-generated normal fusion network model with another part of the data.

[0048] A prediction module: input the front image of the three-dimensional digital human body to be generated into the tested back-generated normal fusion network model, and the back-generated normal fusion network model outputs the three-dimensional digital human body corresponding to the front image of the three-dimensional digital human body to be generated.

[0049] In a third aspect, an embodiment of the present application provides an electronic device, and the electronic device includes a processor, a memory, and a program or instruction stored on the memory and executable on the processor. When the program or instruction is executed by the processor, the steps of the method described in the first aspect are implemented.

[0050] In a fourth aspect, an embodiment of the present application provides a readable storage medium, and a program or instruction is stored on the readable storage medium. When the program or instruction is executed by the processor, the steps of the method described in the first aspect are implemented.

[0051] In the embodiments of the present application, compared with the prior art, the following technical effects are achieved:

[0052] (1) Generate a human body back image by combining joint point information through a potential diffusion network layer, convert the missing back geometry and texture information in a single view into dual-view consistent data, and improve the accuracy of back image prediction;

[0053] (2) Design a first nude fusion layer and a second nude fusion layer to reconstruct the front and back normal maps; through the human body parsing segmentation and normal fusion mechanism, enhance the model's associated modeling of the human body surface and clothing folds, and significantly improve the smoothness and geometric consistency of the dressed transition area;

[0054] (3) Through the cooperation among the back generation module, the front normal reconstruction and fusion module, the back normal prediction and fusion module, and the human body reconstruction implicit function module, improve the generalization ability of the method for complex poses and diverse clothing, and further verify that the method in this application is superior to the existing technologies in terms of accuracy and visual effects. Description of the Drawings

[0055] Figure 1 is a flowchart of a three-dimensional digital human body generation method provided by some embodiments of this application;

[0056] Figure 2 is a structural diagram of a back generation and normal fusion network model of a three-dimensional digital human body generation method provided by some embodiments of this application;

[0057] Figure 3 is a structural diagram of a back generation module of a three-dimensional digital human body generation method provided by some embodiments of this application; Detailed Embodiments

[0058] Next, the technical solutions in the embodiments of this application will be clearly and completely described in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are some, but not all, of the embodiments of this application. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this application.

[0059] The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such data may be interchanged under appropriate circumstances so that the embodiments of this application can be implemented in an order other than those illustrated or described herein. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / " generally represents an "or" relationship between the associated objects before and after.

[0060] Next, a three-dimensional digital human body generation method and system provided by embodiments of this application will be described in detail in conjunction with the drawings, through specific embodiments and their application scenarios.

[0061] Embodiment

[0062] A three-dimensional digital human body generation method includes the following steps:

[0063] S100: Obtain a dataset of human front images, preprocess the dataset of human front images to obtain a target dataset;

[0064] It should be noted that multiple human bodies are three-dimensionally scanned with a scanning span of 10°. For each human body, 36 samples are obtained. The 36 samples corresponding to each human body are rendered to obtain the front image, back image, and three-dimensional solid model corresponding to each human body sample. Among them, the front images are used to form the dataset, and the back images and three-dimensional solid models are used to form the true labels of the dataset.

[0065] S200: Construct a back generation normal fusion network model, which includes a back generation module, a front normal reconstruction fusion module, a back normal prediction fusion module, and a human body reconstruction implicit function module;

[0066] It should be noted that the back generation module includes a CLIP (Contrastive Language-Image Pretraining) image encoder, a VAE (Variational Autoencoder) encoder, an OPENPOSE (whole-body joint point detection network) network layer, a ControlNet (conditional control) network layer, and a latent diffusion network layer. The latent diffusion network layer includes multiple alternately arranged residual blocks and cross-attention blocks; the front normal reconstruction fusion module includes a PIXIE (3D human body pose and shape estimation) network layer, a first nude fusion layer, an A-CE2P (attention-enhanced human body parsing) network layer, and a first dressed fusion layer; the back normal prediction fusion module includes a PIXIE network layer, a second nude fusion layer, an A-CE2P network layer, and a second dressed fusion layer;

[0067] Among them, the residual blocks, CLIP image encoder, VAE encoder, and OPENPOSE network layer are all trained. That is to say, when constructing the back generation normal fusion network model, the residual blocks, CLIP image encoder, VAE encoder, and OPENPOSE network layer are all trained. When training the back generation normal fusion network model, the parameters of the above parts will not change.

[0068] S300: Train the back generation normal fusion network model according to part of the data in the target dataset, and test the trained back generation normal fusion network model with the other part of the data;

[0069] It should be noted that S300 specifically includes:

[0070] S310: Divide the target data set into a target training set and a target test set according to a preset ratio;

[0071] S320: Construct a loss function, and train the back-generated normal fusion network model according to the target data set and the loss function;

[0072] S330: Test the back-generated normal fusion network model according to the target test set to complete the training and testing of the back-generated normal fusion network model.

[0073] Furthermore, in this embodiment, the preset ratio is 8:2, and the target data set is divided into a target training set and a target test set.

[0074] S400: Input the front image of the three-dimensional digital human to be generated into the tested back-generated normal fusion network model, and the back-generated normal fusion network model outputs the three-dimensional digital human corresponding to the front image of the three-dimensional digital human to be generated;

[0075] It should be noted that S400 specifically includes:

[0076] S410: After the front image of the three-dimensional digital human to be generated is input into the back-generated normal fusion network model, the back generation module processes the front image and generates a back image corresponding to the front image;

[0077] S420: Input the front image into the front normal reconstruction fusion module, and the front normal reconstruction fusion module processes the front image to obtain a front fusion normal map;

[0078] S430: Input the front image and the back image into the back normal prediction fusion module, and the back normal prediction fusion module processes the front image and the back image to obtain a back fusion normal map;

[0079] S440: Input the front image, the back image, the front normal fusion map, and the back fusion normal map into the human body reconstruction implicit function module, and the human body reconstruction implicit function module processes the front image, the back image, the front normal fusion map, and the back fusion normal map to obtain a three-dimensional digital human.

[0080] It should be noted that in S410, the back generation module processes the front image and generates a back image corresponding to the front image; specifically:

[0081] S411: Respectively extract features from the front image according to the CLIP image encoder and the VAE encoder to obtain a CLIP feature encoding and a VAE feature encoding;

[0082] S412: Extract key points from the frontal image according to the OPENPOSE network layer to obtain a key point map;

[0083] S413: Extract features from the key point map according to the ControlNet network layer to obtain key point features;

[0084] S414: Process the CLIP feature encoding, VAE feature encoding, and key point features according to the latent diffusion network layer to obtain the back image.

[0085] Specifically, the data processing process of the latent diffusion network layer in S414 is represented by the following formula:

[0086] ,

[0087] Among them, represents the frontal image, represents the key points, represents the initial Gaussian noise, represents the denoising process of the latent diffusion network layer, represents the latent code of the back image, represents the decoding process, represents the back image.

[0088] Furthermore, the back generation module converts the image into a low-dimensional representation through a variational autoencoder (VAE encoder) and generates a new image through an iterative denoising process. To enable the latent diffusion network layer (LDM) to generate a consistent and realistic back image based on the input image; first, the back generation module uses a pre-trained CLIP image encoder and VAE encoder to extract CLIP feature encoding and VAE feature encoding from the frontal image, and these feature encodings are used to condition the LDM to ensure that the output image is consistent with the input image in appearance; the human joint points are also introduced as additional conditional inputs to provide more accurate pose information; specifically, the back generation module extracts the key point map of its back with the help of OPENPOSE; subsequently, this key point map is integrated into the generation process of the latent diffusion network layer (LDM) through the ControlNet network layer. With its conditional control ability, the ControlNet network uses the input key point map as the key guiding information to impose precise constraints on key elements such as pose and shape during the image generation process. This enables the effective generation of the back image based on the human structure and pose information contained in the frontal image, ensuring a high degree of alignment between the generated back image and the frontal image in terms of pose coherence and shape matching, thereby improving the overall quality and accuracy of image generation.

[0089] It should be noted that in S420, the front normal reconstruction and fusion module processes the front image to obtain a front fusion normal map. Specifically:

[0090] S421: Process the front image according to the PIXIE network layer to obtain an initial digital human model, and perform front rendering on the initial digital human model to obtain a front nude normal map;

[0091] S422: Fusion the front image and the front nude normal map according to the first nude and dressed fusion layer to obtain a front dressed normal map;

[0092] S423: Parse and segment the front image according to the A-CE2P network layer to obtain a front mask, and perform Gaussian blur processing on the front mask to obtain a front confidence weight;

[0093] S424: The first dressed fusion layer fuses the front dressed normal map and the front nude normal map according to the front confidence weight to obtain a front fusion normal map.

[0094] Specifically, the data processing of the first nude and dressed fusion layer in S422 is represented by the following formula:

[0095] ,

[0096] where, represents the front nude normal map, represents the front image, represents the fusion process of the first nude and dressed fusion layer, represents the front dressed normal map;

[0097] In addition, during the training process of the first nude and dressed fusion layer, reconstruction loss and Laplacian loss are used to optimize the fusion process of the first nude and dressed fusion layer. Specifically:

[0098] ,

[0099] where, represents the total loss function, represents the reconstruction loss function, represents the Laplacian loss function, represents the weight coefficient.

[0100] Specifically, the front mask in S423 includes a front dressed mask and a front nude mask, the front confidence weight includes a front dressed confidence weight and a front nude confidence weight, and the front confidence generation formula is as follows:

[0101] ,

[0102] Among them, represents the front nude confidence weight, represents Gaussian blur processing, represents the front nude mask, represents the front dressed confidence weight, represents the front dressed mask;

[0103] Specifically, the process of the first dressed fusion layer processing data in S424 is represented by the following formula:

[0104] ,

[0105] Among them, represents the front fusion normal map, represents the front nude confidence weight, represents the front dressed confidence weight, represents the front nude normal map, represents the front dressed normal map.

[0106] Further, first, the frontal image is input into the PIXIE network layer to estimate an initial digital human model (SMPLX model); the PIXIE network is from the paper "Collaborative Regression of Expressive Bodies using Moderation". The PIXIE network is selected because it performs better than other methods in terms of pixel alignment from the mesh to the image. Then, based on the estimated initial digital human model, a frontal nude normal map is rendered from the frontal view using the differentiable renderer of PyTorch3D; the 3D features of the initial digital human model or the frontal nude normal map rendered from it belong to global features. To make the network model more robust, it is extremely important to extract local features. To more fully extract the features from the frontal image, the first nude-clothed fusion layer (deep learning network) is used to fuse the frontal image and the frontal nude normal map to obtain a frontal clothed normal map; the first nude-clothed fusion layer can learn from the frontal data features, focus on the frontal image and frontal-related information, and can better capture the feature patterns of the frontal clothed human body, thus more accurately predicting the frontal normal; to optimize the normal map generated by the model so that it not only has high accuracy at the pixel level but also has a perceptual quality close to real images, a reconstruction loss function and a Laplacian loss function are used for optimization on the first nude-clothed fusion layer. The reconstruction loss function directly measures the difference between the generated normal map and the target normal map at the pixel level to ensure that the generated normal map has accurate geometric details; a reconstruction loss based on cosine similarity is used to better adapt to the unique feature expression of the normal map. In addition, a weight parameter is introduced to control the importance of the Laplacian loss function in the total loss. The Laplacian loss function helps optimize the high-frequency information and detail recovery of the image by calculating the perceptual difference between the generated normal map and the real normal map, thereby reducing ghosting and blurring in the normal map. However, directly using the frontal nude normal map and the frontal clothed normal map as independent inputs may lead to problems of incoordination between the human body and clothing in the reconstruction model because the human body and clothing are closely related. The shape of the clothing is affected by the human body shape and will also occlude or change the appearance of the human body; if the implicit function cannot well coordinate the relationship between these two inputs, it may lead to problems such as the human body model not fitting well with the clothing or having an uncoordinated appearance in the reconstructed human body model; therefore, the frontal normal reconstruction fusion module parses and segments the frontal image according to the A-CE2P network layer to obtain a frontal clothed mask and a frontal nude mask, and performs Gaussian blur processing on the frontal clothed mask and the frontal nude mask to obtain a frontal clothed confidence weight and a frontal nude confidence weight; the first clothed fusion layer fuses the frontal clothed normal map and the frontal nude normal map according to the frontal clothed confidence weight and the frontal nude confidence weight to obtain a frontal fused normal map.

[0107] It should be noted that in S430, the back normal prediction fusion module processes the front image and the back image to obtain the back fusion normal map; specifically:

[0108] S431: Process the front image according to the PIXIE network layer to obtain an initial digital human model, and perform back rendering on the initial digital human model to obtain a back nude normal map;

[0109] S432: Fusion the back image and the front nude normal map according to the second nude fusion layer to obtain a back dressed normal map;

[0110] S433: Parse and segment the back image according to the A-CE2P network layer to obtain a back mask, and perform Gaussian blur processing on the back mask to obtain a back confidence weight;

[0111] S434: The second dressed fusion layer fuses the back dressed normal map and the back nude normal map according to the back confidence weight to obtain a back fusion normal map.

[0112] Specifically, the data processing by the second nude fusion layer in S432 is represented by the following formula:

[0113] ,

[0114] where, represents the front nude normal map, represents the front image, represents the fusion process of the second nude fusion layer, represents the front dressed normal map;

[0115] In addition, during the training process of the second nude fusion layer, the reconstruction loss and the Laplacian loss are also used to optimize the fusion process of the first nude fusion layer, specifically:

[0116] ,

[0117] where, represents the total loss function, represents the reconstruction loss function, represents the Laplacian loss function, represents the weight coefficient.

[0118] Specifically, the back mask in S433 includes a back dressed mask and a back nude mask, the back confidence weight includes a back dressed confidence weight and a back nude confidence weight, and the back confidence generation formula is as follows:

[0119] ,

[0120] Among them, represents the back nude confidence weight, represents Gaussian blur processing, represents the back nude mask, represents the back dressed confidence weight, represents the back dressed mask;

[0121] Specifically, the process of the first dressing fusion layer processing data in S424 is represented by the following formula:

[0122] ,

[0123] Among them, represents the front fusion normal map, represents the back nude confidence weight, represents the back dressed confidence weight, represents the back nude normal map, represents the back dressed normal map.

[0124] Further, first, the back image is input into the PIXIE network layer to estimate an initial digital human model (SMPLX model); the PIXIE network is selected because it performs better than other methods in terms of the pixel alignment effect from the mesh to the image. Then, based on the estimated initial digital human model, the back nude normal map is rendered from the back view using the differentiable renderer of PyTorch3D; both the three-dimensional features of the initial digital human model and the back nude normal map rendered from it belong to global features. To make the network model more robust, it is extremely important to extract local features. Considering that the back image is predicted by the front image through the back generation module, its clarity is different from that of the front image, which means there are obvious differences in the feature distribution and quality of the data. This difference may cause the model to be difficult to balance the two different-quality data of the front and back during the learning process, thus affecting the prediction accuracy of the back normal. Therefore, in order to more fully extract the features from the back image, the second nude fusion layer (with the same structure as the first nude fusion layer) independent of the first nude fusion layer is used to fuse the back image and the back nude normal map to obtain the back dressed normal map; the second nude fusion layer can learn from the back data features, focus on the back image and back-related information, and can better capture the feature rules of the back dressed human body, so as to more accurately predict the back normal; in order to optimize the normal map generated by the model so that it not only has high accuracy at the pixel level but also has a perceptual quality close to the real image, the reconstruction loss function and the Laplacian loss function are also used for optimization on the second nude fusion layer. The reconstruction loss function directly measures the difference between the generated normal map and the target normal map at the pixel level to ensure that the generated normal map has accurate geometric details; a reconstruction loss based on cosine similarity is adopted to better adapt to the unique feature expression of the normal map. In addition, a weight parameter is introduced to control the importance of the Laplacian loss function in the total loss. The Laplacian loss function helps optimize the high-frequency information and detail recovery of the image by calculating the perceptual difference between the generated normal map and the real normal map, thereby reducing the ghosting and blurring in the normal map.However, directly using the back nude normal map and the back clothed normal map as independent inputs may lead to problems of disharmony between the human body and clothing in the reconstructed model. Since the human body and clothing are closely related, the shape of the clothing is affected by the human body form, and at the same time, it will also occlude or change the appearance of the human body. If the implicit function cannot well coordinate the relationship between these two inputs, it may lead to problems such as the human body model not fitting well with the clothing or having disharmonious appearance in the reconstructed human body model. Therefore, the surface normal reconstruction fusion module parses and segments the back image according to the A-CE2P network layer to obtain the back clothed mask and the back nude mask, and performs Gaussian blur processing on the back clothed mask and the back nude mask to obtain the back clothed confidence weight and the back nude confidence weight. The first clothing fusion layer fuses the back clothed normal map and the back nude normal map according to the back clothed confidence weight and the back nude confidence weight to obtain the back fusion normal map.

[0125] It should be noted that in S440, the human body reconstruction implicit function module processes the front image, the back image, the front normal fusion map, and the back fusion normal map to obtain a three-dimensional digital human body. Specifically:

[0126] S441: Sample query points for the initial digital human body model to obtain multiple front query points and multiple back query points;

[0127] S442: Calculate the signed distance from each front query point to each front of the initial digital human body model, and the signed distance from each back query point to each back of the initial digital human body model respectively;

[0128] S443: According to the front image, the front normal fusion map, and each front query point, obtain each front image feature, each front normal vector feature, and each front segmentation feature corresponding to each front query point;

[0129] S444: According to the back image, the back normal fusion map, and each back query point, obtain each back image feature, each back normal vector feature, and each back segmentation feature corresponding to each front query point;

[0130] S445: According to the signed distance, each front image feature, each front normal vector feature, and each front segmentation feature corresponding to each front query point, construct each front input feature corresponding to each front query point;

[0131] S446: According to the signed distance, each back image feature, each back normal vector feature, and each back segmentation feature corresponding to each back query point, construct each back input feature corresponding to each back query point;

[0132] S447: Input each front input feature and each back input feature into the implicit function to obtain each front output feature corresponding to each front input feature and each back output feature corresponding to each back input feature;

[0133] S448: The marching cubes algorithm generates a three-dimensional digital human body based on each front output feature and each back output feature.

[0134] Specifically, the front input features and back input features in S445 and S446 are represented by the following formulas:

[0135] ,

[0136] where, represents the th front input feature, represents the th front signed distance, represents the th front normal vector feature, represents the th front image feature, represents the th front segmentation feature, represents the th back input feature, represents the th back signed distance, the th back normal vector feature, represents the th back image feature, represents the th back segmentation feature.

[0137] Further, query point sampling is performed on the initial digital human model to obtain a plurality of front query points and a plurality of back query points; the signed distance from each front query point to each front of the initial digital human model, and the signed distance from each back query point to each back of the initial digital human model are calculated respectively; according to the front image, the front normal fusion map, and each front query point, each front query point corresponding to each front image feature, each front normal vector feature, and each front segmentation feature is obtained; according to the back image, the back normal fusion map, and each back query point, each front query point corresponding to each back image feature, each back normal vector feature, and each back segmentation feature is obtained; the normal vector feature is obtained by mapping the fused normal map feature with the query point, and a two-dimensional linear interpolation operation is introduced in the normal vector feature, aiming to extract the feature information of the query point position on the two-dimensional feature map; this processing method enables the model to more accurately capture the details of the human body surface from different perspectives; the image feature is obtained by mapping the query point to the image (front image and back image), enabling the model to make full use of the information in the front image and the back image to ensure that the reconstructed human model has a high degree of fidelity at all angles; the segmentation feature provides classification information about each part of the human body, helping the model better understand the human body structure. For example, the segmentation map can distinguish different parts of the human body, such as the head, arms, legs, etc., so that the model can more accurately capture the shape and positional relationship of each part. Then, according to each signed distance of each front query point corresponding to each front, each front image feature, each front normal vector feature, and each front segmentation feature, each front input feature corresponding to each front query point is constructed; according to each signed distance of each back query point corresponding to each back, each back image feature, each back normal vector feature, and each back segmentation feature, each back input feature corresponding to each back query point is constructed; each front input feature and each back input feature are input into a hidden function parameterized by a multi-layer perceptron (MLP) to obtain each front output feature corresponding to each front input feature, and each back output feature corresponding to each back input feature; this function can evaluate the occupancy of each three-dimensional query point and output the corresponding occupancy rate; finally, according to the Marching Cubes algorithm, a three-dimensional digital human is generated based on each front output feature and each back output feature. The Marching Cubes algorithm is a classic three-dimensional reconstruction algorithm, which divides the cube in the space of the hidden function and performs interpolation according to the occupancy of each vertex of the cube to generate a continuous three-dimensional surface (three-dimensional digital human); this method can not only efficiently generate high-quality three-dimensional meshes, but also ensure the smoothness and continuity of the reconstructed model.

[0138] A 3D digital human body generation method according to this embodiment generates a human body back image through a latent diffusion model combined with joint point information, converts the missing back geometry and texture information in a single view into data consistent with a dual-view, and improves the accuracy of back image prediction; designs a first nude fusion layer and a second nude fusion layer to reconstruct the front and back normal maps; through a human body parsing segmentation and normal fusion mechanism, enhances the model's associated modeling of the human body surface and clothing folds, and significantly improves the smoothness and geometric consistency of the dressed transition area; through the cooperation among the back generation module, the normal reconstruction and fusion module, and the human body reconstruction implicit function module, improves the generalization ability of the method for complex poses and diverse clothing, and further verifies that this method is superior to the prior art in terms of accuracy and visual effects.

[0139] It should be noted that for a 3D digital human body generation method provided in an embodiment of this application, the execution subject can be a 3D digital human body generation system, or, or a control module in the 3D digital human body generation system for executing and loading a 3D digital human body generation method. In an embodiment of this application, taking a 3D digital human body generation system executing and loading a 3D digital human body generation method as an example, a 3D digital human body generation method provided in an embodiment of this application is described.

[0140] A 3D digital human body generation system includes:

[0141] An acquisition module: acquires a human body front image dataset, preprocesses the human body front image dataset, and obtains a target dataset;

[0142] A construction module: constructs a back generation normal fusion network model, and the normal fusion network model includes a back generation module, a front normal reconstruction and fusion module, a back normal prediction and fusion module, and a human body reconstruction implicit function module;

[0143] A training and testing module: trains the back generation normal fusion network model according to part of the data in the target dataset, and tests the trained back generation normal fusion network model with the other part of the data;

[0144] A prediction module: inputs the front image of the 3D digital human body to be generated into the tested back generation normal fusion network model, and the back generation normal fusion network model outputs the 3D digital human body corresponding to the front image of the 3D digital human body to be generated.

[0145] A 3D digital human body generation system in an embodiment of the present application may be a device, or a component, an integrated circuit, or a chip in a terminal. The device may be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device may be a mobile phone, a tablet computer, a laptop computer, a handheld computer, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc., and the non-mobile electronic device may be a server, a network attached storage (NAS), a personal computer (PC), etc. The embodiments of the present application do not make specific limitations.

[0146] A 3D digital human body generation system in an embodiment of the present application may be a device with an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems. The embodiments of the present application do not make specific limitations.

[0147] A 3D digital human body generation system provided in an embodiment of the present application can implement Figures 1 to 3 each process implemented by a 3D digital human body generation method in the method embodiment. To avoid repetition, it will not be elaborated here.

[0148] According to a 3D digital human body generation system of this embodiment, first, the acquisition module acquires a dataset of human front images and preprocesses the dataset of human front images to obtain a target dataset; then, the construction module constructs a back generation normal fusion network model, where the normal fusion network model includes a back generation module, a front normal reconstruction and fusion module, a back normal prediction and fusion module, and a human body reconstruction implicit function module; then, through the training and testing module, the back generation normal fusion network model is trained according to part of the data in the target dataset, and another part of the data is used to test the trained back generation normal fusion network model; finally, the prediction module inputs the front image of the 3D digital human body to be generated into the tested back generation normal fusion network model, and the back generation normal fusion network model outputs the 3D digital human body corresponding to the front image of the 3D digital human body to be generated. Through the mutual cooperation between the modules, by inputting a human front image, a 3D digital human body can be obtained after being processed by the system, realizing an automated generation process.

[0149] Optionally, an embodiment of the present application further provides an electronic device, including a processor, a memory, and a program or instruction stored on the memory and executable on the processor. When the program or instruction is executed by the processor, it implements each process of the above embodiment of the three-dimensional digital human body generation method and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0150] An embodiment of the present application further provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by the processor, it implements each process of the above embodiment of the three-dimensional digital human body generation method and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0151] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc, etc.

[0152] It should be noted that in this article, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article or device including that element. In addition, it should be pointed out that the methods and devices in the embodiments of the present application are not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in a reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0153] Through the description of the above embodiments, those skilled in the art can clearly understand that the methods in the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc), and includes several instructions for causing a terminal (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in the various embodiments of the present application.

[0154] The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above specific implementation manners. The above specific implementation manners are merely illustrative rather than restrictive. Under the inspiration of the present application, those of ordinary skill in the art can also make many forms without departing from the gist of the present application and the scope protected by the claims, and all of them fall within the protection scope of the present application.

Claims

1. A three-dimensional digital human body generation method, characterized in that, The method includes: Obtain a dataset of human front images, preprocess the dataset of human front images to obtain a target dataset; Construct a back generation normal fusion network model, which includes a back generation module, a front normal reconstruction fusion module, a back normal prediction fusion module, and a human body reconstruction implicit function module; The back generation module includes a CLIP image encoder, a VAE encoder, an OPENPOSE network layer, a ControlNet network layer, and a latent diffusion network layer. The back generation module processes the front image to generate a back image corresponding to the front image. Specifically: Extract features from the front image according to the CLIP image encoder and the VAE encoder respectively to obtain a CLIP feature encoding and a VAE feature encoding; Extract key points from the front image according to the OPENPOSE network layer to obtain a key point map; Extract features from the key point map according to the ControlNet network layer to obtain key point features; Process the CLIP feature encoding, the VAE feature encoding, and the key point features according to the latent diffusion network layer to obtain the back image; Train the back generation normal fusion network model according to a part of the data in the target dataset, and test the trained back generation normal fusion network model with another part of the data; Input the front image of the three-dimensional digital human body to be generated into the tested back generation normal fusion network model, and the back generation normal fusion network model outputs the three-dimensional digital human body corresponding to the front image of the three-dimensional digital human body to be generated.

2. A three-dimensional digital human body generation method according to claim 1, characterized in that, The training of the back generation normal fusion network model according to a part of the data in the target dataset, and the testing of the trained back generation normal fusion network model with another part of the data; Specifically: Divide the target dataset into a target training set and a target test set according to a preset ratio; Construct a loss function, and train the back generation normal fusion network model according to the target dataset and the loss function; Test the back generation normal fusion network model according to the target test set to complete the training and testing of the back generation normal fusion network model.

3. A three-dimensional digital human body generation method according to claim 1, characterized in that, Input the front image of the three-dimensional digital human body to be generated into the tested back generation normal fusion network model, and the back generation normal fusion network model outputs the three-dimensional digital human body corresponding to the front image of the three-dimensional digital human body to be generated; Specifically: After the front image of the three-dimensional digital human body to be generated is input into the back generation normal fusion network model, the back generation module processes the front image to generate a back image corresponding to the front image; Input the front image into the front normal reconstruction fusion module, and the front normal reconstruction fusion module processes the front image to obtain a front fusion normal map; Input the front image and the back image into the back normal prediction and fusion module, which processes the front image and the back image to obtain a back fused normal map; Input the front image, the back image, the front normal fused map, and the back fused normal map into the human body reconstruction implicit function module, which processes the front image, the back image, the front normal fused map, and the back fused normal map to obtain the 3D digital human body; 4. A three-dimensional digital human body generation method according to claim 1, characterized in that The front normal reconstruction and fusion module includes a PIXIE network layer, a first nude fusion layer, an A-CE2P network layer, and a first dressed fusion layer. The front normal reconstruction and fusion module processes the front image to obtain a front fused normal map; specifically: Process the front image according to the PIXIE network layer to obtain an initial digital human body model, and perform front rendering on the initial digital human body model to obtain a front nude normal map; Fuse the front image and the front nude normal map according to the first nude fusion layer to obtain a front dressed normal map; Parse and segment the front image according to the A-CE2P network layer to obtain a front mask, and perform Gaussian blur processing on the front mask to obtain a front confidence weight; The first dressed fusion layer fuses the front dressed normal map and the front nude normal map according to the front confidence weight to obtain the front fused normal map.

5. A three-dimensional digital human body generation method according to claim 4, characterized in that The back normal prediction and fusion module includes a PIXIE network layer, a second nude fusion layer, an A-CE2P network layer, and a second dressed fusion layer. The back normal prediction and fusion module processes the front image and the back image to obtain a back fused normal map; specifically: Process the front image according to the PIXIE network layer to obtain an initial digital human body model, and perform back rendering on the initial digital human body model to obtain a back nude normal map; Fuse the back image and the front nude normal map according to the second nude fusion layer to obtain a back dressed normal map; Parse and segment the back image according to the A-CE2P network layer to obtain a back mask, and perform Gaussian blur processing on the back mask to obtain a back confidence weight; The second dressed fusion layer fuses the back dressed normal map and the back nude normal map according to the back confidence weight to obtain the back fused normal map.

6. A three-dimensional digital human body generation method according to claim 5, characterized in that, The human body reconstruction implicit function module processes the front image, the back image, the front normal fused map, and the back fused normal map to obtain the 3D digital human body; specifically: Perform query point sampling on the initial digital human body model to obtain a plurality of front query points and a plurality of back query points; Calculate the signed distance from each front query point to each front surface of the initial digital human body model, and the signed distance from each back query point to each back surface of the initial digital human body model respectively; Based on the front image, the front normal fusion map, and each front query point, obtain each front image feature, each front normal vector feature, and each front segmentation feature corresponding to each front query point; Based on the back image, the back normal fusion map, and each back query point, obtain each back image feature, each back normal vector feature, and each back segmentation feature corresponding to each front query point; Based on each front signed distance, each front image feature, each front normal vector feature, and each front segmentation feature corresponding to each front query point, construct each front input feature corresponding to each front query point; Based on each back signed distance, each back image feature, each back normal vector feature, and each back segmentation feature corresponding to each back query point, construct each back input feature corresponding to each back query point; Input each front input feature and each back input feature into the implicit function to obtain each front output feature corresponding to each front input feature and each back output feature corresponding to each back input feature; The marching cubes algorithm generates the three-dimensional digital human body based on each front output feature and each back output feature.

7. A three-dimensional digital human body generation system capable of implementing the three-dimensional digital human body generation method according to any one of claims 1-6, characterized in that, The system includes: An acquisition module: acquire a human front image dataset, and preprocess the human front image dataset to obtain a target dataset; A construction module: construct a back-generated normal fusion network model, and the normal fusion network model includes a back generation module, a front normal reconstruction and fusion module, a back normal prediction and fusion module, and a human body reconstruction implicit function module; A training and testing module: train the back-generated normal fusion network model according to a part of the data in the target dataset, and test the trained back-generated normal fusion network model with another part of the data; A prediction module: input the front image of the three-dimensional digital human body to be generated into the tested back-generated normal fusion network model, and the back-generated normal fusion network model outputs the three-dimensional digital human body corresponding to the front image of the three-dimensional digital human body to be generated.

8. An electronic device, characterized in that, It includes a processor, a memory, and a program or instruction stored on the memory and executable on the processor. When the program or instruction is executed by the processor, the steps of a three-dimensional digital human body generation method according to any one of claims 1-6 are implemented.

9. A readable storage medium, characterized in that, A program or instruction is stored on the readable storage medium. When the program or instruction is executed by the processor, the steps of a three-dimensional digital human body generation method according to any one of claims 1-6 are implemented.

Images