A method, system, electronic device and storage medium for reconstructing three-dimensional hair

Abstract

The application discloses a three-dimensional hair reconstruction method and system, an electronic device and a storage medium, relates to the technical field of computer graphics and computer vision, and the method comprises the following steps: acquiring an RGB portrait to be reconstructed; inputting the RGB portrait to be reconstructed into a three-dimensional voxel hair reconstruction model to obtain target three-dimensional voxel hair; the three-dimensional voxel hair reconstruction model is trained by using a training portrait set, a three-dimensional hair training set and corresponding text; the reconstruction network comprises an image encoder and a trained voxel decoder, the trained voxel decoder is determined according to a trained VAE-Patch_SNGAN network, and the trained VAE-Patch_SNGAN network is used for training the VAE-Patch_SNGAN network by using the three-dimensional hair training set; and a target three-dimensional hair image is reconstructed according to the target three-dimensional voxel hair. The application improves the reconstruction speed and accuracy of the three-dimensional hair.

Description

Technical Field

[0001] This invention relates to the fields of computer graphics and computer vision, and in particular to a method, system, electronic device, and storage medium for reconstructing three-dimensional hair. Background Technology

[0002] Hair is an important physical feature of humans. In recent years, with the rapid development of industries such as gaming, digital virtual human modeling has become increasingly important. Due to the complexity of hair geometry, creating high-quality 3D hair is one of the most time-consuming tasks in modeling high-quality virtual human characters. Therefore, much work has been dedicated to achieving automated 3D hair modeling.

[0003] Existing methods for reconstructing hair geometry from images mainly include two technical approaches: multi-view and single-view. Multi-view modeling approaches borrow from traditional multi-view reconstruction techniques, using feature matching to solve for depth values ​​and obtain a coarse hair point cloud, then constructing individual hair strands based on this point cloud. These approaches often rely on expensive hardware, such as hair capture systems using omnidirectional controllable light sources or using Kinect to scan RGBD images. While multi-view approaches can reconstruct relatively accurate hair geometry, the complex equipment and long processing times make them less user-friendly and hinder widespread deployment and use.

[0004] Single-view-based modeling schemes take an RGB portrait as input and employ data-driven or deep learning methods. Data-driven methods compare the input image with hair similarity data in a database to retrieve the most similar hairstyle as output, then further optimize the retrieved hairstyle to closely resemble the input image. This method is time-consuming, heavily reliant on the database, and has limited ability to generate new data.

[0005] Deep learning methods construct a supervised dataset and learn the mapping relationship between 2D images and 3D hairstyles through neural networks. Supervised datasets, i.e., datasets containing pairs of real hairstyle images and their corresponding 3D hairstyles, are often difficult to obtain. Therefore, most methods use 3D hair data from rendered databases to construct virtual datasets. However, the amount of 3D hair data is limited, and the rendered images differ from the real images, which poses challenges for the network during the inference phase. Furthermore, the geometry of hair differs from that of a human face, which can be fully represented by surface patches. Therefore, when using differentiable rendering, the backpropagation gradient can only be propagated back to the surface, hindering unsupervised learning by the network.

[0006] The advantages of single-view methods are user-friendliness; they can directly use human images as input to reconstruct relatively reasonable hair structures. In particular, neural network-based methods can generate similar hairstyles that are not present in the dataset, and their inference speed is fast. The disadvantages are that they often only reconstruct a general hair shape and structure, lacking detail, and are difficult to model for hair types missing from the training data (such as braids).

[0007] Therefore, existing methods for reconstructing three-dimensional hair suffer from slow reconstruction speed and low accuracy. Summary of the Invention

[0008] The purpose of this invention is to provide a method, system, electronic device, and storage medium for reconstructing three-dimensional hair, thereby improving the speed and accuracy of three-dimensional hair reconstruction.

[0009] To achieve the above objectives, the present invention provides the following solution:

[0010] A method for reconstructing three-dimensional hair, the method comprising:

[0011] Obtain the RGB portrait to be reconstructed;

[0012] The RGB human image to be reconstructed is input into a 3D voxel hair reconstruction model to obtain the target 3D voxel hair. The 3D voxel hair reconstruction model is trained on a reconstruction network using a training set of human images, a 3D hair training set, and the text corresponding to the 3D hair training set. The reconstruction network includes an image encoder and a trained voxel decoder. The trained voxel decoder is determined based on a trained VAE-Patch_SNGAN network, which is trained using the 3D hair training set. The VAE-Patch_SNGAN network includes a 3D voxel generator and a first discriminator. The 3D voxel generator includes a voxel encoder and a voxel decoder.

[0013] Reconstruct a target 3D hair image based on the target 3D voxel hair.

[0014] Optionally, the training process of the three-dimensional voxel hair reconstruction model includes:

[0015] Determine the training image set, the 3D hair training set, and the text corresponding to the 3D hair training set; the training image set includes multiple RGB images for training, and the 3D hair training set includes the occupancy ground truth and orientation ground truth corresponding to multiple initial hair voxels;

[0016] Determine the hair segmentation mask and hair orientation map for each of the aforementioned RGB portrait images used for training;

[0017] Based on the training image set and the 3D hair training set, the projection viewpoint parameters are determined; the projection viewpoint parameters include: rotation matrix and translation vector;

[0018] The VAE-Patch_SNGAN network is trained based on the three-dimensional hair training set to obtain the trained VAE-Patch_SNGAN network.

[0019] A reconstruction network is constructed based on the image encoder and the trained voxel decoder.

[0020] The three-dimensional hair training set is rendered to obtain a set of rendered images;

[0021] Based on the text corresponding to the rendered image set and the 3D hair training set, determine the rendered image set with text descriptions;

[0022] Based on the projection viewpoint parameters, the hair segmentation mask and hair orientation map of the RGB human image used for training, and the set of rendered images with text descriptions, the reconstruction network is trained to obtain the three-dimensional voxel hair reconstruction model.

[0023] Optionally, projection viewpoint parameters are determined based on the training image set and the 3D hair training set, specifically including:

[0024] Identify multiple 2D facial key points in the RGB portrait used for training;

[0025] Based on all the 2D and 3D facial key points, the projection viewpoint parameters are determined; the 3D facial key points are the 3D facial key points of a standard human body that are aligned with the hair strands corresponding to each hair voxel in the 3D hair training set.

[0026] Optionally, the VAE-Patch_SNGAN network is trained based on the three-dimensional hair training set to obtain a trained VAE-Patch_SNGAN network, specifically including:

[0027] Based on the three-dimensional voxel generator and the first discriminator, the VAE-Patch_SNGAN network is constructed;

[0028] All the initial hair voxels are input into the VAE-Patch_SNGAN network to obtain the reconstructed hair voxels corresponding to each initial hair voxel;

[0029] Determine the occupancy and orientation values ​​of each of the reconstructed hair matrix elements;

[0030] Based on the occupancy and orientation values ​​of each reconstructed hair cell and the occupancy and orientation values ​​of each initial hair cell, the first loss function of the VAE-Patch_SNGAN network is constructed.

[0031] The VAE-Patch_SNGAN network is fine-tuned based on the first loss function to obtain the trained VAE-Patch_SNGAN network.

[0032] Optionally, the reconstruction network is trained based on the projection viewpoint parameters, the hair segmentation mask and hair orientation map of the RGB portrait used for training, and the set of rendered images with text descriptions to obtain the three-dimensional voxel hair reconstruction model, specifically including:

[0033] The hair segmentation mask and hair orientation map of the RGB human image used for training are input into the reconstruction network to obtain multiple reconstructed three-dimensional voxel hairs;

[0034] Determine the occupancy and orientation values ​​of each of the reconstructed 3D voxel hairs;

[0035] Determine whether each rendered image with text description in the set of rendered images with text description has a three-dimensional ground truth value;

[0036] If so, then based on the occupancy ground truth and orientation ground truth corresponding to the rendered image with text description and the occupancy and orientation values ​​of each reconstructed 3D voxel hair, a second loss function of the reconstruction network is constructed, and the reconstruction network is fine-tuned based on the second loss function to obtain the 3D voxel hair reconstruction model.

[0037] If not, the reconstructed 3D voxel hair corresponding to the rendered image with text description but no 3D ground truth is input into the differentiable volume renderer, and rendered based on the projection viewpoint parameters to obtain the rendered hair segmentation mask and hair orientation map. The third loss function of the reconstruction network is constructed based on the hair segmentation mask and hair orientation map of the training RGB portrait and the rendered hair segmentation mask and hair orientation map. The reconstruction network is then fine-tuned based on the third loss function to obtain the 3D voxel hair reconstruction model.

[0038] Optionally, the method for determining the three-dimensional hair training set includes:

[0039] Obtain an initial 3D hair set; the 3D hair set includes multiple sets of initial hair point clouds;

[0040] Each initial hair point cloud is converted into an initial hair voxel, and the occupancy value and hair orientation value of each initial hair voxel are determined, thereby determining the three-dimensional hair training set.

[0041] A three-dimensional hair reconstruction system, the system comprising:

[0042] The module for acquiring RGB portraits to be reconstructed is used to acquire RGB portraits to be reconstructed.

[0043] A target 3D voxel hair determination module is used to input the RGB human image to be reconstructed into a 3D voxel hair reconstruction model to obtain the target 3D voxel hair. The 3D voxel hair reconstruction model is trained on a reconstruction network using a training set of human images, a 3D hair training set, and text corresponding to the 3D hair training set. The reconstruction network includes an image encoder and a trained voxel decoder. The trained voxel decoder is determined based on a trained VAE-Patch_SNGAN network. The trained VAE-Patch_SNGAN network is trained using the 3D hair training set. The VAE-Patch_SNGAN network includes a 3D voxel generator and a first discriminator. The 3D voxel generator includes a voxel encoder and a voxel decoder.

[0044] A target 3D voxel hair determination module is used to reconstruct a target 3D hair image based on the target 3D voxel hair.

[0045] An electronic device, comprising:

[0046] One or more processors;

[0047] A storage device on which one or more programs are stored;

[0048] When the one or more programs are executed by the one or more processors, the one or more processors implement the three-dimensional hair reconstruction method as described above.

[0049] A storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the three-dimensional hair reconstruction method as described above.

[0050] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0051] This invention discloses a method, system, electronic device, and storage medium for reconstructing three-dimensional hair. Based on the VAE-Patch_SNGAN network, a three-dimensional voxel hair reconstruction model is constructed, which can improve the detail of the reconstructed hair and improve the accuracy of the three-dimensional hair reconstruction. Moreover, compared with the traditional multi-view reconstruction method, this invention only needs to use a single-view RGB portrait to realize the reconstruction of three-dimensional hair, which improves the reconstruction speed and accuracy of three-dimensional hair. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a schematic diagram of the three-dimensional hair reconstruction method provided in Embodiment 1 of the present invention;

[0054] Figure 2 A schematic diagram illustrating the process of determining the hair segmentation mask and the corresponding hair direction map;

[0055] Figure 3 This is a schematic diagram of a method for generating rendered images with text descriptions based on a diffusion model.

[0056] Figure 4 A schematic diagram of facial landmark detection;

[0057] Figure 5 A flowchart illustrating the process of determining projection angle parameters;

[0058] Figure 6 This is a schematic diagram of the VAE-Patch_SNGAN network structure;

[0059] Figure 7 This is a schematic diagram illustrating the training process of a 3D voxel hair reconstruction model.

[0060] Figure 8 This is a schematic diagram of the projection process from a three-dimensional direction to a two-dimensional direction.

[0061] Figure 9 A schematic diagram illustrating the conversion of 3D hair strand data into 3D voxel hair.

[0062] Figure 10 This is a flowchart combining training and reconstruction. Detailed Implementation

[0063] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] The purpose of this invention is to provide a method, system, electronic device, and storage medium for reconstructing three-dimensional hair, aiming to improve the speed and accuracy of three-dimensional hair reconstruction.

[0065] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0066] Example 1

[0067] Figure 1 This is a schematic diagram of the three-dimensional hair reconstruction method provided in Embodiment 1 of the present invention. Figure 10 A flowchart combining training and reconstruction. (e.g.) Figure 1 and Figure 10 As shown, the three-dimensional hair reconstruction method in this embodiment includes:

[0068] Step 201: Obtain the RGB portrait to be reconstructed.

[0069] Step 202: Input the RGB human image to be reconstructed into the 3D voxel hair reconstruction model to obtain the target 3D voxel hair.

[0070] The 3D voxel hair reconstruction model is trained on the reconstruction network using a training set of human images, a 3D hair training set, and the corresponding text. The reconstruction network includes an image encoder and a trained voxel decoder. The trained voxel decoder is determined based on the trained VAE-Patch_SNGAN network, which is trained using the 3D hair training set. The VAE-Patch_SNGAN network includes a 3D voxel generator and a first discriminator. The 3D voxel generator includes a voxel encoder and a voxel decoder.

[0071] Step 203: Reconstruct the target 3D hair image based on the target 3D voxel hair.

[0072] Specifically, step 203 includes:

[0073] The 3D voxel hair (i.e., the target 3D voxel hair) is converted into the corresponding hair strand data format and input into the renderer to obtain a digitized 3D hairstyle (i.e., the target 3D hair image). Specifically, the method for converting voxel data to hair strand data is as follows: starting from a fixed starting point on the scalp, the next hair strand point is obtained by following the spatial vector stored in the direction field of the voxel. The position of the next hair strand point in the voxel space is queried, the stored direction is read, and the third point is obtained along that direction value. This process is repeated until the voxel occupancy value of the newly generated point is 0, indicating that the hair has grown to the boundary and the hair strand growth is complete. The formula is expressed as:

[0074]

[0075] Where, x nx is the current hair point. n+1 Let 's' be the next hair strand point, and 's' be the step size. This represents the current growth direction of the hair strand stored in the voxel.

[0076] As an optional implementation, the training process of the 3D voxel hair reconstruction model includes:

[0077] Determine the training image set, the 3D hair training set, and the corresponding text for the 3D hair training set; the training image set includes multiple RGB images for training, and the 3D hair training set includes the occupancy ground truth and orientation ground truth corresponding to multiple initial hair voxels.

[0078] Determine the hair segmentation mask and hair orientation map for each training RGB portrait.

[0079] Specifically, such as Figure 2 As shown, the hair segmentation mask and corresponding hair orientation map of the RGB portrait are determined, specifically including:

[0080] The RGB image of the person is input into a matting network to obtain the foreground (the foreground refers to the part of the image that belongs to the person). In this embodiment, deep image matting is used as the matting network to separate the foreground from the background. The foreground is then input into a hair segmentation network built on DeepLabV3+ to obtain the hair segmentation mask of the image. The DeepLabV3+ network can be replaced with other commonly used segmentation networks. The main problem with existing methods for obtaining hair segmentation masks is that when the hair color is similar to the adjacent background color, the network is prone to misjudging the boundary between the hair and the background. In this embodiment, existing matting algorithms can effectively distinguish between the foreground and background, and using the foreground of the person as the input to the segmentation network solves this problem well.

[0081] The intersection of the hair segmentation mask and the RGB image yields the hair region. This region is then converted to a grayscale image and filtered using a directional Gabor filter to determine the direction value of each pixel. The output grayscale image is the hair direction map (i.e., convolving the grayscale image with the filter's transfer function; the convolution result is the response value obtained after filtering the image through filters at various angles. Each pixel has a different response value after filtering at different angles, and the angle corresponding to the maximum response value is the direction of the hair strand at that pixel. The horizontal direction is 0 degrees, and the convolution kernels are distributed between 0 and 180 degrees; this embodiment uses 180 convolution kernels ranging from 0 to 179 degrees). The direction map is a grayscale image of the same size as the input image, where the grayscale value of each pixel represents the two-dimensional direction of the hair strand at that pixel. Due to the presence of noise and dark areas, the original output hair direction map may contain inaccuracies. Therefore, the original hair direction map is further Gaussian smoothed to calculate the final dense direction map. When calculating the hair orientation map, an erosion operation was performed on the hair segmentation mask to remove interference from cheeks and edges in orientation extraction.

[0082] Based on the training set of human images and the 3D hair training set, the projection viewpoint parameters are determined; the projection viewpoint parameters include: rotation matrix and translation vector.

[0083] The VAE-Patch_SNGAN network is trained using a 3D hair training set to obtain a trained VAE-Patch_SNGAN network.

[0084] A reconstruction network is constructed based on the image encoder and the trained voxel decoder.

[0085] The 3D hair training set is rendered to obtain a set of rendered images.

[0086] Based on the text corresponding to the rendered image set and the 3D hair training set, the rendered image set with text descriptions is determined.

[0087] Specifically, such as Figure 3 As shown, based on the text corresponding to the rendered image set and the 3D hair training set, the rendered image set with text descriptions is determined, specifically including:

[0088] The Stable Diffusion model, serving as the base network, was trained using the LAION multimodal dataset to obtain a pre-trained diffusion model. Stable Diffusion is a popular text-to-image network model (used for AI drawing), taking English text descriptions as input and outputting images of those descriptions. The LAION dataset contains a massive amount of text-image pairs.

[0089] Using a pre-trained diffusion model, rendered images and text prompts are input to generate realistic hairstyle images (i.e., a set of rendered images with text descriptions). The text prompts are English descriptions of the visual effect of the generated images, such as "realistic hair." The text prompts are an input to the network, used to add textual descriptions to the images and minimize unstructured distributional differences between the rendered hairstyle images and realistic hairstyle images.

[0090] In this embodiment, the diffusion model consists of a CLIP image encoder, a CLIP text encoder, a Latent Diffusion Model (LDM), and a CLIP decoder. CLIP (Contrastive Language-Image Pre-Training) is a pre-trained neural network model for matching images and text. It can encode images and text and then reconstruct them using a decoder. The diffusion model has been pre-trained on a large-scale dataset and can be directly used. The rendered image is input into the CLIP image encoder to obtain the image encoding, and the text prompt (in this embodiment, the text description is "realistic hair, smooth light") is input into the CLIP text encoder to obtain the text encoding. The image encoding and text encoding are combined and input into the Latent Diffusion Model network. The encoding undergoes a forward process (adding Gaussian noise) and a backward process (denoising), and is finally input into the CLIP decoder to obtain the rendered image with the text description (i.e., the image is rendered with the text description). Figure 3 (The realistic hair in the picture).

[0091] Based on projection viewpoint parameters, a hair segmentation mask and hair orientation map of RGB human images used for training, and a set of rendered images with text descriptions, the reconstruction network is trained to obtain a three-dimensional voxel hair reconstruction model.

[0092] As an optional implementation, projection viewpoint parameters are determined based on the training image set and the 3D hair training set, specifically including:

[0093] Identify multiple 2D facial key points in the RGB portrait used for training.

[0094] Based on all 2D and 3D facial key points, the projection viewpoint parameters are determined. The 3D facial key points are the standard human facial key points aligned with the hair strands corresponding to each hair voxel in the 3D hair training set.

[0095] Specifically, based on the training image set and the 3D hair training set, the projection viewpoint parameters are determined, including:

[0096] like Figure 4As shown, 2D facial key points k (68 commonly used 2D facial key points, each with a fixed physical meaning, such as the tip of the nose, the corner of the eye, etc.) are detected in RGB portraits. In this embodiment, the face_aligment library in Python is used to extract the 68 key points of the face.

[0097] like Figure 5 As shown, based on the definition of facial key points, the 3D facial key points K of the standard human body are calibrated and aligned with the 3D hair strands in the database. Alignment means translating the starting point of the 3D hair strands to the scalp vertex of the 3D human body.

[0098] By minimizing the residual between k' and k obtained from constrained K-projection (projecting 3D facial key points from the front to 2D), the required viewpoint parameters for projection (i.e., projection viewpoint parameters, including the rotation matrix R and translation vector T) are optimized. The above process can be expressed as the following formula: k' = M proj K.

[0099] Among them, M proj The projection process includes translation and rotation of 3D facial key points, perspective projection, and viewport transformation. The rotation matrix R and translation vector T are M. proj The parameters to be optimized during the projection process are as follows: The loss function is ||k-k'||², where ||²| is the L2 norm. The projection viewpoint parameters are obtained by iterative optimization using gradient descent and optimizers such as Adam to minimize the loss function. This embodiment uses PyTorch to solve this optimization problem.

[0100] As an optional implementation, the VAE-Patch_SNGAN network is trained based on a 3D hair training set to obtain a trained VAE-Patch_SNGAN network, specifically including:

[0101] A VAE-Patch_SNGAN network is constructed based on a 3D voxel generator and a first discriminator.

[0102] All initial hair voxels are input into the VAE-Patch_SNGAN network to obtain the reconstructed hair voxels corresponding to each initial hair voxel.

[0103] Determine the occupancy and orientation values ​​of each reconstructed hair cell.

[0104] Based on the occupancy and orientation values ​​of each reconstructed hair cell, as well as the ground truth occupancy and orientation values ​​of each initial hair cell, the first loss function of the VAE-Patch_SNGAN network is constructed.

[0105] The VAE-Patch_SNGAN network is fine-tuned based on the first loss function to obtain the trained VAE-Patch_SNGAN network.

[0106] Specifically, such as Figure 6 As shown, the network structure of the Variational Auto Encoder-Patch_SpectralNormalization GAN (VAE-Patch_SNGAN) with a block discriminator featuring parameter spectrum normalization consists of a 3D voxel generator (3DVAE) and a first discriminator (SN_PatchDiscriminator). The 3D VAE network comprises a voxel encoder and a voxel decoder; its input is a 3D voxel hair (i.e., the initial hair voxel), and its output is also a 3D voxel hair (i.e., the reconstructed hair voxel). The SN_PatchDiscriminator network takes a 3D voxel hair as input and outputs an evaluation matrix. The VAE-Patch_SNGAN network is used to learn the mapping relationship between the encoding and the 3D voxel hair. The advantage of this invention lies in building a 3D convolutional network and introducing the SN_PatchDiscriminator on the basis of the original variational autoencoder, further improving the model's generative ability and reconstruction details. Spectral normalization (SN) ensures that the discriminator parameters satisfy Lipschitz continuity, thus stabilizing the training of generative adversarial networks (GANs). The original GAN's discriminator outputs only a scalar to evaluate the overall output of the generator. PatchGAN's discriminator is designed as a fully convolutional matrix, mapping the input to an evaluation matrix where each element is an evaluation value for a local region in the original image, focusing more on local details.

[0107] The first loss function used in the VAE-Patch_SNGAN network is:

[0108] L VAE-GAN =ω1L occ +ω2L ori +ω3*KL(z||N(0,1))+ω4L g .

[0109]

[0110]

[0111] L g =-E(D(G(z))).

[0112] L d=E(max(0,1-D(v)))+E(max(0,1+D(G(z)))).

[0113] Among them, L VAE-GAN This is the total loss function (i.e., the first loss function) used to train VAE-Patch_SNGAN. occ It is the loss function of the 3D voxel occupancy field (the 3D voxel occupancy field refers to the voxel occupancy value of 1 when hair passes through in 3D space, and the voxel occupancy value of 0 when no hair passes through. The occupancy field is used to mark whether hair has passed through a certain 3D location, similar to a 2D hair segmentation mask, and it is a component of the 3D voxel hair), L ori This is the 3D voxel orientation field loss function (the 3D voxel orientation field refers to the average of all hair growth directions passing through a voxel, stored as a 3D unit vector. The orientation field is used to mark the hair growth direction at a 3D location, similar to a 2D hair orientation map, and is a component of the 3D voxel hair. The occupancy field and the orientation field together constitute the 3D voxel hair), L g L is the loss function for a 3D voxel generator (3D VAE). d This is the loss function of the first discriminator (SN_PatchDiscriminator). ω1 is the weight of the first term, a manually set hyperparameter. ω2 is the weight of the second term, a manually set hyperparameter. ω3 is the weight of the third term, a manually set hyperparameter. ω4 is the weight of the fourth term, a manually set hyperparameter. i This represents the truth value of the field occupied by the i-th voxel. i Let represent the true value of the directional field of the i-th voxel. The value of the occupancy field of the i-th voxel output by the VAE-Patch_SNGAN network; represents the value of the orientation field of the i-th voxel output by the VAE-Patch_SNGAN network. N is the total number of voxels involved in the calculation, and |||1 is the L1 norm. E represents the mean function, D is the discriminator network after spectral normalization (i.e., the first discriminator), v is the ground truth value of the 3D voxel hair, z is the implicit code output by the VAE encoder, and G is the 3D voxel generator.

[0114] As an optional implementation, a reconstruction network is trained based on projection viewpoint parameters, a hair segmentation mask and hair orientation map of the RGB human image used for training, and a set of rendered images with text descriptions to obtain a 3D voxel hair reconstruction model, specifically including:

[0115] The hair segmentation mask and hair orientation map of the RGB human image used for training are input into the reconstruction network to obtain multiple reconstructed 3D voxel hairs.

[0116] Determine the occupancy and orientation values ​​of each reconstructed 3D voxel hair.

[0117] Determine whether each rendered image with text description in the set of rendered images with text description has a 3D ground truth value.

[0118] If so, then based on the occupancy and orientation ground truth values ​​of the rendered image with text descriptions and 3D ground truth values, as well as the occupancy and orientation values ​​of each reconstructed 3D voxel hair, a second loss function of the reconstruction network is constructed, and the reconstruction network is fine-tuned based on the second loss function to obtain the 3D voxel hair reconstruction model.

[0119] If not, the reconstructed 3D voxel hair corresponding to the rendered image with text description but no 3D ground truth is input into the differentiable renderer. Rendering is performed based on the projection viewpoint parameters to obtain the rendered hair segmentation mask and hair orientation map. A third loss function for the reconstruction network is constructed based on the hair segmentation mask and hair orientation map of the RGB portrait used for training, as well as the rendered hair segmentation mask and hair orientation map. The reconstruction network is then fine-tuned based on the third loss function to obtain the 3D voxel hair reconstruction model.

[0120] Specifically, such as Figure 7 As shown, an image encoder (backbone feature extraction network), a differentiable volume renderer for self-supervised learning, and a transformation module and discriminator (i.e., a second discriminator) for supervised training are constructed, which, together with the voxel decoder in the trained VAE-Patch_SNGAN, constitute the reconstruction network. The reconstruction network specifically includes:

[0121] The image encoder, based on ResNet, can be replaced by other common convolutional neural networks to achieve similar results. The input to the image encoder is a hair segmentation mask and hair orientation map from an RGB human image used for training; the output is a 1024-dimensional code.

[0122] The trained voxel decoder follows the image encoder. Its input is the 1024-dimensional code output by the image encoder, and its output is a three-dimensional voxel hair.

[0123] The differentiable volume renderer based on the volume rendering method is a component for self-supervised learning. Its input is a 3D voxel hair output from the voxel decoder, and its output is a hair mask and hair direction map projected from this 3D voxel hair. The specific implementation method is as follows: Given a camera position and a projection plane, the emitted ray is obtained by connecting the camera position to the center of each pixel on the projection plane. It is determined whether the emitted ray intersects with the voxel field of the hair. If it intersects, it proceeds along the ray direction with a certain step size from the intersection point of the ray and the voxel hair bounding box until it encounters the first voxel with a valid occupancy value. The occupancy value and direction value of this voxel are written into the corresponding projection image (in this embodiment, the projection plane is defined as an image of size 400x320). The advantage of this invention is that it projects the 3D direction of the hair strands stored in the voxel into a 2D direction, maintaining consistency with the physical meaning of the input direction map. In particular, since hair can cause occlusion, only the stored value of the first valid voxel intersecting with the ray is used to calculate the shading of the projection map. The formula is expressed as: I = Render(Vox, M).

[0124] Where I represents the rendered image, Vox represents the 3D voxel hair output by the trained voxel decoder, and M represents the projection viewpoint parameters (including the rotation matrix R and translation vector T). The projection process is as follows: Figure 8 As shown.

[0125] The transformation module and discriminator (second discriminator) are components used for supervised learning. The learnable transformation module T consists of 2D convolutional layers, dimension-dilation layers, and 3D convolutional layers, used to transform the input 2D image into features consistent with the 3D voxel hair dimension. The input of T is a hair orientation map obtained by preprocessing a rendered image (the rendered image contains a corresponding 3D voxel hair model), and the output is a tensor consistent with the 3D voxel hair dimension. The discriminator consists of 3D convolutional layers. The tensor formed by concatenating the tensor output by T with the 3D voxel hair output by the voxel decoder is the input of the discriminator. The output of the discriminator is a scalar between 0 and 1, evaluating the degree of matching between the input image and the 3D voxel hair.

[0126] When each rendered image in a set of rendered images with text descriptions has a 3D ground truth value:

[0127] For images with 3D ground truth (3D ground truth refers to the 3D voxel hair model corresponding to the image), see... Figure 7The input (a) is a rendered image obtained from samples in the 3D hair dataset. This image has a corresponding 3D hair model and can be directly used for supervised learning. Supervised learning is then performed directly on the image samples, calculating the loss between the reconstructed voxels and the ground truth voxels; that is, for image samples with 3D ground truth voxels, the loss between the reconstructed voxels and the ground truth voxels is directly calculated. The second loss function is as follows:

[0128]

[0129]

[0130]

[0131]

[0132]

[0133]

[0134] Among them, L occ_3d L is the loss function of the 3D voxel occupancy field during reconstruction. ori_3d L is the loss function of the 3D voxel orientation field during reconstruction. g These three terms are the loss functions of the generator (which consists of the image encoder (backbone feature extraction network) and the trained voxel decoder in the reconstruction network), and the reconstruction network's loss function. d It is the loss function of the second discriminator. i P represents the truth value of the field occupied by the i-th voxel during reconstruction. i This represents the true value of the orientation field of the i-th voxel during reconstruction. This represents the value of the occupancy field of the i-th voxel output by the network during reconstruction. Let F be the value of the orientation field of the i-th voxel output by the network during reconstruction. F represents the true orientation field of the 3D voxel hair during reconstruction. This represents the orientation field of the 3D voxel hair output by the network during reconstruction. N is the total number of voxels involved in the calculation, and ||1|| represents the L1 norm. E denotes the mean function, and D is the discriminator network after spectral normalization. Specifically, this invention introduces a learnable Transform module T. The T module consists of 2D convolutional layers, dimension-dilation layers, and 3D convolutional layers, used to transform the input image I into features with the same dimensions as the orientation field F tensor, and aggregate them as input to the discriminator. The introduction of T further strengthens the connection between the input image and the corresponding 3D hair, improving the reconstruction details.

[0135] When each rendered image in a set with text descriptions lacks a 3D ground truth value:

[0136] For image samples without 3D ground truth, self-supervised learning is performed. The reconstructed voxel hair is input into the differentiable volume renderer and rendered according to the viewpoint obtained in step 102. The loss function is calculated with the input image.

[0137] For image samples without 3D ground truth, the reconstructed voxel hair is input into a differentiable volume renderer, rendered according to the projection viewpoint parameters, and outputs a hair mask and hair orientation map. The output hair segmentation mask and hair orientation map are then compared with the input image to calculate a third loss function. The third loss function is as follows:

[0138] L 2d =α1||I occ -I' occ ||2+α2||I ori -I' ori ||2+α3L style .

[0139]

[0140] Among them, L 2d L is the total loss function for calculating the error between two-dimensional images. style This is a loss function that calculates the style difference between the hair direction map of the input RGB portrait and the hair direction map output by the differentiable renderer. α1 is a manually set hyperparameter used to adjust the weight of the first term. α2 is a manually set hyperparameter used to adjust the weight of the second term. α3 is a manually set hyperparameter used to adjust the weight of the third term. occ For the hair segmentation mask of the input RGB portrait, I ori This is a hair direction map for an input RGB portrait. occ To obtain the hair segmentation mask from the reconstructed 3D voxel hair using a differentiable renderer, I' ori The hair orientation map is obtained by using a differentiable volume renderer to reconstruct the 3D voxel hair. l For I ori The feature map f' output by the l-th layer of the pre-trained VGG network is input into the network. l For I' ori The input is the feature map output from the l-th layer of the pre-trained VGG network. G represents the Gram matrix used to compute the feature map, and C... l H l W l f l The number of channels, height, and width (f) l with f' l(Dimensionality consistent). B represents the total number of network layers involved in style loss calculation, and ||||2 represents the L2 norm. The advantage of this invention lies in introducing the error between hair orientation maps, a common method in image style transfer, to calculate style loss. Style loss calculates the difference in grayscale distribution between two orientation maps, is insensitive to location, and can better capture details. Specifically, when calculating style loss, the orientation map can be divided into blocks, and the style loss of each small block can be calculated, which can better help the network learn local details.

[0141] As an optional implementation method, the method for determining the three-dimensional hair training set includes:

[0142] Obtain the initial 3D hair set; the 3D hair set includes multiple sets of initial hair point clouds.

[0143] Each initial hair point cloud is converted into an initial hair voxel, and the occupancy value and hair orientation value of each initial hair voxel are determined to determine the 3D hair training set.

[0144] Specifically, such as Figure 9 As shown, each hair strand is composed of a series of points in three-dimensional space. Therefore, the initial three-dimensional hair set includes multiple sets of initial hair point clouds.

[0145] A voxel, similar to a pixel in an image, is a cube in three-dimensional space. Converting hair strand data (initial hair point cloud) into voxel data (initial hair voxels) allows for learning hairstyle features through 3D convolution. Specifically, the conversion involves: statistically analyzing the range of x, y, and z coordinates in the initial 3D hair set to obtain a 3D hair bounding box; discretizing this bounding box into a voxel space with a resolution of 128×192×128; setting the occupancy value of voxels without hair strands passing through them to 0; and setting the occupancy value of voxels with hair strands passing through them to 1, with the direction value set to the average of all hair growth directions passing through that voxel.

[0146] Example 2

[0147] The three-dimensional hair reconstruction system in this embodiment includes:

[0148] The module for acquiring RGB portraits to be reconstructed is used to acquire RGB portraits to be reconstructed.

[0149] The target 3D voxel hair determination module is used to input the RGB human image to be reconstructed into the 3D voxel hair reconstruction model to obtain the target 3D voxel hair. The 3D voxel hair reconstruction model is trained on the reconstruction network using the training human image set, the 3D hair training set, and the text corresponding to the 3D hair training set. The reconstruction network includes an image encoder and a trained voxel decoder. The trained voxel decoder is determined based on the trained VAE-Patch_SNGAN network. The trained VAE-Patch_SNGAN network is trained on the VAE-Patch_SNGAN network using the 3D hair training set. The VAE-Patch_SNGAN network includes a 3D voxel generator and a first discriminator. The 3D voxel generator includes a voxel encoder and a voxel decoder.

[0150] The target 3D voxel hair determination module is used to reconstruct a target 3D hair image based on the target 3D voxel hair.

[0151] Example 3

[0152] An electronic device, comprising:

[0153] One or more processors.

[0154] A storage device on which one or more programs are stored.

[0155] When one or more programs are executed by one or more processors, the one or more processors implement the three-dimensional hair reconstruction method as described in Example 1.

[0156] Example 4

[0157] A storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements a three-dimensional hair reconstruction method as described in Example 1.

[0158] Technical effects of the present invention:

[0159] (1) To address the issue of insufficient data, this invention introduces self-supervised learning. Corresponding to the differentiable volume renderer part of this invention, this module projects the 3D voxel hair reconstructed by the network into a 2D image and directly calculates the error with the input 2D image, thus enabling learning even without corresponding 3D hair ground truth. Therefore, by introducing this module, the network can be trained using a large number of natural images even without corresponding 3D hair ground truth.

[0160] (2) To address the issue of unrealistic rendering of images, this invention uses an image generation model to process the rendered images during the construction of the supervised dataset, adding realistic lighting to the rendered 2D images to make them closer to the visual effect of natural images. Compared to existing deep learning-based single-view reconstruction methods, this invention improves the realism of the rendered virtual dataset and alleviates the difference between real images during network inference and virtual images used during training.

[0161] (3) The VAE-Patch_SNGAN network introduced in this invention can improve the details of the reconstructed hair. The discriminator of the original Generative Adversarial Network (GAN) only outputs a scalar between 0 and 1 to evaluate the overall quality of the generated hair. The patch-based GAN (patchgan) outputs an evaluation matrix, in which each element is a local evaluation value of the generated hair. This makes the generator pay more attention to the quality of the local hair details during training.

[0162] (4) Compared with traditional multi-view reconstruction methods, this invention is user-friendly, requiring only a single frontal RGB portrait to reconstruct 3D hair, resulting in better modeling quality. Furthermore, it has a fast inference speed, with modeling time for a single hair taking only a few seconds, making it highly efficient.

[0163] (5) This invention combines the advantages of supervised learning (clear input-output mapping relationship) and self-supervised learning (abundant data, no need for labeled data), and can quickly reconstruct the corresponding 3D hair using a single RGB portrait. It can be applied to digital human body scenes to accurately and efficiently model 3D hairstyles.

[0164] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0165] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for reconstructing three-dimensional hair, characterized in that, The method includes: Obtain the RGB portrait to be reconstructed; The RGB human image to be reconstructed is input into a 3D voxel hair reconstruction model to obtain the target 3D voxel hair. The 3D voxel hair reconstruction model is trained on a reconstruction network using a training set of human images, a 3D hair training set, and the text corresponding to the 3D hair training set. The reconstruction network includes an image encoder and a trained voxel decoder. The trained voxel decoder is determined based on a trained VAE-Patch_SNGAN network, which is trained using the 3D hair training set. The VAE-Patch_SNGAN network includes a 3D voxel generator and a first discriminator. The 3D voxel generator includes a voxel encoder and a voxel decoder. Reconstruct a three-dimensional image of the target hair based on the target three-dimensional voxel hair; The training process includes: Determine the training image set, the 3D hair training set, and the text corresponding to the 3D hair training set; the training image set includes multiple RGB images for training, and the 3D hair training set includes the occupancy ground truth and orientation ground truth corresponding to multiple initial hair voxels; Determine the hair segmentation mask and hair orientation map for each of the aforementioned RGB portrait images used for training; Based on the training image set and the 3D hair training set, the projection viewpoint parameters are determined; the projection viewpoint parameters include: rotation matrix and translation vector; The VAE-Patch_SNGAN network is trained based on the three-dimensional hair training set to obtain the trained VAE-Patch_SNGAN network. A reconstruction network is constructed based on the image encoder and the trained voxel decoder. The three-dimensional hair training set is rendered to obtain a set of rendered images; Based on the text corresponding to the rendered image set and the 3D hair training set, determine the rendered image set with text descriptions; Based on the projection viewpoint parameters, the hair segmentation mask and hair orientation map of the RGB human image used for training, and the set of rendered images with text descriptions, the reconstruction network is trained to obtain the three-dimensional voxel hair reconstruction model. The VAE-Patch_SNGAN network is trained based on the aforementioned 3D hair training set to obtain a trained VAE-Patch_SNGAN network, specifically including: Based on the three-dimensional voxel generator and the first discriminator, the VAE-Patch_SNGAN network is constructed; All the initial hair voxels are input into the VAE-Patch_SNGAN network to obtain the reconstructed hair voxels corresponding to each initial hair voxel; Determine the occupancy and orientation values ​​of each of the reconstructed hair matrix elements; Based on the occupancy and orientation values ​​of each reconstructed hair cell and the occupancy and orientation values ​​of each initial hair cell, the first loss function of the VAE-Patch_SNGAN network is constructed. The VAE-Patch_SNGAN network is fine-tuned based on the first loss function to obtain the trained VAE-Patch_SNGAN network.

2. The method for reconstructing three-dimensional hair according to claim 1, characterized in that, Based on the training image set and the 3D hair training set, the projection viewpoint parameters are determined, specifically including: Identify multiple 2D facial key points in the RGB portrait used for training; Based on all the 2D and 3D facial key points, the projection viewpoint parameters are determined; the 3D facial key points are the 3D facial key points of a standard human body that are aligned with the hair strands corresponding to each hair voxel in the 3D hair training set.

3. The method for reconstructing three-dimensional hair according to claim 1, characterized in that, Based on the projection viewpoint parameters, the hair segmentation mask and hair orientation map of the RGB portrait used for training, and the set of rendered images with text descriptions, the reconstruction network is trained to obtain the three-dimensional voxel hair reconstruction model, specifically including: The hair segmentation mask and hair orientation map of the RGB human image used for training are input into the reconstruction network to obtain multiple reconstructed three-dimensional voxel hairs; Determine the occupancy and orientation values ​​of each of the reconstructed 3D voxel hairs; Determine whether each rendered image with text description in the set of rendered images with text description has a three-dimensional ground truth value; If so, then based on the occupancy ground truth and orientation ground truth corresponding to the rendered image with text description and the occupancy and orientation values ​​of each reconstructed 3D voxel hair, a second loss function of the reconstruction network is constructed, and the reconstruction network is fine-tuned based on the second loss function to obtain the 3D voxel hair reconstruction model. If not, the reconstructed 3D voxel hair corresponding to the rendered image with text description but no 3D ground truth is input into the differentiable volume renderer, and rendered based on the projection viewpoint parameters to obtain the rendered hair segmentation mask and hair orientation map. The third loss function of the reconstruction network is constructed based on the hair segmentation mask and hair orientation map of the training RGB portrait and the rendered hair segmentation mask and hair orientation map. The reconstruction network is then fine-tuned based on the third loss function to obtain the 3D voxel hair reconstruction model.

4. The method for reconstructing three-dimensional hair according to claim 1, characterized in that, The method for determining the three-dimensional hair training set includes: Obtain an initial 3D hair set; the 3D hair set includes multiple sets of initial hair point clouds; Each initial hair point cloud is converted into an initial hair voxel, and the occupancy value and hair orientation value of each initial hair voxel are determined, thereby determining the three-dimensional hair training set.

5. A three-dimensional hair reconstruction system, characterized in that, The system includes: The module for acquiring RGB portraits to be reconstructed is used to acquire RGB portraits to be reconstructed. A target 3D voxel hair determination module is used to input the RGB human image to be reconstructed into a 3D voxel hair reconstruction model to obtain the target 3D voxel hair. The 3D voxel hair reconstruction model is trained on a reconstruction network using a training set of human images, a 3D hair training set, and text corresponding to the 3D hair training set. The reconstruction network includes an image encoder and a trained voxel decoder. The trained voxel decoder is determined based on a trained VAE-Patch_SNGAN network. The trained VAE-Patch_SNGAN network is trained using the 3D hair training set. The VAE-Patch_SNGAN network includes a 3D voxel generator and a first discriminator. The 3D voxel generator includes a voxel encoder and a voxel decoder. A target 3D voxel hair determination module is used to reconstruct a target 3D hair image based on the target 3D voxel hair; The training process includes: Determine the training image set, the 3D hair training set, and the text corresponding to the 3D hair training set; the training image set includes multiple RGB images for training, and the 3D hair training set includes the occupancy ground truth and orientation ground truth corresponding to multiple initial hair voxels; Determine the hair segmentation mask and hair orientation map for each of the aforementioned RGB portrait images used for training; Based on the training image set and the 3D hair training set, the projection viewpoint parameters are determined; the projection viewpoint parameters include: rotation matrix and translation vector; The VAE-Patch_SNGAN network is trained based on the three-dimensional hair training set to obtain the trained VAE-Patch_SNGAN network. A reconstruction network is constructed based on the image encoder and the trained voxel decoder. The three-dimensional hair training set is rendered to obtain a set of rendered images; Based on the text corresponding to the rendered image set and the 3D hair training set, determine the rendered image set with text descriptions; Based on the projection viewpoint parameters, the hair segmentation mask and hair orientation map of the RGB human image used for training, and the set of rendered images with text descriptions, the reconstruction network is trained to obtain the three-dimensional voxel hair reconstruction model. The VAE-Patch_SNGAN network is trained based on the aforementioned 3D hair training set to obtain a trained VAE-Patch_SNGAN network, specifically including: Based on the three-dimensional voxel generator and the first discriminator, the VAE-Patch_SNGAN network is constructed; All the initial hair voxels are input into the VAE-Patch_SNGAN network to obtain the reconstructed hair voxels corresponding to each initial hair voxel; Determine the occupancy and orientation values ​​of each of the reconstructed hair matrix elements; Based on the occupancy and orientation values ​​of each reconstructed hair cell and the occupancy and orientation values ​​of each initial hair cell, the first loss function of the VAE-Patch_SNGAN network is constructed. The VAE-Patch_SNGAN network is fine-tuned based on the first loss function to obtain the trained VAE-Patch_SNGAN network.

6. An electronic device, characterized in that, include: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the three-dimensional hair reconstruction method as described in any one of claims 1 to 4.

7. A storage medium, characterized in that, It stores a computer program, wherein the computer program, when executed by a processor, implements the three-dimensional hair reconstruction method as described in any one of claims 1 to 4.

Images