Single-view three-dimensional human head reconstruction method and system

Abstract

This application discloses a single-view 3D head reconstruction method and system, belonging to the fields of computer vision and 3D reconstruction technology. The method includes: acquiring a single RGB portrait image and preprocessing the RGB portrait image to obtain a preprocessed portrait image; performing 3D reconstruction of the preprocessed portrait image using a 3D-aware multi-view diffusion generation network to obtain a target 3D head model; wherein the 3D-aware multi-view diffusion generation network includes a VAE encoder, a CLIP encoder, a facial keypoint extractor, an improved denoising UNet network, and a VAE decoder. This application deeply integrates facial keypoint attention and conditional diffusion generation, enabling the single-view reconstruction process to have inherent structural priors, thereby significantly improving the realism and consistency of the 3D head reconstruction.

Description

Technical Field

[0001] This application belongs to the field of computer vision and 3D reconstruction technology, specifically relating to a single-view 3D head reconstruction method and system. Background Technology

[0002] 3D head reconstruction is a core research area at the intersection of computer vision and graphics. Its goal is to recover a 3D head model with high geometric accuracy and realism from one or more 2D portrait images. High-quality 3D head models play an irreplaceable role in virtual reality, augmented reality, digital humans, telepresence, film and television production, and gaming. However, due to the complexity of facial structure, rich variations in appearance, frequent occlusion, and the significant loss of geometric information inherent in single-view input, achieving high-fidelity, high-consistency, end-to-end 3D head reconstruction from a single portrait image remains a challenging problem that academia and industry urgently need to overcome.

[0003] Currently, existing single-view 3D head reconstruction methods generally suffer from defects such as missing reconstruction details and lack of 3D geometric perception, resulting in problems such as poor consistency and low fidelity of reconstruction results. Summary of the Invention

[0004] The purpose of this application is to provide a single-view 3D head reconstruction method and system that can solve the technical problems of low fidelity and poor consistency in existing single-view 3D head reconstruction methods.

[0005] To solve the above-mentioned technical problems, this application is implemented as follows:

[0006] In a first aspect, embodiments of this application provide a single-view three-dimensional head reconstruction method, the method comprising:

[0007] A single RGB portrait image is acquired, and the RGB portrait image is preprocessed to obtain a preprocessed portrait image;

[0008] The preprocessed portrait image is reconstructed in three dimensions using a three-dimensional perception multi-view diffusion generation network to obtain a target three-dimensional human head model.

[0009] The 3D perception multi-view diffusion generation network includes a VAE encoder, a CLIP encoder, a face key point extractor, an improved denoising UNet network, and a VAE decoder.

[0010] As an optional implementation of the first aspect of this application, the three-dimensional perceptual multi-view diffusion generation network performs three-dimensional reconstruction on the preprocessed portrait image to obtain a target three-dimensional human head model, specifically:

[0011] The preprocessed portrait image is subjected to key point feature extraction using a facial key point extractor to obtain a key point structure attention map.

[0012] The preprocessed portrait image is subjected to feature encoding processing according to the CLIP encoder to obtain global semantic features;

[0013] The keypoint structure attention map is fused with the global semantic features to obtain fused conditional features;

[0014] The preprocessed portrait image is subjected to feature encoding processing based on the VAE encoder to obtain potential compressed features;

[0015] The potential compression features and the fusion condition features are processed using the improved denoising UNet network to obtain multi-view features;

[0016] The VAE decoder performs feature decoding processing on the multi-view features to obtain consistent images from multiple different viewpoints;

[0017] Based on the consistent images from multiple different perspectives, a three-dimensional geometric reconstruction of the human head is performed to obtain a target three-dimensional human head model;

[0018] The improved denoising UNet network is obtained by incorporating a key point attention modulation module into the cross-attention layer of the basic denoising UNet network.

[0019] As an optional implementation of the first aspect of this application, the step of extracting key point features from the preprocessed portrait image using a facial key point extractor to obtain a key point structure attention map specifically involves:

[0020] Keypoint detection is performed on the preprocessed portrait image using the 3DDFA_V2 model, and the three-dimensional coordinates of each detected keypoint are extracted. ,in Represents depth information;

[0021] A Gaussian distribution heatmap corresponding to each key point is generated based on the three-dimensional coordinates of each key point, and the standard deviation of the Gaussian kernel of each Gaussian distribution heatmap is adaptively adjusted according to the depth information of the corresponding key point.

[0022] The Gaussian distribution heatmaps corresponding to all the key points are superimposed and normalized to obtain the key point structure attention map.

[0023] As an optional implementation of the first aspect of this application, the step of fusing the keypoint structure attention map with the global semantic features to obtain fused conditional features specifically includes:

[0024] The key point structure attention map is mapped using a learnable feature mapping network to obtain a mapping feature that matches the global semantic feature dimension.

[0025] The mapping features are subjected to attention weighting processing based on the channel attention mechanism to obtain weighted mapping features;

[0026] The weighted mapping feature is multiplied element-wise with the global semantic feature to obtain the fusion condition feature;

[0027] The learnable feature mapping network includes two convolutional layers and an activation function.

[0028] As an optional implementation of the first aspect of this application, the key point attention modulation module uses the fusion conditional features as modulation signals to perform gated modulation operations on the query matrix and key matrix in the cross-attention layer, respectively obtaining the gated-modulated query matrix and the gated-modulated key matrix; then, similarity calculation is performed based on the gated-modulated query matrix and the gated-modulated key matrix to obtain the modulated attention weights; then, the modulated attention weights are multiplied by the value matrix in the cross-attention layer to obtain the output features of the cross-attention layer.

[0029] The specific calculation formula is as follows:

[0030] ,

[0031] in, This represents the query matrix in the cross-attention layer. This represents the key matrix in the cross-attention layer. This indicates element-wise multiplication. This represents the modulation intensity coefficients obtained through training. Indicates the characteristics of fusion conditions. This represents the query matrix after gated modulation. Represents the gated modulation key matrix. The output features of the cross-attention layer are represented by softmax, which represents the activation function. This represents the value matrix in the cross-attention layer. express The transpose of the matrix, express Dimensions.

[0032] As an optional implementation of the first aspect of this application, the step of performing three-dimensional geometric reconstruction of the human head based on the consistent images from multiple different perspectives to obtain a target three-dimensional human head model specifically includes:

[0033] The neural radiation field is used to fit the consistent images from multiple different perspectives to establish a three-dimensional space containing density and color fields.

[0034] The three-dimensional space containing the density field and color field is rendered using volumetric rendering technology to obtain reconstructed three-dimensional volume data.

[0035] The three-dimensional mesh is extracted from the reconstructed three-dimensional volume data using the differentiable moving cube algorithm to obtain the target three-dimensional human head model.

[0036] As an optional implementation of the first aspect of this application, the training process of the 3D-aware multi-view diffusion generation network includes a first training stage and a second training stage. The first training stage is used to train the generation of consistent images from multiple different perspectives based on preprocessed portrait images. The first training stage uses perceptual loss and regression loss for loss calculation. The second training stage is used for end-to-end training of the 3D-aware multi-view diffusion generation network. The second training stage uses regression loss and normal loss for loss calculation. The normal loss calculation process is as follows:

[0037] The pre-trained normal estimation network extracts normals from the current training images in the training set to obtain the true normal map.

[0038] The 3D perception multi-view diffusion generation network processes the current training image to obtain a training 3D head model, and performs multiple random view renderings on the training 3D head model according to the differentiable renderer to obtain a predicted normal map.

[0039] The loss between the true normal map and the predicted normal map is calculated based on the normal loss function to obtain the normal loss value.

[0040] Secondly, embodiments of this application provide a single-view 3D head reconstruction system, the system comprising:

[0041] Preprocessing module: acquires a single RGB portrait image and preprocesses the RGB portrait image to obtain a preprocessed portrait image;

[0042] 3D Reconstruction Module: Based on a 3D perception multi-view diffusion generation network, the preprocessed portrait image is reconstructed in 3D to obtain a target 3D human head model;

[0043] The 3D perception multi-view diffusion generation network includes a VAE encoder, a CLIP encoder, a face key point extractor, an improved denoising UNet network, and a VAE decoder.

[0044] Thirdly, embodiments of this application provide an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0045] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0046] In the embodiments of this application, compared with the prior art, the following technical effects are achieved:

[0047] This application integrates facial keypoint attention and conditional diffusion generation to enable the single-view reconstruction process to have inherent structural priors: keypoints generate keypoint structure attention maps based on their 3D geometric positions, guiding the denoising network pixel by pixel to focus on stable feature regions such as the corners of the eyes, the tip of the nose, and the corners of the mouth, thus suppressing identity drift and geometric distortion from the source; regardless of whether it is a frontal view or a large-angle side view, the generated image is strictly aligned with the input in terms of facial proportions and facial feature localization, completely alleviating the inherent problem of traditional diffusion models that are realistic in appearance but misaligned in structure, thereby significantly improving the realism and consistency of 3D head reconstruction. In addition, by introducing a two-stage collaborative optimization as a decoupling mechanism, the first stage uses key point-global semantic fusion features as a condition to dominate the high-noise region, ensuring geometric consistency and head contour integrity from multiple perspectives; the second stage uses normal loss as a regularization to dominate the low-noise region, focusing on skin micro-texture and hair details; the two stages share CLIP identity embedding as a cross-view signature to achieve identity-structure-texture separation control, thereby locking the core facial features even in continuous perspectives, exaggerated expressions, or scenes with accessories, avoiding side collapse and detail blurring, and supporting one-time forward inference to output high-fidelity 3D heads from any angle in real time. Attached Figure Description

[0048] Figure 1 This is a flowchart of a single-view three-dimensional head reconstruction method provided by some embodiments of this application;

[0049] Figure 2 This is a schematic diagram of the structure of a multi-view diffusion generation network for 3D perception in a single-view 3D head reconstruction method provided by some embodiments of this application. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0052] The following description, in conjunction with the accompanying drawings, details a single-view three-dimensional head reconstruction method and system provided in this application through specific embodiments and application scenarios.

[0053] A single-view 3D head reconstruction method includes:

[0054] S100: Acquire a single RGB portrait image and preprocess the RGB portrait image to obtain a preprocessed portrait image;

[0055] It should be noted that a single RGB portrait image is obtained, scaled to a resolution of 512×512, and the pixel values ​​are normalized to the range of [0,1] to obtain a preprocessed portrait image.

[0056] S200: Based on the 3D perception multi-view diffusion generation network, the preprocessed portrait image is reconstructed in 3D to obtain the target 3D human head model;

[0057] The S200's 3D perception multi-view diffusion generation network includes a VAE encoder, a CLIP encoder, a face key point extractor, a denoising UNet network, and a VAE decoder.

[0058] As an optional implementation of this embodiment, S200 is specifically as follows:

[0059] S210: Extract key point features from the preprocessed portrait image using the face key point extractor to obtain a key point structure attention map;

[0060] S220: Perform feature encoding on the preprocessed portrait image according to the CLIP encoder to obtain global semantic features;

[0061] S230: The key point structure attention map is fused with the global semantic features to obtain the fused conditional features;

[0062] S240: Perform feature encoding on the preprocessed portrait image according to the VAE encoder to obtain potential compressed features;

[0063] S250: The potential compressed features and fusion conditional features are processed by the improved denoising UNet network to obtain multi-view features;

[0064] S260: Based on the VAE decoder, feature decoding processing is performed on the multi-view features to obtain consistent images from multiple different viewpoints;

[0065] S270: Perform three-dimensional geometric reconstruction of the human head based on consistent images from multiple different perspectives to obtain a target three-dimensional human head model;

[0066] The improved denoising UNet network in S250 is obtained by incorporating a key point attention modulation module into the cross-attention layer of the basic denoising UNet network.

[0067] It should be noted that the facial landmark extractor uses the 3DDFA_V2 model to extract features from 68 3D facial landmarks in the preprocessed portrait image, obtaining a landmark structure attention map. A CLIP encoder is used to encode features in the preprocessed portrait image to extract global semantic features; the CLIP encoder uses a CLIP-ViT / B-16 visual encoder. The landmark structure attention map and global semantic features are fused to obtain fusion conditional features. Then, the preprocessed portrait image is encoded by a VAE encoder to compress it into the latent space, thus obtaining latent compressed features. Next, the latent compressed features and fusion conditional features are processed by an improved denoising UNet network to obtain multi-view features. The improved denoising UNet network is obtained by incorporating a landmark attention modulation module into the cross-attention layer of the original basic denoising UNet network. Then, the multi-view features are decoded by a VAE decoder to obtain 8 consistent images from different perspectives. Finally, the 3D geometry of the head is reconstructed from the 8 consistent images from different perspectives to obtain the target 3D head model.

[0068] As a specific implementation of this embodiment, the key point structure attention map is fused with global semantic features to obtain fused conditional features, which are expressed by the following formula:

[0069] ,

[0070] in, Indicates the characteristics of fusion conditions. This represents a key-point structure attention map. Represents a learnable mapping matrix. Represents global semantic features;

[0071] As an optional implementation of this embodiment, in S210, key point features are extracted from the preprocessed portrait image using a facial key point extractor to obtain a key point structure attention map, specifically as follows:

[0072] S211: Perform keypoint detection on the preprocessed portrait image based on the 3DDFA_V2 model, and extract the 3D coordinates of each detected keypoint. ,in Represents depth information;

[0073] S212: Generate a Gaussian distribution heatmap corresponding to each key point based on the three-dimensional coordinates of each key point. The standard deviation of the Gaussian kernel of each Gaussian distribution heatmap is adaptively adjusted according to the depth information of the corresponding key point.

[0074] S213: Overlay the Gaussian distribution heatmaps corresponding to all key points and normalize them to obtain the key point structure attention map.

[0075] It should be noted that the 3DDFA_V2 model is used to detect key points in the preprocessed portrait image, and the three-dimensional coordinates of each detected key point are extracted. ,in The system represents depth information; it generates a Gaussian distribution heatmap for each keypoint based on the 3D coordinates of each keypoint, and the standard deviation of the Gaussian kernel of each Gaussian distribution heatmap is adaptively adjusted according to the depth information of the corresponding keypoint. The greater the depth, the smaller the standard deviation of the Gaussian kernel. Then, the Gaussian distribution heatmaps of all 68 keypoints are superimposed in the spatial dimension, and the salient features are retained through max pooling operation. Finally, normalization is performed to form the final keypoint structure attention map.

[0076] As one specific implementation method of this embodiment, the Gaussian kernel standard deviation is adaptively adjusted based on the depth information of the corresponding key points, as expressed by the following formula.

[0077] ,

[0078] in, This represents the Gaussian kernel standard deviation. Indicates the basic standard deviation. This represents the depth adjustment coefficient. It represents depth information.

[0079] As an optional implementation of this embodiment, in S230, the key point structure attention map is fused with the global semantic features to obtain fused conditional features, specifically as follows:

[0080] S231: Map the key point structure attention map according to the learnable feature mapping network to obtain the mapping features that match the global semantic feature dimension;

[0081] S232: Perform attention weighting on the mapped features according to the channel attention mechanism to obtain weighted mapped features;

[0082] S233: Multiply the weighted mapping features element-wise with the global semantic features to obtain the fused conditional features;

[0083] The learnable feature mapping network in S231 includes two convolutional layers and an activation function.

[0084] It should be noted that the learnable feature mapping network maps the keypoint structure attention map through two convolutional layers and an activation function, mapping the keypoint structure attention map to the same dimension as the global semantic features, thus obtaining the mapped features. The channel attention mechanism then calculates the channel importance of the mapped features to obtain the corresponding channel weights, and then multiplies the channel weights with the elements of the mapped features to obtain the weighted mapping features. The channel attention mechanism adopts the attention module in SENet. Finally, the weighted mapping features are multiplied element-wise with the global semantic features to obtain the fusion conditional features, resulting in the structure-enhanced fusion conditional features.

[0085] As an optional implementation of this embodiment, the key point attention modulation module in the improved denoising UNet network of S250 uses the fused conditional features as the modulation signal to perform gated modulation operations on the query matrix and key matrix in the cross-attention layer, respectively, to obtain the gated modulated query matrix and the gated modulated key matrix; then, similarity calculation is performed based on the gated query matrix and the gated modulated key matrix to obtain the modulated attention weights; then, the modulated attention weights are multiplied by the value matrix in the cross-attention layer to obtain the output features of the cross-attention layer;

[0086] The specific calculation formula is as follows:

[0087] ,

[0088] in, This represents the query matrix in the cross-attention layer. This represents the key matrix in the cross-attention layer. This indicates element-wise multiplication. This represents the modulation intensity coefficients obtained through training. Indicates the characteristics of fusion conditions. This represents the query matrix after gated modulation. Represents the gated modulation key matrix. The output features of the cross-attention layer are represented by softmax, which represents the activation function. This represents the value matrix in the cross-attention layer. express The transpose of the matrix, express Dimensions.

[0089] As an optional implementation of this embodiment, in S270, a three-dimensional geometric reconstruction of the human head is performed based on consistent images from multiple different viewpoints to obtain a target three-dimensional human head model, specifically as follows:

[0090] S271: Fit the consistent images from multiple different perspectives based on the neural radiation field to establish a three-dimensional space containing density field and color field.

[0091] S272: Render the three-dimensional space containing the density field and color field according to the volume rendering technology to obtain the reconstructed three-dimensional volume data.

[0092] S273: Based on the differentiable moving cube algorithm, the 3D volume data of the reconstructed viewpoint is used to extract a 3D mesh to obtain the target 3D human head model.

[0093] It should be noted that Neural Radiation Field (NeRF) was used to fit consistent images from eight different viewpoints to establish a density field and color field in three-dimensional space. Volumetric rendering technology was used to synthesize reconstructed three-dimensional volume data using ray marching. Then, the DMTet (Differential Moving Cube) algorithm was used to extract an explicit three-dimensional mesh (target three-dimensional human head model) from the reconstructed three-dimensional volume data. The DMTet algorithm first establishes a signed distance field (SDF) in three-dimensional space, then extracts triangular meshes using a differentiable marching tetrahedra algorithm, and finally optimizes the vertex positions and face normal positions of the triangular meshes using a differentiable renderer to obtain the target three-dimensional human head model.

[0094] As an optional implementation of this embodiment, the training process of the 3D perception multi-view diffusion generation network in S200 includes a first training stage and a second training stage. The first training stage is used to train the generation of consistent images from multiple different perspectives based on the preprocessed portrait image. The first training stage uses perceptual loss and regression loss for loss calculation. The second training stage is used for end-to-end training of the 3D perception multi-view diffusion generation network. The second training stage uses regression loss and normal loss for loss calculation. The normal loss calculation process is as follows:

[0095] The pre-trained normal estimation network extracts normals from the current training images in the training set to obtain the true normal map.

[0096] A 3D-perception multi-view diffusion generation network processes the current training image to obtain a training 3D head model. The training 3D head model is then rendered from multiple random viewpoints using a differentiable renderer to obtain a predicted normal map.

[0097] The loss between the true normal map and the predicted normal map is calculated based on the normal loss function, and the normal loss value is obtained.

[0098] It should be noted that the training set for the 3D perception multi-view diffusion generation network uses the publicly available Cafca dataset, which contains 1,500 synthetic objects, each rendered in three different environments, randomly presented with 13 expressions from 30 viewpoints. For evaluation, 40 objects were selected, each containing 30 test camera poses, and the most frontal view was manually selected as the network input. The training process is divided into two stages. The first stage involves generating consistent images from multiple different perspectives from the pre-processed portrait image. In this stage, the parameters of the VAE encoder, VAE decoder, and CLIP encoder are fixed, and only the improved denoising UNet network is trained. The loss function in this stage is a first composite function of perceptual loss and regression loss. The second stage is the end-to-end training of the 3D perceptual multi-view diffusion generation network. The loss function in this training process is a second composite function of regression loss and normal loss. The normal loss calculation process is as follows: First, the pre-trained normal estimation network extracts normals from the current training image in the training set to obtain the true normal map. Then, the 3D perceptual multi-view diffusion generation network processes the current training image to obtain the trained 3D head model, and renders the trained 3D head model from multiple random perspectives using a differentiable renderer to obtain the predicted normal map. Finally, the loss between the true normal map and the predicted normal map is calculated based on the normal loss function to obtain the normal loss value.

[0099] According to a specific implementation of this embodiment, the first composite loss function and the second composite loss function are expressed by the following formula:

[0100]

[0101]

[0102] ,

[0103] in, Denotes the first composite loss function. This represents the regression loss function in the first stage. Represents the perceptual loss function; The weights represent the perceptual loss function. This represents the output of the VAE decoder when processing the current training image. This represents the current training image in the training set. This represents the second composite loss function. This represents the regression loss function in the second stage. Represents the normal loss function. The weights represent the weights of the normal loss function. This represents the corresponding angle image of the trained 3D human head model obtained by processing the current training image. This represents the predicted normal map of the trained 3D human head model corresponding to the current training image. This represents the true normal map of the current training image.

[0104] According to a single-view 3D head reconstruction method in this embodiment, the single-view reconstruction process has a built-in structural prior by fusing facial key point attention and conditional diffusion generation depth: key points generate key point structure attention maps based on their 3D geometric positions, and guide the denoising network pixel by pixel to focus on stable feature areas such as the corners of the eyes, the tip of the nose, and the corners of the mouth, thereby suppressing identity drift and geometric distortion from the source; regardless of whether it is a frontal view or a large-angle side view, the generated image is strictly aligned with the input in terms of facial proportions and facial feature positioning, completely alleviating the inherent problem of traditional diffusion models that are realistic in appearance but misaligned in structure. In addition, by introducing a two-stage collaborative optimization as a decoupling mechanism, the first stage uses key point-global semantic fusion features as a condition to dominate the high-noise region, ensuring geometric consistency and head contour integrity from multiple perspectives; the second stage uses normal loss as a regularization to dominate the low-noise region, focusing on skin micro-texture and hair details; the two stages share CLIP identity embedding as a cross-view signature to achieve identity-structure-texture separation control, thereby locking the core facial features even in continuous perspectives, exaggerated expressions, or scenes with accessories, avoiding side collapse and detail blurring, and supporting one-time forward inference to output high-fidelity 3D heads from any angle in real time.

[0105] It should be noted that the single-view 3D head reconstruction method provided in this application embodiment can be executed by a single-view 3D head reconstruction system, or a control module within that system for executing and loading a single-view 3D head reconstruction method. This application embodiment uses the execution of a single-view 3D head reconstruction system to load a single-view 3D head reconstruction method as an example to illustrate the single-view 3D head reconstruction method provided in this application embodiment.

[0106] A single-view 3D head reconstruction system, comprising:

[0107] Preprocessing module: Acquires a single RGB portrait image and preprocesses the RGB portrait image to obtain a preprocessed portrait image;

[0108] 3D Reconstruction Module: Based on a 3D perception multi-view diffusion generation network, the preprocessed portrait image is reconstructed to obtain a target 3D human head model;

[0109] The 3D perception multi-view diffusion generation network includes a VAE encoder, a CLIP encoder, a face key point extractor, an improved denoising UNet network, and a VAE decoder.

[0110] The single-view 3D head reconstruction system in this application embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), etc. This application embodiment does not impose specific limitations.

[0111] The single-view 3D head reconstruction system in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0112] The single-view 3D head reconstruction system provided in this application embodiment can achieve... Figures 1 to 2 The various processes and effects of a single-view 3D head reconstruction method implemented in the method embodiment are not described in detail here to avoid repetition.

[0113] Optionally, embodiments of this application also provide an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the various processes of the above-described single-view three-dimensional head reconstruction method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0114] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described single-view three-dimensional head reconstruction method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0115] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0116] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0117] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0118] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A single-view 3D head reconstruction method, characterized in that, The method includes: A single RGB portrait image is acquired, and the RGB portrait image is preprocessed to obtain a preprocessed portrait image; The preprocessed portrait image is reconstructed in three dimensions using a three-dimensional perception multi-view diffusion generation network to obtain a target three-dimensional human head model. The 3D perception multi-view diffusion generation network includes a VAE encoder, a CLIP encoder, a face key point extractor, an improved denoising UNet network, and a VAE decoder. The 3D-perception multi-view diffusion generation network performs 3D reconstruction on the preprocessed portrait image to obtain the target 3D human head model, specifically: The preprocessed portrait image is subjected to key point feature extraction using a facial key point extractor to obtain a key point structure attention map. The preprocessed portrait image is subjected to feature encoding processing according to the CLIP encoder to obtain global semantic features; The keypoint structure attention map is fused with the global semantic features to obtain fused conditional features; The preprocessed portrait image is subjected to feature encoding processing based on the VAE encoder to obtain potential compressed features; The improved denoising UNet network is used to process the potential compression features and the fusion condition features to obtain multi-view features; The VAE decoder performs feature decoding processing on the multi-view features to obtain consistent images from multiple different viewpoints; Based on the consistent images from multiple different perspectives, a three-dimensional geometric reconstruction of the human head is performed to obtain a target three-dimensional human head model; The improved denoising UNet network is obtained by incorporating a key point attention modulation module into the cross-attention layer of the basic denoising UNet network; The keypoint attention modulation module uses the fusion conditional features as modulation signals to perform gated modulation operations on the query matrix and key matrix in the cross-attention layer, obtaining the gated-modulated query matrix and the gated-modulated key matrix, respectively. Then, it calculates the similarity based on the gated-modulated query matrix and the gated-modulated key matrix to obtain the modulated attention weights. Finally, it performs matrix multiplication with the value matrix in the cross-attention layer to obtain the output features of the cross-attention layer. The specific calculation formula is as follows: ， in, This represents the query matrix in the cross-attention layer. This represents the key matrix in the cross-attention layer. This indicates element-wise multiplication. This represents the modulation intensity coefficients obtained through training. Indicates the characteristics of fusion conditions. This represents the query matrix after gated modulation. Represents the gated modulation key matrix. The output features of the cross-attention layer are represented by softmax, which represents the activation function. This represents the value matrix in the cross-attention layer. express The transpose of the matrix, express Dimensions.

2. The single-view three-dimensional head reconstruction method according to claim 1, characterized in that, The step of extracting key point features from the preprocessed portrait image using a facial key point extractor to obtain a key point structure attention map is as follows: Keypoint detection is performed on the preprocessed portrait image using the 3DDFA_V2 model, and the three-dimensional coordinates of each detected keypoint are extracted. , where z represents depth information; A Gaussian distribution heatmap corresponding to each key point is generated based on the three-dimensional coordinates of each key point, and the standard deviation of the Gaussian kernel of each Gaussian distribution heatmap is adjusted according to the depth information of the corresponding key point. The Gaussian distribution heatmaps corresponding to all the key points are superimposed and normalized to obtain the key point structure attention map.

3. The single-view three-dimensional head reconstruction method according to claim 1, characterized in that, The process of fusing the keypoint structure attention map with the global semantic features to obtain fused conditional features is as follows: The key point structure attention map is mapped using a learnable feature mapping network to obtain a mapping feature that matches the global semantic feature dimension. The mapping features are subjected to attention weighting processing based on the channel attention mechanism to obtain weighted mapping features; The weighted mapping feature is multiplied element-wise with the global semantic feature to obtain the fusion condition feature; The learnable feature mapping network includes two convolutional layers and an activation function.

4. The single-view three-dimensional head reconstruction method according to claim 1, characterized in that, The step of performing three-dimensional geometric reconstruction of the human head based on the consistent images from multiple different perspectives to obtain the target three-dimensional human head model is as follows: The neural radiation field is used to fit the consistent images from multiple different perspectives to establish a three-dimensional space containing density and color fields. The three-dimensional space containing the density field and color field is rendered using volumetric rendering technology to obtain reconstructed three-dimensional volume data. The three-dimensional mesh is extracted from the reconstructed three-dimensional volume data using the differentiable moving cube algorithm to obtain the target three-dimensional human head model.

5. The single-view three-dimensional head reconstruction method according to claim 1, characterized in that, The training process of the 3D perception multi-view diffusion generation network includes a first training stage and a second training stage. The first training stage is the training process for generating consistent images from multiple different perspectives based on preprocessed portrait images. The first training stage uses perceptual loss and regression loss for loss calculation. The second training stage is used for end-to-end training of the 3D perception multi-view diffusion generation network. The second training stage uses regression loss and normal loss for loss calculation. The normal loss calculation process is as follows: The pre-trained normal estimation network extracts normals from the current training images in the training set to obtain the true normal map. The 3D perception multi-view diffusion generation network processes the current training image to obtain a training 3D human head model, and performs multiple random view renderings on the training 3D human head model according to the differentiable renderer to obtain a predicted normal map. The loss between the true normal map and the predicted normal map is calculated based on the normal loss function to obtain the normal loss value.

6. A single-view three-dimensional head reconstruction system, used to implement the single-view three-dimensional head reconstruction method according to any one of claims 1-5, characterized in that, The system includes: The preprocessing module is used to acquire a single RGB portrait image and preprocess the RGB portrait image to obtain a preprocessed portrait image. The 3D reconstruction module is used to perform 3D reconstruction on the preprocessed portrait image based on the 3D perception multi-view diffusion generation network to obtain the target 3D human head model. The 3D perception multi-view diffusion generation network includes a VAE encoder, a CLIP encoder, a face key point extractor, an improved denoising UNet network, and a VAE decoder.

7. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein when the program or instructions are executed by the processor, they implement the steps of a single-view three-dimensional head reconstruction method as described in any one of claims 1-5.

8. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of a single-view three-dimensional head reconstruction method as described in any one of claims 1-5.

Images