A head reconstruction method based on collaborative optimization of generation and inverse rendering

Abstract

The application discloses a head reconstruction method based on generation and inverse rendering collaborative optimization, comprising the following steps: step 1, generating a corresponding binary mask; step 2, fitting a three-dimensional geometric grid of a face matched with an input person based on detected two-dimensional face key points to obtain a unified head coordinate system; step 3, projecting pixels of an input image to a UV space of the three-dimensional geometric grid of the face to form a defective texture with original illumination and self-occlusion; performing illumination regularization and an inverse rendering network to output a multi-channel PBR material map; step 4, generating a group of videos rotating around the head and camera internal and external parameters thereof, and cutting the videos into a multi-view hair image sequence and corresponding virtual camera parameters; step 5, performing multi-view hair reconstruction to obtain a hair geometry representation or equivalent geometry representation representing a three-dimensional structure of hair; and step 6, performing spatial alignment and combination to output a head PBR three-dimensional asset.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and computer graphics, and in particular to a method for geometric and PBR (Physically Based Rendering) material reconstruction of human face hair based on a single viewpoint. Background Technology

[0002] With the rapid development of applications such as virtual reality, augmented reality, digital humans, games, and film and television, the demand for ordinary users to quickly generate highly realistic 3D avatars on general-purpose devices is becoming increasingly urgent. Traditional 3D character creation processes rely on professional artists for modeling, sculpting, and texture painting, which is time-consuming and costly, making it difficult to meet the needs of large-scale personalized asset production.

[0003] In face reconstruction, existing technologies can be broadly categorized into multi-view and single-view methods. Multi-view stereo reconstruction methods acquire multi-view images simultaneously or around multiple cameras, utilizing multi-view geometry and volume optimization to achieve high-precision geometry, but they have high requirements for the acquisition environment and equipment. Single-view methods are closer to the usage scenarios of ordinary users, often employing 3D deformable facial geometric meshes and restoring personalized geometry by fitting parameters; recent work has also used deep neural networks to directly regress geometry and materials. However, many single-view methods output "light-baked textures," with severe coupling between lighting and materials, resulting in models lacking realism under new lighting conditions.

[0004] In the areas of inverse rendering and PBR material reconstruction, existing research has proposed decomposing channels such as albedo, roughness, specular highlights, and normals from a single face image to enable the model to relight. These methods typically rely on a mixed training set containing lighting stage data, resulting in complex network structures and training strategies. They also primarily focus on the face region, offering limited support for complex structures such as hair.

[0005] In hair reconstruction, traditional multi-view scanning and single-view reconstruction face significant challenges due to the numerous fine filaments, complex geometry, semi-transparent texture, and anisotropic specular reflection of hair. Multi-view hair reconstruction methods typically require users to shoot videos around themselves or collect data in strictly controlled photographic environments, relying on hair filament priors and multi-view volumetric optimization to recover the hair chain structure. These methods are highly dependent on multi-view data and are difficult to apply directly to scenarios with only a single portrait as input.

[0006] Existing single-view head reconstruction schemes often treat the face and hair as two relatively independent sub-tasks: focusing on geometry and material decomposition for the face, while relying on independent priors or simple completion strategies for the hair, lacking unified physical lighting modeling and coordinate system constraints. Especially in schemes that attempt to use generative models to "synthesize multiple views" to assist hair reconstruction, it is usually necessary to back-infer camera pose and lighting parameters from the generated video, introducing new uncertainties and accumulated errors, which can easily lead to inconsistencies between the reconstructed hair geometry and the face geometry and lighting estimates.

[0007] Therefore, there is an urgent need for a method that can uniformly model the geometry and PBR materials of the face and hair under the condition of only a single portrait input, explicitly estimate and share the head lighting prior, and avoid the problem of multiple solutions caused by solving the camera and lighting again from the synthetic video, so as to realize a stable and reliable single-view head PBR asset reconstruction method; at the same time, it is also necessary to not pre-fix the specific mesh topology in the output stage, so that users can flexibly specify the target topology and perform geometry and material baking according to the actual project needs. Summary of the Invention

[0008] Purpose of the invention: The technical problem to be solved by the present invention is to provide an automated reconstruction method that is robust and can automatically generate high-quality head PBR 3D assets (including face 3D geometry, face PBR material and hair 3D geometry) from a single portrait, overcoming the shortcomings of the existing technology.

[0009] To address the aforementioned technical problems, this invention discloses a head reconstruction method based on collaborative optimization of generation and inverse rendering, comprising the following steps:

[0010] Step 1: Take a single-view RGB portrait as the input image and use an image segmentation network to identify the face and hair regions, generating the corresponding binary mask;

[0011] Step 2: For the face region, fit the detected two-dimensional face key points to the three-dimensional geometric mesh of the face that matches the input person, as well as the camera extrinsic parameters used to describe the head pose and the camera intrinsic parameters used to describe the imaging model, and use the head coordinate system of this pose as a unified head coordinate system.

[0012] Step 3: Back-project the pixels of the input image onto the UV space of the face's 3D geometric mesh (UV space is a two-dimensional coordinate system that uses two axes, U and V, to represent planar positions to avoid confusion with the X, Y, Z space of the 3D model), forming a fragmented texture with original lighting and self-occlusion; perform lighting regularization and inverse rendering on the fragmented texture;

[0013] Step 4: Using the pose of the three-dimensional geometric mesh of the face obtained by face parameter estimation, define the virtual camera trajectory in a unified head coordinate system. Generate a set of videos rotating around the head and their camera intrinsic and extrinsic parameters through a video generation network, and divide the video into a multi-view hair image sequence and its corresponding virtual camera parameters.

[0014] Step 5: Perform multi-view hair reconstruction using the multi-view hair image sequence, the corresponding binary mask, the orientation map predicted by the video generation network, and the corresponding virtual camera parameters. Only optimize the hair geometry and its local normals so that the rendering results under different viewpoints meet the photometric consistency constraint in a unified lighting field, and obtain the hair strand geometric representation or equivalent geometric representation representing the three-dimensional structure of the hair.

[0015] Step 6: Under the unified head coordinate system, spatially align and combine the 3D geometric mesh of the face with its PBR material map and the hair strands to output a head PBR 3D asset that can be directly used in the physically based rendering workflow.

[0016] Step 2, which involves fitting a 3D face parameterization model based on the detected 2D facial key points, includes:

[0017] Step 2-1: Use a face landmark detector to extract several two-dimensional face landmarks from the input image;

[0018] Step 2-2: Construct an objective function that includes keypoint reprojection error, contour constraints, and regularization terms. Minimize the distance between the keypoints of the 3D face parameterization model projected onto the image plane under the current camera parameters and the detected 2D keypoints. Iteratively optimize the shape and pose parameters to obtain a personalized 3D face geometry that highly matches the input person.

[0019] Step 3, the lighting regularization and inverse rendering steps, includes:

[0020] Step 3-1: Using the aforementioned 3D facial geometry mesh and camera intrinsic and extrinsic parameters, backproject the pixels of the input image onto the UV space of the 3D facial geometry mesh to form a fragmented texture with original lighting and self-occlusion. Regularize the lighting of the input image and repair missing areas in the backprojected fragmented texture to obtain a complete facial texture under preset uniform lighting conditions, which serves as the input to the inverse rendering network. Simultaneously, backproject the 3D coordinates of the 3D facial geometry mesh itself and the corresponding normal directions onto the UV space of the 3D facial geometry mesh to form a geometric prior map, i.e., a UV position map. With UV normal mapping ;

[0021] Step 3-2: Input the complete face texture, UV position map, and UV normal map into the UV space structure-based inverse rendering network (FaceUVFormer) for inverse rendering.

[0022] Step 3-2 specifically involves:

[0023] The inverse rendering network is structured as a 2D-3D hybrid denoising network based on a UV-space Transformer architecture, used to generate or complete PBR material maps for faces in UV space. This network can be used as a denoiser for diffusion models or as a conditional generation network.

[0024] The hybrid denoising network adopts a U-Net backbone structure, connecting feature layers of the same scale via skip connections. At least one Hybrid Block module is set in each scale layer of the U-Net, and the input of the Hybrid Block module is the UV feature map of the current scale. The output is .

[0025] The Hybrid Block module includes: a two-dimensional UV branch of the UV Head Block, which is used to perform two-dimensional convolution on the UV feature map obtained by extracting face texture, normal map and position map to extract local high-frequency details;

[0026] UV→Point mapping module: used to map UV feature maps to 3D point cloud features through rasterization, sampling or indexing to obtain dense point features;

[0027] Point Block 3D point cloud branch: used to sparsify, encode position, and perform self-attention and MLP updates on 3D point features to learn 3D neighborhood and global dependencies across UV islands;

[0028] Point→UV mapping module: used to scatter or resample the updated 3D point cloud features back to the UV feature map;

[0029] Feature Fusion Module: Used to fuse the output of the 2D UV branch with the output of the 3D point cloud branch back projection of the Point Block to obtain the output features of the Hybrid Block module.

[0030] The UV Head Block is used to learn local textures and details in the UV plane, including sequentially connected convolution, normalization, and activation units, specifically: 1×1 convolution for channel adjustment; normalization layer (LayerNorm or equivalent BatchNorm, GroupNorm); SiLU activation and 3×3 convolution; normalization layer; SiLU activation and 1×1 convolution; SiLU activation and 3×3 convolution.

[0031] The two-dimensional UV branch of the UV Head Block includes a conditional modulation mechanism: based on a global conditional vector. Channel scale vectors are generated using MLP (Multilayer Perceptron). With bias vector For intermediate features Perform channel-by-channel scaling and translation to satisfy In addition, gating scales can be introduced. Gating the output of this branch is performed to fuse it with the jump connection feature, satisfying... ,in This is a jump connection feature of U-Net at the same scale.

[0032] The global conditional vector y is the conditional input of the network, which is encoded by at least one conditional information, including: global features of the input face image (e.g., extracted by an image encoder); the above-mentioned embedding vector is input into a multilayer perceptron (MLP) and fused to obtain the global conditional vector y. and Channel-by-channel modulation parameters are obtained by MLP prediction from the global conditional vector y. This is the channel scale vector. For channel shift vectors, and This is used to scale and translate intermediate features channel by channel, thereby injecting conditional information into the network features;

[0033] Correspondingly, intermediate features Features after conditional modulation The gated scale, obtained by MLP prediction from the global conditional vector y, is used to control the intensity of the output of a certain branch before fusing it with the skip connection feature to suppress noise or unreliable branch features. This indicates the output characteristics of the UV head block. This represents the skip connection features of U-Net passed from the encoder path to the decoder path at the same scale. This indicates the characteristics after gating fusion.

[0034] The Point Block 3D point cloud branch is used to recover cross-island connections caused by UV parameterized segmentation in 3D space, and it includes the following processing: processing dense point features obtained from UV→Point. Perform grid pooling to obtain sparse point features. To introduce 3D positional encoding for sparse point features, a linear layer is first used to reduce the dimensionality of the input channels, followed by sparse convolution to inject positional encoding information, and then a linear layer is used to increase the dimensionality back to the original channel dimension, thus reducing the computational cost of high-dimensional features. Self-attention updates are performed on the sparse point features. The attention mechanism can employ efficient serialized attention based on spatial filling curves, grouping points by sequence code and performing attention within each group to improve efficiency. The attention output is updated via a non-linear mapping through an MLP feedforward network. Conditional modulation and gating mechanisms are introduced, based on a global conditional vector. Generated by MLP Scaling, translation, and gating are applied to the point features respectively; sparse point features are then processed. The coordinates of the dense point are obtained by scattering back the pooling index. Before backprojecting the point branch back to the UV space, the point branch gating coefficients can be additionally learned based on the conditional vectors. And it is integrated with the UV branch output to meet the requirements. ,in This is the feature map after Point→UV mapping.

[0035] The inverse rendering network fuses the UV coordinate information of the complete face texture through positional encoding and is pre-trained on face PBR data containing various lighting conditions to decompose the input complete face texture into physically decoupled material property channels such as albedo, roughness, specular and normal.

[0036] Step 4, which involves generating a multi-view hair image sequence, specifically includes:

[0037] Step 4-1: Input the frontal view image into the video generation network, using the virtual camera trajectory defined by the unified head coordinate system as the view condition;

[0038] Step 4-2: The video generation network generates a video with continuously changing inter-frame perspectives that corresponds one-to-one with a preset virtual perspective. The multi-view hair image sequence extracted from the video approximately satisfies the appearance consistency while maintaining the input hair style and overall shape.

[0039] The number of virtual perspectives mentioned in step 4-1 shall not be less than three perspectives.

[0040] The multi-view hair reconstruction described in step 5 specifically obtains the three-dimensional geometry of the hair through the following methods:

[0041] Step 5-1: Input the multi-view hair image sequence and its virtual camera intrinsic and extrinsic parameters into a reconstruction framework based on Gaussian volume or hair chain representation, and optimize the hair position, direction and density parameters;

[0042] Step 5-2: Construct a joint loss function by comparing the photometric error and projection contour error between the rendering results from each viewpoint and the corresponding input image;

[0043] Step 5-3: The reconstructed hair geometry is represented in the form of hair strand chains, anisotropic Gaussian or a combination thereof, and aligned to the face and scalp region through a unified head coordinate system.

[0044] The output head PBR 3D asset is stored in the form of a 3D geometric mesh of the face, 3D geometry of the hair and its corresponding PBR material textures, and can be exported as a general 3D asset format, material format and scene description format, so as to be directly imported into a real-time rendering engine or offline rendering system that supports physically based rendering workflow for relighting and animation driving; when exporting or using, the geometry and PBR materials can be baked and mapped according to the target mesh topology specified by the user, without having to fix the specific topology in advance during the reconstruction stage.

[0045] This invention provides an automated pipeline for generating 3D facial portrait assets based on a single view, which can output high-fidelity head 3D assets that can be directly used in physically based rendering (PBR) workflows in a single step without multi-view data and manual intervention.

[0046] Beneficial effects:

[0047] 1. Extremely low data requirements: Only one frontal or near-frontal portrait photo is needed, without special equipment or multi-view data.

[0048] 2. Collaborative Reconstruction of Face and Hair: A complete process is proposed, which solves the reconstruction problems of face and hair, which are of different difficulties.

[0049] 3. High-quality PBR materials: Drawing on the latest advancements in inverse rendering, PBR standard materials decoupled from lighting have been generated, enabling the model to be used for high-quality real-time rendering and relighting.

[0050] 4. Innovative hair reconstruction approach: It cleverly utilizes generative models to "create" multi-view data, bypassing the difficulties of physical acquisition, and providing a brand-new and effective solution for single-view hair reconstruction.

[0051] 5. High degree of automation and practicality: The entire process requires no manual intervention, and the generated model assets can be directly imported into mainstream game engines and graphics software, which has extremely high practical application value. Attached Figure Description

[0052] Figure 1 This is an overall flowchart of an embodiment of the present invention.

[0053] Figure 2 The input image.

[0054] Figure 3 This is the output result. Detailed Implementation

[0055] Reference Figure 1 The overall process of the present invention embodiment shown is as follows: A head reconstruction method based on collaborative optimization of generation and inverse rendering includes the following steps:

[0056] Step 1: Use a single-view RGB portrait as the input image (e.g., Figure 2 As shown in the figure, an image segmentation network is used to identify the face region and hair region, and generate corresponding binary masks;

[0057] Step 2: For the face region, fit the detected two-dimensional face key points to the three-dimensional geometric mesh of the face that matches the input person, as well as the camera extrinsic parameters used to describe the head pose and the camera intrinsic parameters used to describe the imaging model, and use the head coordinate system of this pose as a unified head coordinate system.

[0058] Step 3: Using the face 3D geometric mesh and camera intrinsic and extrinsic parameters, backproject the pixels of the input image onto the UV space of the face 3D geometric mesh to form a fragmented texture with original lighting and self-occlusion; perform lighting regularization on the fragmented texture to obtain a complete texture under preset uniform lighting conditions, and input the complete texture into a pre-trained inverse rendering network to output a multi-channel PBR material map including albedo map, roughness map, specular map, and normal map;

[0059] Step 4: Using the pose of the three-dimensional geometric mesh of the face obtained by face parameter estimation, define the virtual camera trajectory in a unified head coordinate system. Generate a set of videos rotating around the head and their camera intrinsic and extrinsic parameters through a video generation network, and divide the video into a multi-view hair image sequence and its corresponding virtual camera parameters.

[0060] Step 5: Perform multi-view hair reconstruction using the multi-view hair image sequence, the corresponding binary mask, the orientation map predicted by the video generation network, and the corresponding virtual camera parameters. Only optimize the hair geometry and its local normals so that the rendering results under different viewpoints meet the photometric consistency constraint in a unified lighting field, and obtain the hair strand geometric representation or equivalent geometric representation representing the three-dimensional structure of the hair.

[0061] Step 6: Under the unified head coordinate system, spatially align and combine the 3D geometric mesh of the face, its PBR material map, and the hair strands to output a head PBR 3D asset that can be directly used in a physically based rendering workflow. For example... Figure 3 As shown, the output PBR texture, 3D face mesh, and hair geometry reconstruction results render a result that matches the features of the input image.

[0062] Step 2, which involves fitting a 3D face parameterization model based on the detected 2D facial key points, includes:

[0063] Step 2-1: Use a face landmark detector to extract several two-dimensional face landmarks from the input image;

[0064] Step 2-2: Construct an objective function that includes keypoint reprojection error, contour constraints, and regularization terms. Minimize the distance between the keypoints of the 3D face parameterization model projected onto the image plane under the current camera parameters and the detected 2D keypoints. Iteratively optimize the shape and pose parameters to obtain a personalized 3D face geometry that highly matches the input person.

[0065] In this embodiment, specifically:

[0066] First, a facial landmark detection algorithm is run on the face region of the input image to extract dozens of 2D landmarks (such as the corners of the eyes, the tip of the nose, and the contours of the lips). Then, a 3D face parametric model, the FLAME model, is selected as the basic geometric representation. This model has shape parameters that define identity and expression. We construct an optimization objective, the core of which is to minimize the distance between the 3D landmarks projected onto the image plane by the parametric model under the current camera parameters and the detected 2D landmarks (i.e., reprojection error). After optimization, the resulting parametric model instance is the reconstructed 3D face geometry.

[0067] Step 3, the lighting regularization and inverse rendering steps, includes:

[0068] Step 3-1: Using the aforementioned 3D facial geometry mesh and camera intrinsic and extrinsic parameters, backproject the pixels of the input image onto the UV space of the 3D facial geometry mesh to form a fragmented texture with original lighting and self-occlusion. Regularize the lighting of the input image and repair missing areas in the backprojected fragmented texture to obtain a complete facial texture under preset uniform lighting conditions, which serves as the input to the inverse rendering network. Simultaneously, backproject the 3D coordinates of the 3D facial geometry mesh itself and the corresponding normal directions onto the UV space of the 3D facial geometry mesh to form a geometric prior map, i.e., a UV position map. With UV normal mapping ;

[0069] Step 3-2: Input the complete face texture, UV position map, and UV normal map into the UV space-based inverse rendering network (FaceUVFormer) for inverse rendering.

[0070] Step 3-2 specifically involves:

[0071] The inverse rendering network is structured as a 2D-3D hybrid denoising network based on a UV-space Transformer architecture, used to generate or complete PBR material maps for faces in UV space. This network can be used as a denoiser for diffusion models or as a conditional generation network.

[0072] The hybrid denoising network adopts a U-Net backbone structure, connecting feature layers of the same scale via skip connections. At least one Hybrid Block module is set in each scale layer of the U-Net, and the input of the Hybrid Block module is the UV feature map of the current scale. The output is .

[0073] The Hybrid Block module includes: a two-dimensional UV branch of the UV Head Block, which is used to perform two-dimensional convolution on the UV feature map obtained by extracting face texture, normal map and position map to extract local high-frequency details;

[0074] UV→Point mapping module: used to map UV feature maps to 3D point cloud features through rasterization, sampling or indexing to obtain dense point features;

[0075] Point Block 3D point cloud branch: used to sparsify, encode position, and perform self-attention and MLP updates on 3D point features to learn 3D neighborhood and global dependencies across UV islands;

[0076] Point→UV mapping module: used to scatter or resample the updated 3D point cloud features back to the UV feature map;

[0077] Feature Fusion Module: Used to fuse the output of the 2D UV branch with the output of the 3D point cloud branch back projection of the Point Block to obtain the output features of the Hybrid Block module.

[0078] The UV Head Block is used to learn local textures and details in the UV plane, including sequentially connected convolution, normalization, and activation units, specifically: 1×1 convolution for channel adjustment; normalization layer (LayerNorm or equivalent BatchNorm, GroupNorm); SiLU activation and 3×3 convolution; normalization layer; SiLU activation and 1×1 convolution; SiLU activation and 3×3 convolution.

[0079] The two-dimensional UV branch of the UV Head Block includes a conditional modulation mechanism: based on a global conditional vector. Channel scale vectors are generated using MLP (Multilayer Perceptron). With bias vector For intermediate features Perform channel-by-channel scaling and translation to satisfy In addition, gating scales can be introduced. Gating the output of this branch is performed to fuse it with the jump connection feature, satisfying... ,in This is a jump connection feature of U-Net at the same scale.

[0080] The global condition vector y is the conditional input of the network and is encoded by at least one conditional information, including: global features of the input face image (e.g., extracted by an image encoder). The above-mentioned embedding vector is input into a multilayer perceptron (MLP) and fused to obtain the global condition vector y. and Channel-by-channel modulation parameters are obtained by MLP prediction from the global conditional vector y. This is the channel scale vector. For channel shift vectors, and This is used to scale and translate intermediate features channel by channel, thereby injecting conditional information into the network features;

[0081] Correspondingly, intermediate features Features after conditional modulation The gated scale, obtained by MLP prediction from the global conditional vector y, is used to control the intensity of the output of a certain branch before fusing it with the skip connection feature to suppress noise or unreliable branch features. This indicates the output characteristics of the UV head block. This represents the skip connection features of U-Net passed from the encoder path to the decoder path at the same scale. This indicates the characteristics after gating fusion.

[0082] The Point Block 3D point cloud branch is used to recover cross-island connections caused by UV parameterized segmentation in 3D space, and it includes the following processing: processing dense point features obtained from UV→Point. Perform grid pooling to obtain sparse point features. To introduce 3D positional encoding for sparse point features, a linear layer is first used to reduce the dimensionality of the input channels, followed by sparse convolution to inject positional encoding information, and then a linear layer is used to increase the dimensionality back to the original channel dimension, thus reducing the computational cost of high-dimensional features. Self-attention updates are performed on the sparse point features. The attention mechanism can employ efficient serialized attention based on spatial filling curves, grouping points by sequence code and performing attention within each group to improve efficiency. The attention output is updated via a non-linear mapping through an MLP feedforward network. Conditional modulation and gating mechanisms are introduced, based on a global conditional vector. Generated by MLP Scaling, translation, and gating are applied to the point features respectively; sparse point features are then processed. The coordinates of the dense point are obtained by scattering back the pooling index. Before backprojecting the point branch back to the UV space, the point branch gating coefficients can be additionally learned based on the conditional vectors. And it is integrated with the UV branch output to meet the requirements. ,in This is the feature map after Point→UV mapping.

[0083] In this embodiment, the input to the network includes, but is not limited to: the complete face texture and geometric prior map obtained in step 3-1, including UV position maps. With UV normal mapping Time step (When using diffusion generation or denoising training); and conditional information, including global image embeddings, text embeddings, or pose / lighting embeddings from the input image.

[0084] Global condition vector It can be obtained by fusing image embedding, text embedding, and temporal step embedding after MLP mapping, and used to conditionally modulate the UV Head Block and Point Block to ensure that the network output is consistent with the input face appearance, lighting prior, and geometry. The network output is a multi-channel PBR material map in UV space. These correspond to albedo (BaseColor / Albedo), roughness, specular, and normal, respectively. In the implementation that only generates texture color maps, the output can also be an RGB texture map. In this patent implementation, the above-mentioned PBR multi-channel map is preferred.

[0085] The inverse rendering network fuses the UV coordinate information of the complete face texture through positional encoding and is pre-trained on face PBR data containing various lighting conditions to decompose the input complete face texture into physically decoupled material property channels such as albedo, roughness, specular and normal.

[0086] Step 4, which involves generating a multi-view hair image sequence, specifically includes:

[0087] Step 4-1: Input the frontal view image into the video generation network, using the virtual camera trajectory defined by a unified head coordinate system as the view condition; in this embodiment, the video generation network is an Image-to-Video model.

[0088] Step 4-2: The video generation network generates a video with continuously changing inter-frame perspectives that corresponds one-to-one with preset virtual perspectives. The multi-view hair image sequence extracted from the video has at least three virtual perspectives, while maintaining the input hair style and overall shape.

[0089] The multi-view hair reconstruction described in step 5 specifically obtains the three-dimensional geometry of the hair through the following methods:

[0090] Step 5-1: Input the multi-view hair image sequence and its virtual camera intrinsic and extrinsic parameters into a reconstruction framework based on Gaussian volume or hair chain representation (in this embodiment, the reconstruction framework is the Gaussian Haircut model) to optimize the hair position, direction and density parameters;

[0091] Step 5-2: Construct a joint loss function by comparing the photometric error and projection contour error between the rendering results from each viewpoint and the corresponding input image;

[0092] Step 5-3: The reconstructed hair geometry is represented in the form of hair strand chains, anisotropic Gaussian or a combination thereof, and aligned to the face and scalp region through a unified head coordinate system.

[0093] The output head PBR 3D asset is stored in the form of a 3D geometric mesh of the face, 3D geometry of the hair and its corresponding PBR material textures, and can be exported as a general 3D asset format, material format and scene description format, so as to be directly imported into a real-time rendering engine or offline rendering system that supports physically based rendering workflow for relighting and animation driving; when exporting or using, the geometry and PBR materials can be baked and mapped according to the target mesh topology specified by the user, without having to fix the specific topology in advance during the reconstruction stage.

[0094] This invention provides a head reconstruction method based on collaborative optimization of generation and inverse rendering. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A head reconstruction method based on collaborative optimization of generation and inverse rendering, characterized in that, Includes the following steps: Step 1: Take a single-view RGB portrait as the input image and use an image segmentation network to identify the face and hair regions, generating the corresponding binary mask; Step 2: For the face region, fit the detected two-dimensional face key points to the three-dimensional geometric mesh of the face that matches the input person, as well as the camera extrinsic parameters used to describe the head pose and the camera intrinsic parameters used to describe the imaging model, and use the head coordinate system of the head pose as a unified head coordinate system. Step 3: Back-project the pixels of the input image onto the UV space of the three-dimensional geometric mesh of the face to form a fragmented texture with original lighting and self-occlusion. Perform illumination regularization and inverse rendering on the incomplete texture; Step 4: Using the pose of the three-dimensional geometric mesh of the face obtained by face parameter estimation, define the virtual camera trajectory in a unified head coordinate system, and generate a multi-view hair image sequence and its corresponding virtual camera parameters through a video generation network. Step 5: Perform multi-view hair reconstruction using the multi-view hair image sequence, the corresponding binary mask, the orientation map predicted by the video generation network, and the corresponding virtual camera parameters to obtain a geometric representation or equivalent geometric representation of the hair strands representing the three-dimensional structure of the hair. Step 6: Under the unified head coordinate system, spatially align and combine the 3D geometric mesh of the face with its PBR material map and the hair strands to output a head PBR 3D asset that can be directly used in the physically based rendering workflow.

2. The head reconstruction method based on collaborative optimization of generation and inverse rendering as described in claim 1, characterized in that, Step 2, which involves fitting a 3D face parameterization model based on the detected 2D facial key points, includes: Step 2-1: Use a face landmark detector to extract several two-dimensional face landmarks from the input image; Step 2-2: Minimize the distance between the key points of the 3D face parameterization model projected onto the image plane under the current camera parameters and the detected 2D key points, and iteratively optimize the shape and pose parameters to obtain a personalized 3D face geometry that highly matches the input person.

3. The head reconstruction method based on collaborative optimization of generation and inverse rendering as described in claim 1, characterized in that, Step 3, the lighting regularization and inverse rendering steps, includes: Step 3-1: The input image pixels are back-projected onto the UV space of the 3D face geometry mesh to form a fragmented texture with original lighting and self-occlusion. Simultaneously, the 3D coordinates of the face geometry mesh itself and the corresponding normal directions are back-projected onto its UV space to form a geometric prior map, i.e., a UV position map. With UV normal mapping ; Step 3-2: Input the complete face texture, UV position map, and UV normal map into the UV space structure-based inverse rendering network for inverse rendering.

4. The head reconstruction method based on collaborative optimization of generation and inverse rendering according to claim 3, characterized in that, Step 3-2 specifically involves: The structure of the inverse rendering network is as follows: it is a 2D-3D hybrid denoising network structure based on the UV space Transformer architecture; the hybrid denoising network adopts the U-Net backbone structure and connects feature layers of the same scale through skip connections; at least one Hybrid Block module is set in each scale layer of U-Net, and the input of the Hybrid Block module is the UV feature map of the current scale.

5. The head reconstruction method based on collaborative optimization of generation and inverse rendering according to claim 4, characterized in that, The Hybrid Block module includes: a two-dimensional UV branch of the UV Head Block, a UV→Point mapping module, a three-dimensional point cloud branch of the PointBlock, a Point→UV mapping module, and a feature fusion module.

6. The head reconstruction method based on collaborative optimization of generation and inverse rendering according to claim 5, characterized in that, The UV Head Block includes convolution, normalization and activation units connected in sequence.

7. The head reconstruction method based on collaborative optimization of generation and inverse rendering according to claim 5, characterized in that, The two-dimensional UV branch of the UV Head Block includes a conditional modulation mechanism: based on a global conditional vector. Channel scale vectors generated by MLP With bias vector The intermediate features are scaled and translated channel by channel; And introduce gating scale Gating the output of this branch is used to fuse it with the jump connection feature. The global condition vector y is the conditional input of the network, which is encoded by conditional information. The conditional information includes: global features of the input face image, optional text description features, and time step embeddings corresponding to diffusion time steps or network stage identifiers. The intermediate feature is a feature after conditional modulation.

8. The head reconstruction method based on collaborative optimization of generation and inverse rendering according to claim 7, characterized in that, The Point Block 3D point cloud branch performs grid pooling on the dense point features obtained by UV→Point to obtain sparse point features; Three-dimensional position encoding is introduced for sparse point features. Points are grouped by sequence code and attention is performed within the group to improve efficiency. Scaling, translation and gating are performed on the point features respectively. The sparse point features are scattered back to the dense point coordinates by pooling index. The point branches are projected back to the UV space and fused with the UV branch output.

9. The head reconstruction method based on collaborative optimization of generation and inverse rendering according to claim 1, characterized in that, Step 4, which involves generating a multi-view hair image sequence, specifically includes: Step 4-1: Input the frontal view image into the video generation network, using the virtual camera trajectory defined by the unified head coordinate system as the view condition; Step 4-2: The video generation network generates a video with continuously changing inter-frame perspectives that corresponds one-to-one with a preset virtual perspective. The multi-view hair image sequence extracted from the video approximately satisfies the appearance consistency while maintaining the input hair style and overall shape.

10. A head reconstruction method based on collaborative optimization of generation and inverse rendering according to claim 1, characterized in that, The multi-view hair reconstruction described in step 5 specifically obtains the three-dimensional geometry of the hair through the following methods: Step 5-1: Input the multi-view hair image sequence and its virtual camera intrinsic and extrinsic parameters into a reconstruction framework based on Gaussian volume or hair chain representation, and optimize the hair position, direction and density parameters; Step 5-2: Construct a joint loss function by comparing the photometric error and projection contour error between the rendering results from each viewpoint and the corresponding input image; Step 5-3: The reconstructed hair geometry is represented in the form of hair strand chains, anisotropic Gaussian or a combination thereof, and aligned to the face and scalp region through a unified head coordinate system.

Images