A robust 3D face reconstruction method based on a depth-guided two-stream network

Abstract

The present invention proposes a three-dimensional face reconstruction method based on a depth-guided two-stream network, constructs an end-to-end depth-guided two-stream network, and improves the robustness and generalization in complex environments. This method integrates image and depth prior information to enhance the performance of the reconstruction system under complex backgrounds, extreme lighting and large postures. Specifically: a facial geometry perception module is designed to generate accurate facial masks using image semantic features and depth priors to suppress background interference; a bidirectional cross-attention module is introduced to achieve efficient fusion of RGB and depth information and enhance feature representation capabilities; key modules work together to alleviate depth ambiguity problems, generate high-quality UV position maps and texture maps, and obtain high-precision three-dimensional face representations. This method outperforms mainstream methods in multiple evaluation indicators, especially in difficult scenes such as complex backgrounds, verifying the effectiveness, robustness and application potential of the framework, and providing technical innovation and practical value for face reconstruction in extreme environments.

Description

Technical Field

[0001] The present invention mainly relates to technical fields such as deep learning, intelligent image processing, convolutional neural networks and three-dimensional reconstruction. By utilizing appropriate image depth prior information, it particularly relates to a single-view three-dimensional reconstruction method of human face based on deep learning. Background Art

[0002] With the rapid development of computer technology and digital media technology, images of objects have become easier to obtain. However, images are only simple two-dimensional information, and the information they convey is limited. Therefore, how to obtain more information about objects has gradually become the focus of current research. Three-dimensional face reconstruction technology aims to accurately restore the three-dimensional structure of the face from two-dimensional images, and has important application value in augmented reality (AR), virtual reality (VR), virtual character generation and other fields. Existing three-dimensional deformable model (3DMM) methods usually rely on extracting features from images to infer three-dimensional geometric information. However, the performance of these methods in complex scenes is often significantly limited, such as when involving shadows and large posture changes. This limitation is mainly attributed to the lack of clear depth information constraints, which leads to pose uncertainty and scale ambiguity in the generated three-dimensional face. Researchers have proposed a large number of three-dimensional face reconstruction algorithms to solve the above problems.

[0003] Specifically, three-dimensional deformable models (3DMMs) have emerged as a key approach to addressing this problem. 3DMMs construct 3D facial representations using statistical methods, significantly improving the fidelity and interpretability of generated 3D faces. This is primarily due to the precise control of coefficients such as expression and shape. However, existing 3DMM methods still face numerous challenges when applied to complex scenes. For example, in outdoor scenes, complex lighting conditions, severe occlusions, and large pose variations significantly impact reconstruction performance. The root cause of these issues lies in the lack of explicit depth information constraints, which leads to depth ambiguity and, in turn, makes it difficult to accurately estimate the pose uncertainty and scale of the 3D face. Specifically, when relying solely on RGB images, strong light can cast shadows on facial features, interfering with the extraction of geometric features and making it difficult to accurately estimate facial orientation and scale. Similarly, at extreme viewing angles, the lack of reliable depth information as a geometric constraint often results in significant directional errors in the reconstructed face. Existing techniques for 3D face reconstruction in complex scenes present significant limitations, necessitating a method that can effectively integrate depth information to improve the robustness and accuracy of reconstruction. Summary of the Invention

[0004] To solve the above problems, the present invention proposes a robust three-dimensional face reconstruction method based on a depth-guided two-stream network. By combining RGB information with depth prior information, their respective advantages are fully utilized to achieve better three-dimensional reconstruction effects. On the one hand, an innovative bidirectional cross-attention module is adopted to capture the complementary information between image and depth modalities to improve the understanding of complex scenes and to integrate image and depth information to enhance three-dimensional reconstruction of the face. On the other hand, in order to further improve the accuracy of three-dimensional facial position and scale estimation, an accurate facial visibility mask is introduced to guide the network to focus on facial features. Finally, through the synergistic effect of the two key modules, background interference is effectively suppressed, and the final accurate three-dimensional reconstruction model is obtained.

[0005] The technical solution adopted by the present invention is a robust 3D face reconstruction method based on a depth-guided dual-stream network architecture. It is implemented in the following steps:

[0006] Step 1: Construction of a two-stream network guided by image and depth information. A depth-guided two-stream network is provided, which extracts features from the image and depth map respectively through independent encoders, thereby fully exploiting the independent features of the two modalities.

[0007] Step 2: Facial geometry perception module. To alleviate the problems of background interference and detail loss in facial image feature extraction, a facial geometry perception module is designed. This module combines the semantic information of the image with the depth prior to generate an accurate facial mask, effectively suppressing background interference and improving the accuracy of 3D face position and scale estimation.

[0008] Step 3: Construct a bidirectional cross-attention module. Through a bidirectional feature interaction mechanism, the image and depth features extracted above are effectively interacted, promoting bidirectional information flow between RGB features and depth features. This mechanism not only achieves complementary synergy between image and depth information, but also significantly enhances feature representation capabilities, thereby improving the model's robustness and accuracy in 3D face reconstruction tasks.

[0009] The present invention is also characterized in that:

[0010] The specific implementation process of step 1 is as follows: Given an RGB image I rgb ∈R 3×256×256 and a depth map I depth ∈R 1×256×256 , independent encoder E rgb and E depth Extract features from RGB and depth images respectively to obtain feature representation F rgb and E depth .

[0011] The specific implementation process of step 2 is as follows: In order to improve the accuracy of 3D face pose and scale estimation, the facial geometry perception module combines the RGB information of the image with the depth prior to generate an accurate facial visibility mask to guide the network to focus on key facial areas. First, the facial geometry perception module uses image features and depth features to generate an accurate facial visibility mask. and As input. Then feature level fusion is performed, expressed as:

[0012] F init =F rgb +F depth

[0013] Then use channel attention A ca (·) and spatial attention A sa (·) to model feature dependencies:

[0014]

[0015] To enhance the adaptive fusion between modalities, a confidence estimation module is introduced to generate a confidence map by computing the pixel-wise difference between modal features:

[0016] C conf =Sigmod(Conv 3×3 (|F rgb -F depth |))

[0017] In addition, using the pixel attention module A pixel To achieve fine-grained feature fusion, the input of pixel attention is and F init , Hadamard product is used express:

[0018]

[0019] Then, using Conv 1×1 Adjust the number of channels, and then generate the facial mask M through the Sigmoid activation function. Finally, the enhanced features are obtained by element-wise multiplication with M. and

[0020] and

[0021] The specific implementation process of step 3 is as follows: In order to effectively utilize the complementary information between the image and the depth image, we designed a bidirectional cross attention module to enhance the multimodal feature representation through a bidirectional feature interaction mechanism. First, taking the depth stream as an example, we first perform a convolution operation on the above features to obtain and We then obtain query, key, and value vectors by applying a linear transformation on the augmented input features:

[0022]

[0023] Where W q 、W k and W v is a learnable linear transformation matrix.

[0024] Next, the interaction between RGB and depth features is calculated through the attention mechanism, aiming to fully tap the complementary information between the two modalities. Image features and depth features usually contain different types of spatial information, and using either modality alone may not be able to effectively capture all key information. Through the attention mechanism, the model can pay more attention to the correlation between the two features when fusing them, thereby improving the final feature expression ability. Adding the image query to the attention-weighted depth features allows the model to introduce depth information to enhance the understanding of the scene structure while maintaining the original image features. The specific formula is as follows:

[0025] F att_out =LayerNorm(Q img +Attention(Q rgb , K depth , V depth )) Furthermore, in order to further enhance feature fusion and retain depth information, we add the F output of the attention mechanism to att_out Features and deep features of input Fusion. Then, we use a nonlinear activation function ReLU to introduce nonlinear feature expression, thereby improving the expressiveness and robustness of the model:

[0026] Similarly, the RGB stream is operated in the same way as above. Finally, the fused features of the two branches are input into the fusion module after a round of convolution respectively to generate the final feature representation:

[0027] F fusion =Fusion(F fused_rgb , F fused_depth )

[0028] It should be noted that the overall architecture of this fusion module is very similar to that of the facial geometry perception module, but the main difference is that the Sigmoid function of the final layer is removed.

[0029] The bidirectional cross-attention module serves as the main feature extraction module for the 3D face reconstruction model. The reconstruction network fuses the attention feature map from the face attention supervision network with the depth detail supplementary information. This requires addressing two issues: 1) Since image and depth information have different spatial dimensions, the area of ​​occluded pixels and the occlusion level in the image are unknown; 2) Since the depth channel prior contains inconsistent feature information compared to the original image, effectively fusing them and fully utilizing the detail prior is key to improving the quality of the 3D reconstruction model. Through a bidirectional feature interaction mechanism, the information flow between image and depth features is promoted to obtain fused features, achieving complementary synergy between image and depth information, significantly enhancing the model's feature representation capabilities, and improving the robustness and accuracy of the final 3D face reconstruction.

[0030] In summary, the main contributions of the present invention are:

[0031] (1) The present invention provides a robust three-dimensional face reconstruction method based on a depth information guided dual-stream network. By using independent encoders to extract image and depth features respectively, a depth information guided dual-stream network framework is designed. The interaction between features is enhanced through a bidirectional cross-attention module, and cross-modal complementary information is optimized. The image input of the cross-attention supervision network with coordinated depth information and image information is used to output an attention feature map that perceives the uneven feature distribution information of complex backgrounds and large postures. The generated attention feature map will guide the main network to perform feature screening and focus on more important feature areas, significantly improving the robustness and accuracy of three-dimensional face reconstruction.

[0032] (2) The present invention provides a robust 3D face reconstruction method based on a depth-guided dual-stream network. To alleviate the problems of background interference and detail loss in facial image feature extraction, a supplementary facial geometry perception module is introduced. This module combines the semantic information of the image with the depth prior to generate an accurate facial mask, effectively suppressing background interference, enhancing the feature extraction capability, and improving the accuracy of 3D face position and scale estimation.

[0033] (3) The present invention provides a robust three-dimensional face reconstruction method based on a dual-stream network guided by depth information. In order to enhance the model's feature representation capability, a bidirectional cross-attention module is introduced. As the main feature extraction module of the face three-dimensional reconstruction model, the reconstruction network fuses the attention feature map and depth detail supplementary information from the face attention supervision network, and promotes the information flow between image features and depth features through a bidirectional feature interaction mechanism to obtain fused features, thereby achieving complementary synergy between image and depth information, significantly enhancing the model's feature representation capability, and introducing depth information as a geometric constraint to solve the posture uncertainty and scale ambiguity problems of traditional methods under complex lighting, shadows and large posture changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a schematic diagram of the overall structure of a robust 3D face reconstruction method using a deep-guided dual-stream network proposed in the present invention;

[0035] Figure 2 This is a schematic diagram comparing the effects of 3D facial reconstruction proposed by the present invention. DETAILED DESCRIPTION

[0036] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the described embodiments are intended only to facilitate understanding of the present invention and do not limit it in any way. The accompanying drawings are in a very simplified form and are not in precise proportions. They are only used to conveniently and clearly assist in illustrating the embodiments of the present invention. The structures shown in the accompanying drawings are part of the actual structure. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0037] This paper proposes a 3D face reconstruction method based on a depth-guided dual-stream network framework. First, the network extracts features from the image and depth map using independent encoders. Second, a facial geometry perception module combines the image's semantic information with depth priors to construct a refinement network to compensate for background interference during reconstruction. Finally, a bidirectional cross-attention module fuses the attention feature map and supplementary depth detail information. This fused feature achieves enhanced representational 3D face reconstruction, effectively improving the accuracy of 3D face position and scale estimation.

[0038] The overall model is Figure 1 As shown in the figure, it contains three modules, namely, a depth-guided multimodal feature extraction two-stream network, a facial geometry perception module, and a bidirectional cross-attention module.

[0039] The purpose of the two-stream feature extraction network is to generate image semantic and depth information feature maps with non-uniform feature distribution information, which serve as the attention basis for subsequent main network feature selection.

[0040] First, the depth-guided multimodal feature extraction dual-stream supervised network proposed in this invention follows the classic encoding-decoding structure, takes the RGB image and depth map as the input of the dual-stream network, extracts image and depth features through independent encoders respectively, and obtains image semantic feature maps and depth feature maps of the same size.

[0041] Next, to improve the network's accuracy in 3D face estimation, the present invention introduces a facial geometry-aware module, which combines the image's RGB information with depth prior information to generate an accurate facial visibility mask. This mask guides the network to focus on key facial regions and suppresses interference from irrelevant areas, thereby improving the accuracy of pose and scale estimation. Specifically, the facial geometry-aware module performs feature-level fusion of image and depth features, leveraging channel and spatial attention mechanisms to model dependencies between features, further enhancing adaptive fusion between modalities.

[0042] The facial geometry perception module receives image and depth features as input. To model inter-feature dependencies, channel attention and spatial attention mechanisms are employed. To enhance adaptive inter-modal fusion, a confidence estimation module is introduced. This module generates a confidence map by calculating the pixel-by-pixel differences between RGB and depth modal features and processes it using a sigmoid activation function. Furthermore, the pixel attention module further enhances fusion through fine-grained feature fusion. Finally, the facial mask M is generated by adjusting the number of channels and using a sigmoid activation function. The enhanced feature representation is then obtained through element-wise multiplication with M.

[0043] To effectively utilize the complementary information between image and depth images, the present invention designs a bidirectional cross-attention module. This module enhances the multimodal feature representation of RGB and depth features through a bidirectional feature interaction mechanism. In this process, the network first performs a convolution operation on the input features to obtain enhanced input features. It then further calculates the interaction between the RGB and depth features through an attention mechanism. This process fully exploits the correlation between the two modalities, thereby improving the model's estimation accuracy of 3D facial pose and scale.

[0044] Finally, a decoder network is used to perform regression prediction on the reconstructed face image. The purpose of the decoder is to integrate the features of the network and the detail refinement network to achieve the final 3D face reconstruction.

[0045] The decoder network gradually restores low-dimensional feature maps to higher-dimensional spatial resolution through a series of progressively upsampling convolution operations. Input features first pass through a preliminary decoding module consisting of multiple transposed convolutional layers. Each convolution operation effectively expands the spatial resolution of the feature map and gradually increases the number of channels. Through these transposed convolutional layers, the network gradually recovers image details, ensuring that the final output has sufficient spatial information and feature representation capabilities.

[0046] Based on the initially decoded feature map, the network splits into two independent branches for further feature generation. The first branch ultimately outputs a three-channel texture coordinate position map. The second branch is responsible for generating horizontal and vertical coordinate maps. Through different decoding paths, these two branches output key spatial information relevant to face reconstruction.

[0047] Ultimately, these outputs work together to provide complete spatial coordinates for reconstructing the face, ultimately restoring the structure and details of the original face. This process not only accurately captures the position and shape of facial features, but also reconstructs facial texture at a fine-grained level, providing rich feature information for subsequent facial analysis tasks.

[0048] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A robust 3D face reconstruction method using a deep-guided two-stream network, characterized by: Step 1: A two-stream network guided by image and depth information is constructed. Separate encoders are used to extract preliminary features from the image and depth map, respectively. A facial geometry perception module is then used to combine the image's semantic information with depth priors to guide the network's focus on facial region features. A bidirectional cross-attention module is used to enhance the interaction between features and optimize cross-modal complementary information. Finally, a decoder network is used to reconstruct the 3D face. Step 2: Construction of the facial geometry perception module. To alleviate the problems of background interference and detail loss in facial image feature extraction, a facial geometry perception module is designed. This module combines the semantic information of the image with the depth prior to generate an accurate facial mask, effectively suppressing background interference and improving the accuracy of 3D face position and scale estimation. Specifically, the facial geometry perception module performs feature-level fusion of image features and depth features, using channel and spatial attention mechanisms to extract dependencies between features, further enhancing the adaptive fusion between modalities. A confidence estimation module is introduced to generate a confidence map by calculating the pixel-by-pixel difference between RGB and depth modal features, and processing it through the Sigmoid activation function. The pixel attention module further improves the fusion effect through fine-grained feature fusion. Finally, the number of channels is adjusted and the Sigmoid activation function is used to generate the facial mask M. The enhanced feature representation is obtained by element-wise multiplication with M. Step 3: Construction of the bidirectional cross-attention module, as the main feature extraction module of the face 3D reconstruction model, the network fuses feature maps and promotes the information flow between image features and depth features through a bidirectional feature interaction mechanism to obtain fused features, thereby achieving complementary synergy between image and depth information, significantly enhancing the feature representation ability of the model and improving the robustness and accuracy of the final 3D face reconstruction.

2. The robust 3D face reconstruction method using a deep-guided two-stream network according to claim 1, characterized in that: The construction of the dual-stream network guided by the image and depth information in step 1 adopts the classic codec structure as the main network structure. At the same time, the hybrid feature extraction module containing image information and depth information is used as the core of the network extraction to predict and regress the feature map of the feature image semantics and depth position.

Images