A three-dimensional facial defect completion method based on multi-modal deep learning

Abstract

The application discloses a three-dimensional facial defect completion method based on a multi-modal deep learning, and belongs to the technical field of facial defect completion, and comprises the following steps: step 1, using a point cloud feature encoder to extract three-dimensional point cloud features of a facial defect patient; step 2, using an image feature encoder to extract two-dimensional image features of the facial defect patient; step 3, using a multi-modal feature fusioner to perform feature fusion on the extracted three-dimensional point cloud features and two-dimensional image features; step 4, using a three-dimensional face grid collected to construct a 3DMM model through a principal component analysis method, using a three-dimensional facial repair decoder to predict 3DMM coefficients according to features output by the multi-modal feature fusioner, and generating a repaired three-dimensional face grid; and step 5, using a three-dimensional supervision loss to capture global shape information, and simultaneously introducing a two-dimensional image supervision loss to capture local detail information, so as to complete the completion of the three-dimensional facial defect. The application improves the efficiency and quality of digital design of a prosthesis.

Description

Technical Field

[0001] This invention belongs to the field of facial defect completion technology, and in particular relates to a three-dimensional facial defect completion method based on multimodal deep learning. Background Technology

[0002] Oral and maxillofacial defects are commonly seen in oral clinical practice due to congenital defects, trauma, tumor resection, and other causes. Depending on the location of the defect, they can be classified as jawbone defects, facial defects, soft palate defects, and tongue defects, with an incidence rate as high as 2.4‰. These defects severely impact patients' physical and mental health, as well as their quality of life. Among oral and maxillofacial defects, facial defects often manifest as the loss or absence of facial organs such as the ears, nose, eyes, and orbits, resulting in facial deformities. The face is often referred to as the "scenic area" of the human body, and facial tissue defects often cause more severe physical and psychological trauma than other areas, easily leading to serious psychological problems, difficulty in normal work and social interaction, and even loss of confidence in life. Therefore, the main goal of facial defect repair in oral clinical practice is to restore the normal appearance of the patient's face as much as possible.

[0003] Currently, there are two main methods for facial defect repair: surgical restoration and prosthetic restoration. Surgical restoration uses the patient's own tissue transplantation, which has a good effect on the restoration of tissue structure and function. However, for many facial defects, such as those of the eyes, ears, and nose, surgical methods cannot achieve satisfactory results. In oral clinics, prosthetic restoration is often used for restoration. Compared with surgical methods, prosthetic restoration has advantages such as relatively lower cost and risk, no limitation on donor tissue, and no secondary surgical trauma. However, the traditional prosthetic fabrication process is complex, involving more than ten steps, including impression making, wax model making, and prosthetic processing. In particular, wax model making emphasizes the lifelike effect of the prosthetic, which requires restorative dentists and technicians to not only have sufficient medical knowledge and experience, but also high aesthetic literacy and sculpting skills. This makes prosthetic fabrication highly dependent on experience, with a long production cycle and a high failure rate, making it difficult to promote and popularize in primary care settings.

[0004] In recent years, with the development of digital technology, CAD (Computer-Aided Design) and CAM (Computer-Aided Manufacturing) have been widely used in dentistry for the design and fabrication of facial prostheses, and have been successfully applied in clinical dentistry, bringing new approaches to prosthesis fabrication. Currently, domestic and international research on the digital design of facial prostheses mainly uses mirroring and database methods to construct target reference data for different types of facial defects. However, these methods require extensive manual adjustments based on point cloud data topology to solve the problem of three-dimensional matching between the target reference data and the patient's personalized facial features. Existing digital technologies have not yet solved the key problem of automatically constructing optimal three-dimensional target reference data for facial defect repair. Summary of the Invention

[0005] The purpose of this invention is to provide a three-dimensional facial defect repair method based on multimodal deep learning, which solves the problems of existing methods relying on manual adjustments, requiring relatively professional oral medical knowledge and practical experience, and being difficult to promote at the grassroots level. At the same time, the two existing digital design methods for facial prostheses cannot effectively repair midline defects. The facial method cannot repair midline defects, and the database method is heavily dependent on the quality of the collected face database for midline defect repair.

[0006] To achieve the above objectives, this invention provides a three-dimensional facial defect repair method based on multimodal deep learning, including a point cloud feature encoder, an image feature encoder, a multimodal feature fusion unit, a three-dimensional facial repair decoder, and a discriminator based on differentiable rendering, comprising the following steps:

[0007] Step 1: Use a point cloud feature encoder to extract the 3D point cloud features of patients with facial defects;

[0008] Step 2: Use an image feature encoder to extract two-dimensional image features from patients with facial defects;

[0009] Step 3: Use a multimodal feature fusion tool to fuse the 3D point cloud features and 2D image features extracted in Step 1 and Step 2.

[0010] Step 4: Using the collected 3D face mesh, construct a 3DMM model using principal component analysis. Use a 3D face restoration decoder to predict the 3DMM coefficients based on the features output by the multimodal feature fusion unit, and generate the restored 3D face mesh.

[0011] Step 5: Use 3D supervised loss to capture global shape information. At the same time, based on the differentiable rendering discriminator, during training, the predicted 3D face mesh and the real 3D face mesh are rendered into images, and the 2D image supervised loss is introduced to capture local detail information, thus completing the 3D face defect repair.

[0012] Preferably, the point cloud feature encoder includes a point cloud Transformer module and an attention-based point cloud downsampling module; the attention-based point cloud downsampling module performs weighted aggregation of features by calculating the contribution of the center point's neighboring points to the center point's features, and the specific formula for weighted aggregation is as follows:

[0013]

[0014] Among them, X dp For the final downsampled point cloud, softmax is the softmax activation function, k and v are the key and value of the local point set, respectively, and q c Let T be the center point of the local point set, T be the transpose operation, and d be the number of channels being queried.

[0015] Preferably, the image feature encoder includes a ResNet residual module and a multi-scale fusion module; the specific formula for the multi-scale fusion module is as follows:

[0016]

[0017] Δ,M=Conv 3*3 (F)

[0018]

[0019] Where Unsample represents the upsampling operation, Concat represents the merging operation, and Conv represents the concatenation operation. k*k This represents a convolution with a kernel size of k*k, Δ represents the offset of the deformable convolution, M represents the mask of the deformable convolution, DCN represents the deformable convolution, and F... i This represents the output feature of ResNet in the i-th stage. This represents the upsampled feature of the output feature in the i-th stage. This represents the feature obtained by deformable convolution of the output feature of the (i-1)th stage. This represents the output characteristics of the multi-scale fusion module.

[0020] The preferred formula for the multimodal feature fusion machine is as follows:

[0021]

[0022] Where, q i This represents a query for the i-th modality, where i = 1; k iThe key represents the i-th mode, i = 1, 2; v i M represents the value of the i-th mode, where i = 1, 2; i This represents the i-th modal feature, where i = 1; This represents a refined modality 1 query feature; This represents the features of mode 1 after feature fusion.

[0023] Preferably, the formula for the 3DMM model constructed in step 4 is as follows:

[0024]

[0025] Where V represents the coordinates of all vertices on the 3D face mesh, α i S represents the i-th 3DMM coefficient. i It represents the basis of the i-th 3DMM model.

[0026] Preferably, the three-dimensional supervision loss L 3d Supervision loss L for 2D images 2d Total loss L total The specific expression for the total loss is as follows:

[0027] L total =L 3d (V pred V gt )+L 2d (Img pred 1mg gt )

[0028] Among them, V pred and V gt These are the point cloud coordinates on the restored 3D face mesh and the point cloud coordinates on the real 3D face mesh, respectively. pred and Img gt These are images obtained by differentiable rendering of the repaired 3D face mesh and the real 3D face mesh, respectively.

[0029] Therefore, the present invention employs the above-mentioned method for three-dimensional facial defect completion based on multimodal deep learning, which has the following beneficial effects:

[0030] (1) A 3DMM model with prior knowledge of faces was constructed by collecting 3D face meshes. This model can cover most of the morphological changes of faces and can be used to generate repaired 3D faces.

[0031] (2) A new multimodal three-dimensional face restoration network is proposed. By combining face three-dimensional point cloud information and face two-dimensional image information to jointly predict the restored three-dimensional face, the model can generate the optimal three-dimensional face target reference data that conforms to the personalized face features of patients with facial defects.

[0032] (3) No manual adjustment is required; personalized three-dimensional facial target reference data can be automatically generated for patients with facial defects.

[0033] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0034] Figure 1 This is an overall flowchart of a three-dimensional facial defect completion method based on multimodal deep learning according to the present invention;

[0035] Figure 2 This is the overall architecture diagram of the point cloud feature encoder of the present invention;

[0036] Figure 3 This is an architecture diagram of the attention-based point cloud downsampling module of this invention;

[0037] Figure 4 This is a diagram illustrating the overall architecture of the image feature encoder of this invention.

[0038] Figure 5 This is the overall architecture diagram of the multimodal feature fusion device of the present invention;

[0039] Figure 6 This is the overall architecture diagram of the three-dimensional facial restoration decoder of the present invention;

[0040] Figure 7 This is a diagram of the cross-attention mechanism architecture in the 3D facial restoration decoder of this invention;

[0041] Figure 8 This is a diagram of the user interface of the present invention. Detailed Implementation

[0042] The following detailed description of embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0043] Please see Figure 1-8 A method for 3D facial defect repair based on multimodal deep learning includes a point cloud feature encoder, an image feature encoder, a multimodal feature fusion unit, a 3D facial repair decoder, and a discriminator based on differentiable rendering. Specifically, it includes the following steps:

[0044] Step 1: Extract 3D point cloud features from patients with facial defects using a point cloud feature encoder. The point cloud feature encoder includes a point cloud Transformer module and an attention-based point cloud downsampling module. The attention-based point cloud downsampling module performs weighted aggregation of features by calculating the contribution of the center point's neighboring points to the center point's features. The specific formula for weighted aggregation is as follows:

[0045]

[0046] Among them, X dp For the final downsampled point cloud, softmax is the softmax activation function, k and v are the key and value of the local point set, respectively, and q c Let T be the center point of the local point set, T be the transpose operation, and d be the number of channels queried.

[0047] Step 2: Extract two-dimensional image features from patients with facial defects using an image feature encoder. The image feature encoder includes a ResNet residual module and a multi-scale fusion module. The specific formula for the multi-scale fusion module is as follows:

[0048]

[0049] Δ,M=Conv 3*3 (F)

[0050]

[0051] Where Unsample represents the upsampling operation, Concat represents the merging operation, and Conv represents the concatenation operation. k*k This represents a convolution with a kernel size of k*k, Δ represents the offset of the deformable convolution, M represents the mask of the deformable convolution, DCN represents the deformable convolution, and F... i This represents the output feature of ResNet in the i-th stage. This represents the upsampled feature of the output feature in the i-th stage. This represents the feature obtained by deformable convolution of the output feature of the (i-1)th stage. This represents the output features of the multi-scale fusion module;

[0052] Step 3: Use a multimodal feature fusion engine to fuse the 3D point cloud features and 2D image features extracted in Step 1 and Step 2; the specific formula for the multimodal feature fusion engine is as follows:

[0053]

[0054] Where, q i This represents a query for the i-th modality, where i = 1; k iThe key represents the i-th mode, i = 1, 2; v i M represents the value of the i-th mode, where i = 1, 2; i This represents the i-th modal feature, where i = 1; This represents a refined modality 1 query feature; This represents the features of mode 1 after feature fusion;

[0055] Step 4: Using the collected 3D face mesh, construct a 3DMM model using principal component analysis. Then, use a 3D face restoration decoder to predict the 3DMM coefficients based on the features output by the multimodal feature fusion unit, and generate the restored 3D face mesh. The formula for the constructed 3DMM model is as follows:

[0056]

[0057] Where V represents the coordinates of all vertices on the 3D face mesh, α i S represents the i-th 3DMM coefficient. i It represents the basis of the i-th 3DMM model.

[0058] Step 5: Use 3D supervised loss to capture global shape information. Simultaneously, during training, a differentiable-renderable discriminator renders the predicted 3D face mesh and the real 3D face mesh into images and introduces 2D image supervised loss to capture local detail information, thus completing the 3D facial defect repair. 3D supervised loss L 3d Supervision loss L for 2D images 2d Total loss L total The specific expression for the total loss is as follows:

[0059] L total =L 3d (V pred V gt )+L 2d (Img pred 1mg gt )

[0060] Among them, V pred and V gt These are the point cloud coordinates on the restored 3D face mesh and the point cloud coordinates on the real 3D face mesh, respectively. pred and Img gt These are images obtained by differentiable rendering of the repaired 3D face mesh and the real 3D face mesh, respectively.

[0061] Example

[0062] The collected 3D point cloud information from patients with facial defects is input into a point cloud feature encoder to obtain 3D facial features. The structure of the point cloud feature encoder is as follows: Figure 2As shown, to avoid information loss caused by simply using max pooling in point cloud downsampling, this invention proposes an attention-based point cloud downsampling method, such as... Figure 3 As shown, features are weighted and aggregated by calculating the contributions of the center point's neighbors to the center point's features. The specific formula for weighted aggregation is as follows:

[0063]

[0064] Among them, X dp For the final downsampled point cloud, softmax is the softmax activation function, k and v are the key and value of the local point set, respectively, and q c Let T be the center point of the local point set, T be the transpose operation, and d be the number of channels being queried.

[0065] The previous two-dimensional image information of the patient with facial defects is input into an image feature encoder to obtain two-dimensional facial features. The main component of this invention uses ResNet (Residual Network), in order to better utilize the image features from coarse to fine, such as... Figure 4 As shown, this invention proposes a novel multi-scale fusion module. It upsamples deep features rich in semantic information and then combines them with shallow features rich in detail information to predict the offset and mask of deformable convolutions for feature alignment during image feature fusion. Finally, the predicted deformable convolutions are applied to the shallow features and added to the deep features to obtain multi-scale image features. The specific formula for the multi-scale fusion module is as follows:

[0066]

[0067] Δ,M=Conv 3*3 (F)

[0068]

[0069] Where Unsample represents the upsampling operation, Concat represents the merging operation, and Conv represents the concatenation operation. k*k This represents a convolution with a kernel size of k*k, Δ represents the offset of the deformable convolution, M represents the mask of the deformable convolution, DCN represents the deformable convolution, and F... i This represents the output feature of ResNet in the i-th stage. This represents the upsampled feature of the output feature in the i-th stage. This represents the feature obtained by deformable convolution of the output feature of the (i-1)th stage. This represents the output characteristics of the multi-scale fusion module.

[0070] Compared to single-modal features, multimodal feature fusion can utilize rich facial details in images as auxiliary information for 3D facial information, reducing the possible solution space, such as... Figure 5 As shown, this paper proposes a novel feature-level fusion multimodal feature fusion engine, which aims to combine image features encoded by an image feature encoder with point cloud features encoded by a point cloud feature encoder to provide better feature representation for a 3D face restoration decoder, thereby improving the accuracy of 3D face restoration tasks. The specific formula of the multimodal feature fusion engine is as follows:

[0071]

[0072] Where, q i This represents a query for the i-th modality, where i = 1; k i The key represents the i-th mode, i = 1, 2; v i M represents the value of the i-th mode, where i = 1, 2; i This represents the i-th modal feature, where i = 1; This represents a refined modality 1 query feature; This represents the features of mode 1 after feature fusion.

[0073] Most existing point cloud completion decoders first use the global features output by the point cloud feature encoder to directly predict the complete sparse point cloud through a multilayer perceptron. Then, they use a folding mechanism to amplify local points by a certain factor, forming the final complete dense point cloud. This method does not consider the constraints between different parts of the face, resulting in a rough surface and a few outliers in the repaired face. Furthermore, it is difficult to restore the face shape when the missing area is large. To address these problems, this invention introduces a parametric face model into the 3D face restoration task, transforming the problem of predicting point cloud positions into predicting identity parameters in the parametric face model, thus introducing statistical prior knowledge of faces into the 3D face restoration task. This invention first constructs a 3DMM model with face prior knowledge from the collected 3D face mesh using principal component analysis, and then... Figure 6 and Figure 7 The 3D face restoration decoder shown predicts the coefficients of the 3DMM and generates the restored 3D face mesh. The formula for the constructed 3DMM model is as follows:

[0074]

[0075] Where V represents the coordinates of all vertices on the 3D face mesh, α i S represents the i-th 3DMM coefficient. i It represents the basis of the i-th 3DMM model.

[0076] Because supervised learning using only the 3D loss function of the point cloud on the 3D face mesh restored by the 3D face restoration decoder and the point cloud on the real 3D face mesh would cause the model to fail to capture local details, this invention introduces a discriminator based on differentiable rendering during the training phase. This discriminator performs differentiable rendering on both the restored 3D face mesh and the real 3D face mesh, and supervises the learning of the rendered images at both the 2D pixel level and the perceptual level. This module can be discarded during the inference phase. The total training loss function is as follows:

[0077] L total =L 3d (V pred V gt )+L 2d (Img pred 1mg gt )

[0078] Among them, V pred and V gt These are the point cloud coordinates on the restored 3D face mesh and the point cloud coordinates on the real 3D face mesh, respectively. pred and Img gt These are images obtained by differentiable rendering of the repaired 3D face mesh and the real 3D face mesh, respectively.

[0079] The final user interface is as follows Figure 8 As shown, the left side displays the model parameter selection and data preprocessing settings, the middle section shows the network input, including 3D point cloud of the defective face and 2D face image, and the right side displays the network output, i.e. the repaired 3D face.

[0080] Therefore, this invention employs a three-dimensional facial defect completion method based on multimodal deep learning, which eliminates the need for manual adjustments and automatically predicts the prosthesis morphology of the facial defect area in a personalized manner based on the morphological characteristics of the defective face. This effectively improves the efficiency and quality of digital design and fabrication of facial prostheses in clinical dentistry, promotes the widespread adoption of digital facial defect repair technology at the grassroots level, and enhances the prosthesis design skills of primary care physicians. On the other hand, it also provides dental experts with an efficient and convenient reference scheme for three-dimensional prosthesis morphology design, demonstrating significant scientific research significance and clinical application value.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for 3D facial defect repair based on multimodal deep learning, comprising a point cloud feature encoder, an image feature encoder, a multimodal feature fusion unit, a 3D facial repair decoder, and a discriminator based on differentiable rendering, characterized in that, Includes the following steps: Step 1: Use a point cloud feature encoder to extract the 3D point cloud features of patients with facial defects; Step 2: Use an image feature encoder to extract two-dimensional image features from patients with facial defects; Step 3: Use a multimodal feature fusion tool to fuse the 3D point cloud features and 2D image features extracted in Step 1 and Step 2. The specific formula for the multimodal feature fusion machine is as follows: Where, q i This represents a query for the i-th modality, where i = 1; k i The key represents the i-th mode, i = 1, 2; v i M represents the value of the i-th mode, where i = 1, 2; i This represents the i-th modal feature, where i = 1; This represents a refined modality 1 query feature; This represents the features of mode 1 after feature fusion; Step 4: Using the collected 3D face mesh, construct a 3DMM model using principal component analysis. Use a 3D face restoration decoder to predict the 3DMM coefficients based on the features output by the multimodal feature fusion unit, and generate the restored 3D face mesh. Step 5: Use 3D supervised loss to capture global shape information. At the same time, based on the differentiable rendering discriminator, during training, the predicted 3D face mesh and the real 3D face mesh are rendered into images, and the 2D image supervised loss is introduced to capture local detail information, thus completing the 3D face defect repair.

2. The method for three-dimensional facial defect completion based on multimodal deep learning according to claim 1, characterized in that: The point cloud feature encoder includes a point cloud Transformer module and an attention-based point cloud downsampling module. The attention-based point cloud downsampling module performs weighted aggregation of features by calculating the contribution of the center point's neighboring points to the center point's features. The specific formula for weighted aggregation is as follows: Among them, X dp For the final downsampled point cloud, softmax is the softmax activation function, k and v are the key and value of the local point set, respectively, and q c Let T be the center point of the local point set, T be the transpose operation, and d be the number of channels being queried.

3. The method for three-dimensional facial defect completion based on multimodal deep learning according to claim 2, characterized in that: The image feature encoder includes a ResNet residual module and a multi-scale fusion module; the specific formula for the multi-scale fusion module is as follows: Δ,M=Conv 3*3 (F) Where Unsample represents the upsampling operation, Concat represents the merging operation, and Conv represents the concatenation operation. k*k This represents a convolution with a kernel size of k*k, Δ represents the offset of the deformable convolution, M represents the mask of the deformable convolution, DCN represents the deformable convolution, and F... i This represents the output feature of ResNet in the i-th stage. This represents the upsampled feature of the output feature in the i-th stage. This represents the feature obtained by deformable convolution of the output feature of the (i-1)th stage. This represents the output characteristics of the multi-scale fusion module.

4. The method for three-dimensional facial defect completion based on multimodal deep learning according to claim 3, characterized in that: The formula for the 3DMM model constructed in step 4 is as follows: Where V represents the coordinates of all vertices on the 3D face mesh, α i S represents the i-th 3DMM coefficient. i It represents the basis of the i-th 3DMM model.

5. The method for three-dimensional facial defect completion based on multimodal deep learning according to claim 4, characterized in that: 3D supervision loss L 3d Supervision loss L for 2D images 2d Total loss L total The specific expression for the total loss is as follows: L total =L 3d (V pred ,V gt )+L 2d (Img pred ,Img gt ) Among them, V pred and V gt These are the point cloud coordinates on the restored 3D face mesh and the point cloud coordinates on the real 3D face mesh, respectively. pred and Img gt These are images obtained by differentiable rendering of the repaired 3D face mesh and the real 3D face mesh, respectively.

Images