High-definition tooth restoration method and device based on GAN network

Abstract

The invention discloses a high-definition tooth restoration method and device based on a GAN network. The high-definition tooth restoration method comprises the following steps: extracting facial key points from an acquired facial image, determining a tooth contour and a tooth mask based on the facial key points, segmenting a mouth reference image from the facial image, and carrying out image enhancement and refinement processing on the mouth reference image. And superposing the tooth contour and the tooth mask image to obtain a tooth image to be restored. Finally, performing normalization processing on the tooth image to be repaired and the mouth region reference image as model input; and constructing a high-definition tooth restoration model based on the GAN network, and performing high-definition tooth restoration by using the high-definition tooth restoration model. According to the method and the device, the generation quality of the TFG method in the tooth area can be improved, and high-precision synchronization of the mouth shape and the voice is ensured, so that the requirements of applications such as virtual reality, augmented reality and film and television production on high-quality face generation are met.

Description

Technical Field

[0001] The present invention belongs to the technical field of combining facial image processing and enhancement, and specifically relates to a high-definition tooth restoration method and device based on a GAN network. Background Art

[0002] With the rapid development of artificial intelligence technology, Talking Face Generation (TFG) technology has become an important research direction in the field of human-computer interaction. TFG technology aims to generate facial animation synchronized with speech by analyzing audio signals and combining facial features. It is widely used in scenarios such as virtual assistants, video conferencing, games, and film production. This technology can significantly improve the realism and affinity of computer-generated characters by simulating human facial expressions and lip shape changes.

[0003] The traditional method of producing digital humans first requires complex 3D modeling, and then drives the movement of digital humans through motion capture or keyframe animation. This method is not only time-consuming and costly, but also difficult to adapt to the rapidly changing market needs, especially in production environments that require a large amount of personalized content. In addition, traditional methods also have limitations in dealing with lip sync and the naturalness of facial expressions, and often fail to achieve satisfactory results.

[0004] In recent years, with the development of deep learning technologies such as Generative Adversarial Networks (GANs) and autoregressive models, TFG methods based on generative models have begun to emerge. These methods can automatically learn the laws of facial movements from a large amount of facial videos and corresponding audio data, and then generate realistic facial animations. However, existing TFG methods still face challenges in generating high-quality visual content, especially in terms of clarity and detail in the dental area.

[0005] Teeth are an important part of the face, and their clarity is crucial to improving the realism of face generation. However, in existing TFG research, the generation effect of the tooth area is often unsatisfactory, with problems such as blurring, distortion, or inconsistency with real tooth features. This is mainly because the high-frequency detail information of the tooth area is difficult to capture in facial images, and the dynamic changes of the tooth area during facial movement are complex, making it difficult for existing models to accurately predict and generate high-quality tooth details.

[0006] In addition, the existing TFG method also has shortcomings in processing lip shape and speech synchronization. The naturalness and realism of facial animation depends not only on the accuracy of facial expressions, but also on the precise matching of lip shape and speech. However, in practical applications, due to errors in audio signal processing and facial feature extraction, as well as the limited learning ability of the model for complex speech information, the generated lip shape is often out of sync with the input speech.

[0007] In order to solve the above problems, researchers have tried a variety of improvement strategies, such as introducing multi-scale feature fusion, using attention mechanism, optimizing model structure, etc. These methods have improved the quality of face generation to a certain extent, but there is still much room for improvement in tooth clarity and lip synchronization. Therefore, developing a TFG method that can generate high-definition teeth and maintain lip synchronization is of great significance to promote the development of virtual digital human technology. Summary of the invention

[0008] In view of the above, the purpose of the present invention is to provide a high-definition tooth restoration method and device based on a GAN network, which can effectively improve the generation quality of the TFG method in the tooth area and ensure high-precision synchronization of lip shape and speech, so as to meet the needs of high-quality facial generation in applications such as virtual reality, augmented reality, and film and television production.

[0009] To achieve the above-mentioned purpose of the invention, the embodiment provides a high-definition tooth restoration method based on a GAN network, comprising the following steps:

[0010] Extract facial key points from the acquired facial image, determine tooth contours and tooth masks based on the facial key points, superimpose the tooth contour and tooth mask images to obtain a tooth image to be restored, segment the mouth area from the facial image as a reference image, enhance and refine the mouth area reference image, and then normalize the tooth image to be restored and the mouth area reference image;

[0011] A high-definition tooth restoration model is constructed based on a GAN network, wherein the generator contained in the GAN network adopts an encoding structure and a decoding structure. The encoding structure uses two branches to extract and encode features of the normalized tooth image to be restored and the reference image of the mouth area respectively. The decoding structure decodes the encoding results of the two branches to perform tooth restoration to obtain the restored tooth image. The discriminator contained in the GAN network is used to distinguish the authenticity of the restored tooth image. The generator after the two-stage training is used as the high-definition tooth restoration model.

[0012] High-definition dental restorations are performed using high-definition dental restoration models.

[0013] Preferably, the MobileNetV3 network is used to extract facial key points, and the extracted facial key points include mouth contour points and relevant feature points of the tooth area. The tooth contour and tooth mask are determined based on the tooth contour points, and the mouth area reference image M is segmented from the facial image. reference .

[0014] Preferably, when determining the tooth contour, firstly, a contour generation function C is used based on the facial key point set P facial Generate tooth contour image M tooth_contour :

[0015] M tooth_contour =C(P facial )

[0016] Through the mask generation function M, the facial key point set P facial Perform mask processing to obtain the initial tooth mask image M mask :

[0017] M mask =M(P facial )

[0018] Perform an XOR operation on the initial tooth mask image and the mouth area image reference image, so that the tooth area of ​​the mouth area reference image forms a mask, and the final tooth mask image is obtained.

[0019] M tooth_mask =U(M mask ,M reference )

[0020] Wherein, U represents the image XOR operation.

[0021] The tooth contour image and the tooth mask image are superimposed to obtain the tooth image to be repaired M tooth .

[0022] Preferably, the mouth area reference image is enhanced and refined, the tooth image to be repaired and the mouth area reference image are normalized, contrast enhancement and sharpening are performed during image enhancement, edge sharpening and smoothing are performed during refinement, and the pixel values ​​of the image are normalized to between [0, 1] or [-1, 1] during normalization.

[0023] Preferably, the two branch structures adopted by the encoding structure are the same, each branch structure includes a feature extraction module and a fine-grained feature fusion module, and the two models are used to perform feature extraction and fine-grained feature fusion to realize the encoding process. The fine-grained feature fusion module specifically includes channel fusion, HourGlass network, feature fusion, and feature refinement, wherein channel fusion refers to gradually reducing the number of channels of the feature map through convolution operations, while increasing the depth of the features to extract finer features, the HourGlass network is used to extract multi-scale features, feature fusion refers to feature fusion of the output results of the channel fusion pair with the extracted multi-scale features, and feature refinement refers to enhancing the details of the feature fusion results as the encoding results through convolution and activation operations.

[0024] Preferably, each fine-grained feature fusion module includes a convolutional layer, a maximum pooling layer, three channel fusions, an HourGlass network, a channel fusion, and feature refinement connected in sequence.

[0025] The HourGlass network uses two branches. One branch extracts fine features by channel fusion through channel indentation, and the other branch extracts multi-scale features by combining two pairs of maximum pooling and channel fusion, and upsampling. The fine features and multi-scale features are then fused to obtain the feature fusion result.

[0026] Feature refinement adopts at least one depth-wise separable convolution or at least one residual connection constructed between depth-wise separable convolutions.

[0027] Preferably, the loss function L used when training the GAN network is total Including the adversarial loss of the generator Reconstruction loss of the generator The perceptual loss L of the generator perc , the discriminant loss L of the discriminator D , and the discriminator's R1 loss L R1 ,

[0028]

[0029] Among them, λ1, λ2, λ3, and λ4 represent the weights of each loss, and λ is a hyperparameter used to control the strength of R1 loss. is the distribution between the tooth reference image and the restored tooth image. The sampling is obtained, is the discriminator pair The judgment result of Yes The gradient of ||·||2 represents the L2 norm, which is used to calculate the size of the gradient vector, and E represents the expected value. to calculate.

[0030] To achieve the above-mentioned purpose, the embodiment of the present invention provides a high-definition tooth restoration device based on a GAN network, including: an image processing module, a model building module, and a tooth restoration module.

[0031] The image processing module is used to extract facial key points from the acquired facial image, determine the tooth contour and tooth mask based on the facial key points, superimpose the tooth contour and tooth mask images to obtain the tooth image to be restored, segment the mouth area from the facial image as a reference image, perform image enhancement and refinement processing on the mouth area reference image, and then perform normalization processing on the tooth image to be restored and the mouth area reference image;

[0032] The model construction module is used to construct a high-definition tooth restoration model based on a GAN network, wherein the generator included in the GAN network adopts an encoding structure and a decoding structure, the encoding structure adopts two branches to respectively extract and encode features of the normalized tooth image to be restored and the reference image of the mouth area, the decoding structure decodes the encoding results of the two branches to perform tooth restoration to obtain a restored tooth image, the discriminator included in the GAN network is used to distinguish the authenticity of the restored tooth image, and the generator after the two-stage training is used as the high-definition tooth restoration model;

[0033] The tooth restoration module is used to perform high-definition tooth restoration using a high-definition tooth restoration model.

[0034] To achieve the above-mentioned purpose of the invention, an embodiment further provides a computing device, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned high-definition tooth restoration method based on the GAN network.

[0035] In order to achieve the above-mentioned purpose of the invention, the embodiment further provides a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, the above-mentioned high-definition tooth restoration method based on the GAN network is implemented.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] More accurate and efficient facial key point detection method: Through the key point detection method of the MobileNetV3 network architecture, it can detect facial key points with high precision and achieve real-time facial tracking on ordinary consumer-grade hardware, which is suitable for application scenarios that require fast response. It can adapt to different lighting conditions and facial expressions, and maintain high tracking accuracy even in complex environments.

[0038] High-definition detail generation: High-definition restoration is performed specifically on the details of the tooth area. The visual clarity and realism of the teeth are significantly improved through the advanced fine-grained feature fusion (FGFF) module and optimized loss function in the generator of the GAN network.

[0039] Synchronicity and continuity: While enhancing the clarity of teeth, the present invention maintains the synchronization between lip shape and speech, ensures the natural and smooth animation, and improves the user experience.

[0040] Generalization ability: Through a two-stage fine-tuning strategy, the present invention can adapt to different facial features and expressions, enhancing the generalization ability of the model and enabling it to work stably in a variety of scenarios.

[0041] Computational efficiency: The present invention focuses on computational efficiency during design. Through optimized network structure and parallel processing technology, it achieves fast image generation speed and meets the needs of real-time applications.

[0042] Personalized customization: The present invention allows for personalized customization. Through fine-tuning of specific tasks, it can be optimized for specific characters or scenarios to meet the specific needs of users.

[0043] Robustness: Through specific data preprocessing and enhancement techniques, the present invention has high robustness to noise and variation of input data, and can stably generate high-quality results on data of different qualities. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 is a flow chart of a high-definition tooth restoration method based on a GAN network provided in an embodiment;

[0046] Figure 2 is a schematic diagram of the structure of a generator provided in an embodiment;

[0047] Figure 3 Schematic diagram of the structure of a high-definition tooth restoration device based on a GAN network provided in an embodiment. DETAILED DESCRIPTION

[0048] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0049] The inventive concept of the present invention is to solve the deficiencies of the prior art in terms of tooth clarity and lip synchronization. The present invention provides a high-definition tooth restoration method based on a GAN network, which focuses on significantly improving the visual quality of the tooth area in the speech-driven face generation (TFG) technology, that is, the effect of high-definition restoration of the tooth area, while ensuring high synchronization and temporal coherence with the speech. First, in the data preprocessing stage, through the precise extraction of facial key points and the generation of masks based on key points, the details of the tooth area are enhanced to avoid blurring of the lip edges and provide standardized input data for the model. Then, in the pre-training stage, the model learns the universal representation of facial features and movements on a large-scale facial action dataset, and uses a variety of loss functions to optimize the generation quality and synchronization of the model. Most importantly, in the specific person fine-tuning stage, HDTR makes detailed adjustments to individual features, uses the data of specific people, learns their unique tooth shapes and facial features, and further optimizes the loss function to adapt to individual differences. The fine-tuning at this stage significantly improves the performance of the model in mouth image reconstruction and enhances the ability to retain tooth clarity and image details.

[0050] like Figure 1 As shown, the embodiment provides a high-definition tooth restoration method based on a GAN network, comprising the following steps:

[0051] S1, extract facial key points from the acquired facial image, determine the tooth contour and tooth mask based on the facial key points, superimpose the tooth contour and tooth mask images to obtain the tooth image to be repaired, segment the mouth area from the facial image as a reference image, enhance and refine the mouth area reference image, and then normalize the tooth image to be repaired and the mouth area reference image.

[0052] In the embodiment, facial images are obtained from a camera or other video source, and these images usually contain RGB color channels. The MobileNetV3 network is used to extract facial key points from the facial images, specifically including the three-dimensional coordinates of some key points such as eyes, eyebrows, nose, and mouth.

[0053] Specifically, the open source real-time face and facial feature point tracking library OpenSeeFace is used. It is based on the MobileNetV3 network architecture. By converting the model to ONNX format and using onnxruntime for inference, it achieves high-speed single face tracking of 30-60fps on the CPU, significantly reducing the demand for hardware. OpenSeeFace uses a unique 66-point facial feature point model, which can maintain stable tracking effects even under adverse conditions such as low light, high noise, and low resolution, and performs particularly well in head posture changes and mouth motion capture.

[0054] The network structure of MobileNetV3 is the latest achievement of the MobileNet series of lightweight deep learning models. The network structure can be divided into three main parts:

[0055] Starting part: It is a convolution layer with a larger convolution kernel, which is used to capture the preliminary features of the input image. In MobileNetV3, the starting part usually contains a 3x3 convolution layer, followed by a batch normalization (BN) layer and an activation function.

[0056] Middle part: This is the core of the network composed of multiple inverted residual blocks, each of which can contain an expansion layer, a depthwise separable convolution layer, and an output layer. Depthwise separable convolution is the core feature of the MobileNet series, which decomposes the standard convolution into depthwise convolution and pointwise convolution. In MobileNetV3, this structure is further optimized to improve efficiency. The general form of the inverted residual structure can be expressed as:

[0057] out1=Depthwise(Conv(Expand(input)))

[0058] out2 = Pointwise(out1)

[0059] out3 = Activation (out2)

[0060] Among them, Expand, Depthwise, and Pointwise represent the expansion layer, depth convolution layer, and point-by-point convolution layer respectively, and Activation represents the activation function.

[0061] Last part: In the last part of the network, there are one or more 1x1 convolutional layers to convert the feature map into the final output. In classification tasks, this is usually a global average pooling layer followed by a 1x1 convolutional layer, and finally a fully connected layer to output the class prediction. This process can be expressed as follows:

[0062] out4=GlobalAvgPool(Conv(out3))

[0063] out5=FC(out4)

[0064] Among them, GlobalAvgPool represents the global average pooling layer, Conv represents the last 1x1 convolutional layer, and FC represents the fully connected layer.

[0065] In the inverted residual structure of MobileNetV3, a new activation function h-swish is used, and the sigmoid function in the SE module is approximated to improve computational efficiency. In addition, MobileNetV3 also introduces the NetAdapt algorithm to optimize some layers of the network and redesigns the time-consuming structure in the network. OpenSeeFace converts the MobileNetV3 model to ONNX format and uses onnxruntime for inference to achieve efficient operation on the CPU.

[0066] The facial key points extracted based on the MobileNetV3 network include mouth contour points and relevant feature points of the tooth area. The embodiment determines the tooth contour and tooth mask based on the tooth contour points.

[0067] Tooth region segmentation and image enhancement directly affect the performance of the model and the final restoration effect. The present invention improves the traditional lip mask into a mask only for teeth through innovative data processing methods, effectively eliminating the blurring of the lip edges, while significantly enhancing the detail clarity of the tooth region. The key steps include accurate segmentation of the tooth region, appropriate application of image enhancement technology, and data normalization processing, ensuring that the data set is both rich and suitable for model training. In addition, by fine-tuning the reference image of the mouth region, the image quality is further improved, and the input data of the model is optimized, thereby improving the generalization ability of the model and the restoration effect, providing a solid data foundation for generating high-definition and high-synchronization tooth regions.

[0068] In the embodiment, the tooth contour and the tooth mask are determined based on the extracted facial key points, and the mouth area reference image M is generated. reference Specifically include: through the contour generation function C based on the facial key point set P facialGenerate a tooth outline image:

[0069] M tooth_contour =M(P facial )

[0070] The tooth area is masked by the mask generation function M to obtain the initial tooth mask image M mask :

[0071] M mask =M(P facial )

[0072] Perform an XOR operation on the initial tooth mask image and the mouth area image reference image so that the tooth area of ​​the mouth area reference image forms a mask, and obtain the final tooth mask image:

[0073] M tooth_mask =U(M mask ,M reference )

[0074] Wherein, U represents the image XOR operation.

[0075] The tooth contour image and the tooth mask image are superimposed to obtain the tooth image to be repaired M tooth .

[0076] The reference image of the mouth area is enhanced and refined, and the image of the tooth to be restored and the reference image of the mouth area are normalized. Contrast enhancement and sharpening are performed during image enhancement, and edge sharpening and smoothing are performed during refinement. During normalization, the pixel values ​​of the image are normalized to between [0, 1] or [-1, 1]. The formula is:

[0077] I enhanced =ε(M reference ,α)

[0078] Among them, I enhanced is the enhanced reference image of the mouth area, M reference is the reference image of the mouth area, ε is the enhancement function, and α is the parameter that controls the degree of enhancement.

[0079] Refinement processing refers to the refinement of the mouth area reference image, including edge sharpening and smoothing to eliminate possible artifacts or unnatural effects. The formula is:

[0080] I refined =S(I enhance ,β)

[0081] Among them, I refined is the reference image of the mouth area after refinement, S is the refinement function, β is the parameter that controls the processing intensity, and I enhanceRepresents the enhanced mouth area reference image.

[0082] The normalization process is to make the input data suitable for model training. The enhanced mouth area reference image and the tooth to be repaired image are normalized so that their pixel values ​​range between [0, 1] or [-1, 1].

[0083] In the embodiment, the image of the tooth to be restored and the reference image of the mouth area are used as input images of the high-definition tooth restoration model.

[0084] S2, a high-definition tooth restoration model is constructed based on the GAN network to repair the image of the tooth to be restored.

[0085] In an embodiment, a generator included in a GAN network adopts a coding structure and a decoding structure. The coding structure adopts two branches to extract and encode features of the image of the tooth to be repaired and the reference image of the mouth area respectively. The decoding structure decodes the coding results of the two branches to perform tooth restoration to obtain a repaired tooth image. The discriminator included in the GAN network is used to determine the authenticity of the repaired tooth image. The trained generator is used as a high-definition tooth restoration model.

[0086] Specifically, the two branch structures used in the encoding structure are the same. Each branch structure includes a feature extraction module and a fine-grained feature fusion module (Fine-Grained Feature Fusion, FGFF). The feature extraction module extracts features based on a convolutional neural network, specifically extracting rich features related to teeth and surrounding areas. The convolutional neural network specifically includes convolution operations, feature dimension reduction, pooling layers, residual connections, multi-scale feature fusion, and regularization technology.

[0087] Specifically, in a convolutional neural network, each convolutional layer that implements the convolution operation uses a set of convolution kernels (or filters) to scan the input image, capture local features, and output a feature map. As the network level deepens, the spatial dimensions (i.e., height and width) of the feature map gradually decrease, while the depth (i.e., number of channels) of the feature gradually increases. This design helps to extract higher-level semantic information from the original pixel-level information. Pooling layers (such as maximum pooling or average pooling) are used between consecutive convolutional layers to reduce the spatial size of the feature map, thereby reducing the number of parameters and the amount of computation, while making feature detection more robust. In order to solve the problem of gradient vanishing or gradient exploding in deep networks, residual connections are introduced between convolutional layers. Residual connections allow information to be directly passed from earlier layers to later layers. In the final stage of feature extraction, feature fusion technology is used to merge feature maps at different levels to restore the spatial resolution loss caused by the pooling operation while retaining multi-scale feature information. To prevent overfitting, regularization techniques such as dropout or batch normalization may also be included in the feature extraction layer.

[0088] Through these carefully designed network structures and operations, the feature extraction layer can efficiently extract multi-level features that are crucial for dental restoration from facial images, laying a solid foundation for subsequent fine-grained feature fusion and high-definition tooth generation.

[0089] The fine-grained feature fusion module is the core part, which is used to extract and fuse the high-frequency detail features of teeth and their surrounding areas. The specific steps include channel fusion (CF), HourGlass network, feature fusion, and feature refinement. Channel fusion refers to gradually reducing the number of channels of the feature map through convolution operations, while increasing the depth of the features to extract finer features. The HourGlass network is used to extract multi-scale features to capture features from local to global. Feature fusion refers to the fusion of the output results of channel fusion with the extracted multi-scale features. Feature refinement is used to improve the fine-grainedness of the features, which refers to enhancing the details of the feature fusion results through convolution and activation operations as the encoding result.

[0090] In one embodiment, Figure 2As shown, each fine-grained feature fusion module includes a sequentially connected convolution layer, a maximum pooling layer, three channel fusions, an HourGlass network, a channel fusion, and feature refinement, wherein the HourGlass network uses two branches, one branch extracts fine features by channel fusion through channel indentation, and the other branch sequentially undergoes a combination of two pairs of maximum pooling and channel fusion, and upsampling to extract multi-scale features, and the fine features and multi-scale features are fused to obtain a feature fusion result; feature refinement uses at least one depth-wise separable convolution or at least one residual connection constructed between depth-wise separable convolutions.

[0091] Through these steps, the fine-grained feature fusion module can effectively extract and fuse the fine-grained features of the tooth area, providing rich feature support for the tooth restoration task. This fine-grained feature extraction and fusion strategy is the key to achieving high-definition tooth restoration.

[0092] In the embodiment, the encoding results of the tooth image to be repaired and the mouth area reference image obtained by the two branches are superimposed and input into the decoding structure, and the repaired tooth image is obtained after decoding.

[0093] In the embodiment, the discriminator is composed of a series of convolutional layers and activation functions, and its purpose is to provide a binary output to determine whether the input image is a real image or an image generated by the generator.

[0094] When training the above GAN, the loss function L is used total Including the adversarial loss of the generator Reconstruction loss of the generator The perceptual loss L of the generator perc , the discriminant loss L of the discriminator D , and the discriminator's R1 loss L R1 .

[0095] Fighting Losses Used to encourage the generator to produce images that are difficult for the discriminator to distinguish, reconstruction loss Ensure that the generator output is close to the real image at the pixel level. Perceptual loss L perc Use a pre-trained network (such as VGG) to measure the difference between the generated image and the real image at the feature level. D Refers to training the discriminator to distinguish between real images and restored tooth images generated by the generator.

[0096] R1 loss L R1It is a regularization technique used to stabilize and improve the GAN training process. This loss function is implemented by penalizing the gradients of the discriminator, especially when these gradients are close to zero, that is, the discriminator is too sensitive to small changes in the input image. R1 loss helps avoid overfitting the discriminator to the generated images and ensures that the generator has enough space to generate diverse images.

[0097]

[0098] Among them, λ is a hyperparameter used to control the strength of R1 loss. is the distribution between the tooth reference image and the restored tooth image. The sampling is obtained, is the discriminator pair The judgment result of Yes The gradient of ||·||2 represents the L2 norm, which is used to calculate the size of the gradient vector, and E represents the expected value. to calculate.

[0099] Then the total loss function L total It is expressed as:

[0100]

[0101] Among them, λ1, λ2, λ3, and λ4 represent the loss weights of each loss. By minimizing the total loss function L total , GAN can generate realistic images while maintaining the stability and effectiveness of the discriminator. The design of this comprehensive loss function helps to balance the generation quality and discrimination ability of the network, and promotes the generator to produce more realistic and diverse outputs.

[0102] In the embodiment, a two-stage training strategy is adopted. It includes pre-training in the first stage and fine-tuning in the second stage. In the first stage of the high-definition dental restoration model, namely the pre-training stage, the model is trained using a large-scale facial image dataset to learn common facial features and structures. The goal of this stage is to allow the model to capture the key points and expression changes of the face, laying the foundation for high-definition restoration of the dental area. By using standardized training techniques, the model establishes a preliminary understanding of facial details at this stage, which includes learning the general features of tooth shape and surrounding tissues, preparing conditions for subsequent targeted fine-tuning. Pre-training does not involve the optimization of specific tasks, but focuses on the acquisition of broad features and the enhancement of model stability.

[0103] The model is learned on a large-scale facial action dataset through a series of sophisticated steps, including initializing the generator and discriminator networks, inputting preprocessed key point sequences and corresponding video frames, and defining a loss function to optimize model performance. The generator is responsible for generating realistic teeth images, while the discriminator evaluates the authenticity of the generated images. In adversarial training, the GAN loss Guide the adversarial process between the generator and the discriminator, and reconstruct the loss Ensuring pixel-level consistency, perceptual loss High-level feature differences are captured through the VGG network. The autoregressive architecture enhances the temporal continuity of video frames, and the optical flow enhancement technology improves the authenticity of dynamic changes through the multi-scale dynamic discriminator. The texture adhesion improvement scheme uses aliasing-free convolution and global translation and rotation transformation modules to further improve the quality of generated videos. Backpropagation and optimization algorithms (such as Adam) are used to update network parameters. Performance evaluation is performed through quantitative indicators (PSNR, SSIM) and qualitative analysis. Hyperparameter adjustment ensures the efficiency and stability of model training. Specialized training of the multi-scale dynamic discriminator improves the sensitivity to dynamic changes, and the testing of model stability and generalization ability ensures the robustness of the model on diverse data. Finally, the integrated training of innovative modules and the preservation of pre-trained models lay the foundation for subsequent fine-tuning and application, making HDTR show a wide range of application potential and excellent performance in the field of digital human video generation.

[0104] The second phase of the HD dental restoration model focuses on task-specific fine-tuning, where the model is carefully adjusted to ensure superior performance in HD dental restoration. Task-specific datasets that are closely related to HD dental restoration are selected, which may contain HD facial images of specific people, especially those with high definition in the dental area. These samples specifically contain facial features and dental details of specific people. Using this selected data, the model is carefully adjusted based on pre-training to optimize its performance on specific tasks. The fine-tuning process begins with subtle adjustments to the model parameters to adapt to the characteristics of the new dataset. The model is then iteratively trained using suitable loss functions, such as perceptual loss and reconstruction loss, to further improve the clarity and realism of the dental area. The calculation of the loss function exploits both pixel-level differences and high-level feature differences to ensure the visually high quality of the generated images.

[0105] In the fine-tuning stage, it is necessary to reduce the learning rate, use optimizers such as Adam, and adjust their parameters to ensure that the model can carefully adapt to the new data set while avoiding over-adjustment of pre-trained parameters. Through the small batch gradient descent method, the model parameters are continuously updated to minimize the loss function and improve the restoration effect. Through these meticulous adjustments, the second stage of fine-tuning ensures the professional performance of the model in high-definition tooth restoration, improves the visual quality and detail clarity of the tooth area, and meets the specific needs of high-definition tooth generation.

[0106] During the fine-tuning process, the performance of the model on the training set was monitored by the validation set to evaluate its generalization ability. The restoration effect of the model was measured by quantitative indicators such as PSNR and SSIM, as well as qualitative evaluations such as visual inspection of the clarity and naturalness of the generated images. Ultimately, the fine-tuned model showed significant improvements in dental details, lip synchronization, and overall visual quality, enabling it to generate high-definition dental video sequences that are highly consistent with the facial movements of specific people.

[0107] S3, high-definition tooth restoration using high-definition tooth restoration models.

[0108] The facial image is processed by S1 to obtain a tooth image to be repaired and a mouth area reference image, and the tooth image to be repaired and the mouth area reference image are input into a high-definition tooth restoration model, and the restored tooth image is obtained by decoding with an encoder.

[0109] like Figure 3 As shown, the embodiment also provides a high-definition tooth restoration device 30 based on the GAN network, including an image processing module 31, a model building module 32, and a tooth restoration module 33, wherein the image processing module 31 is used to extract facial key points from the acquired facial image, determine the tooth contour and the tooth mask based on the facial key points, superimpose the tooth contour and the tooth mask image to obtain the tooth image to be restored, segment the mouth area from the facial image as a reference image, perform image enhancement and refinement on the mouth area reference image, and then normalize the tooth image to be restored and the mouth area reference image; the model building module 32 is used to build a high-definition tooth restoration model based on the GAN network; the tooth restoration module 33 is used to perform high-definition tooth restoration using the high-definition tooth restoration model.

[0110] It should be noted that the high-definition tooth restoration device based on the GAN network provided in the above embodiment should be illustrated by the division of the above functional modules when performing high-definition tooth restoration. The above functional distribution can be completed by different functional modules as needed, that is, the internal structure of the terminal or server is divided into different functional modules to complete all or part of the functions described above. In addition, the high-definition tooth restoration device based on the GAN network provided in the above embodiment and the high-definition tooth restoration method based on the GAN network embodiment belong to the same concept. The specific implementation process is detailed in the high-definition tooth restoration method based on the GAN network embodiment, which will not be repeated here.

[0111] Based on the same inventive concept, an embodiment further provides a computing device, including a memory and one or more processors, wherein executable codes are stored in the memory, and when the one or more processors execute the executable codes, the high-definition tooth restoration method based on the GAN network is implemented, specifically including the following steps:

[0112] S1, extracting facial key points from the acquired facial image, determining tooth contours and tooth masks based on the facial key points, superimposing the tooth contour and tooth mask images to obtain a tooth image to be restored, segmenting the mouth area from the facial image as a reference image, performing image enhancement and refinement processing on the mouth area reference image, and then normalizing the tooth image to be restored and the mouth area reference image;

[0113] S2, a high-definition tooth restoration model is constructed based on the GAN network to repair the tooth image to be restored;

[0114] S3, high-definition tooth restoration using high-definition tooth restoration models.

[0115] The computing device provided in the embodiment, in addition to the processor and memory, also includes hardware required for other services such as internal bus, network interface, memory, etc. at the hardware level. The memory is a non-volatile memory, and the processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the above-mentioned high-definition tooth restoration method based on the GAN network. Of course, in addition to the software implementation, the present invention does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0116] Based on the same inventive concept, an embodiment further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned high-definition tooth restoration method based on the GAN network is implemented, which specifically includes the following steps:

[0117] S1, extracting facial key points from the acquired facial image, determining tooth contours and tooth masks based on the facial key points, superimposing the tooth contour and tooth mask images to obtain a tooth image to be restored, segmenting the mouth area from the facial image as a reference image, performing image enhancement and refinement processing on the mouth area reference image, and then normalizing the tooth image to be restored and the mouth area reference image;

[0118] S2, a high-definition tooth restoration model is constructed based on the GAN network to repair the tooth image to be restored;

[0119] S3, high-definition tooth restoration using high-definition tooth restoration models.

[0120] In the embodiment, computer-readable media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data.

[0121] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A high-definition tooth restoration method based on a GAN network, characterized in that: The following steps are involved: Extract facial key points from the acquired facial image, determine tooth contours and tooth masks based on the facial key points, superimpose the tooth contour and tooth mask images to obtain a tooth image to be restored, segment the mouth area from the facial image as a reference image, enhance and refine the mouth area reference image, and then normalize the tooth image to be restored and the mouth area reference image; A high-definition tooth restoration model is constructed based on a GAN network, wherein the generator contained in the GAN network adopts an encoding structure and a decoding structure. The encoding structure uses two branches to extract and encode features of the normalized tooth image to be restored and the reference image of the mouth area respectively. The decoding structure decodes the encoding results of the two branches to perform tooth restoration to obtain the restored tooth image. The discriminator contained in the GAN network is used to distinguish the authenticity of the restored tooth image. The generator after the two-stage training is used as the high-definition tooth restoration model. High-definition dental restorations are performed using high-definition dental restoration models.

2. The high-definition tooth restoration method based on the GAN network according to claim 1, characterized in that: The MobileNetV3 network is used to extract facial key points. The extracted facial key points include mouth contour points and related feature points of the tooth area. The tooth contour and tooth mask are determined based on the tooth contour points, and the mouth area reference image M is segmented from the facial image. reference .

3. The high-definition tooth restoration method based on the GAN network according to claim 1 or 2, characterized in that: When determining the tooth contour, first use the contour generation function C based on the facial key point set P facial Generate tooth contour image M tooth_contour : M tooth_contour =C(P facial ) Through the mask generation function M, the facial key point set P facial Perform mask processing to obtain the initial tooth mask image M mask : M mask =M(P facial ) Perform an XOR operation on the initial tooth mask image and the mouth area image reference image so that the tooth area of ​​the mouth area reference image forms a mask, and obtain the final tooth mask image: M tooth_mask =U(M mask ,M reference ) Wherein, U represents the image XOR operation. The tooth contour image and the tooth mask image are superimposed to obtain the tooth image to be repaired M tooth .

4. The high-definition tooth restoration method based on the GAN network according to claim 1, characterized in that: The reference image of the mouth area is enhanced and refined, and the image of the tooth to be restored and the reference image of the mouth area are normalized. Contrast enhancement and sharpening are performed during image enhancement, and edge sharpening and smoothing are performed during refinement. During normalization, the pixel values ​​of the image are normalized to between [0, 1] or [-1, 1].

5. The high-definition tooth restoration method based on the GAN network according to claim 1, characterized in that: The two branch structures used in the encoding structure are the same. Each branch structure includes a feature extraction module and a fine-grained feature fusion module. The two models are used to extract features and perform fine-grained feature fusion to realize the encoding process. The fine-grained feature fusion module specifically includes channel fusion, HourGlass network, feature fusion, and feature refinement. Channel fusion refers to gradually reducing the number of channels of the feature map through convolution operations, while increasing the depth of the features to extract finer features. The HourGlass network is used to extract multi-scale features. Feature fusion refers to the feature fusion of the output results of the channel fusion pair with the extracted multi-scale features. Feature refinement refers to enhancing the details of the feature fusion results as the encoding results through convolution and activation operations.

6. The high-definition tooth restoration method based on the GAN network according to claim 5, characterized in that: Each fine-grained feature fusion module consists of a sequentially connected convolutional layer, a maximum pooling layer, three channel fusions, an HourGlass network, a channel fusion, and feature refinement. The HourGlass network uses two branches. One branch extracts fine features by channel fusion through channel indentation, and the other branch extracts multi-scale features by combining two pairs of maximum pooling and channel fusion, and upsampling. The fine features and multi-scale features are then fused to obtain the feature fusion result. Feature refinement adopts at least one depth-wise separable convolution or at least one residual connection constructed between depth-wise separable convolutions.

7. The high-definition tooth restoration method based on the GAN network according to claim 1, characterized in that: The loss function L used when training the GAN network total Including the adversarial loss of the generator Reconstruction loss of the generator The perceptual loss L of the generator perc , the discriminant loss L of the discriminator D , and the discriminator's R1 loss L R1 , Among them, λ1, λ2, λ3, and λ4 represent the weights of each loss, and λ is a hyperparameter used to control the strength of R1 loss. is the distribution between the tooth reference image and the restored tooth image. The sampling is obtained, is the discriminator pair The judgment result of Yes The gradient of ||·||2 represents the L2 norm, which is used to calculate the size of the gradient vector, and E represents the expected value. to calculate.

8. A high-definition tooth restoration device based on a GAN network, characterized in that: include: Image processing module, model building module, tooth restoration module, The image processing module is used to extract facial key points from the acquired facial image, determine the tooth contour and tooth mask based on the facial key points, superimpose the tooth contour and tooth mask images to obtain the tooth image to be restored, segment the mouth area from the facial image as a reference image, perform image enhancement and refinement processing on the mouth area reference image, and then perform normalization processing on the tooth image to be restored and the mouth area reference image; The model construction module is used to construct a high-definition tooth restoration model based on a GAN network, wherein the generator included in the GAN network adopts an encoding structure and a decoding structure, the encoding structure adopts two branches to respectively extract and encode features of the normalized tooth image to be restored and the reference image of the mouth area, the decoding structure decodes the encoding results of the two branches to perform tooth restoration to obtain a restored tooth image, the discriminator included in the GAN network is used to distinguish the authenticity of the restored tooth image, and the generator after the two-stage training is used as the high-definition tooth restoration model; The tooth restoration module is used to perform high-definition tooth restoration using a high-definition tooth restoration model.

9. A computing device comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the one or more processors execute the executable code, they are used to implement the high-definition tooth restoration method based on the GAN network as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the high-definition tooth restoration method based on a GAN network described in any one of claims 1 to 7 is implemented.

Images