Tooth area enhancement method and device applied to voice-driven face video generation

Abstract

The invention discloses a tooth region enhancement method and device applied to voice-driven face video generation, and the method comprises the steps: obtaining an original video frame, cutting a mouth region, and generating a to-be-corrected tooth region image frame; inputting the to-be-corrected tooth area image frame into a pre-trained orthodontic doctor network to obtain a corrected tooth area image frame; wherein the orthodontic physician network comprises a detail-oriented tooth aligner for performing feature extraction based on a double-branch structure and a memory-oriented tooth stabilizer for maintaining global structural consistency and local dynamic features of teeth by using long and short term memory; and stitching the corrected tooth area image frame into a corresponding original video frame, and synthesizing a final face video frame. According to the method, the problems of time consistency and visual reality of the tooth area in voice-driven face video generation are solved, and the overall quality of the video and the user experience are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of video generation, and in particular relates to a method and device for enhancing a teeth region for speech-driven facial video generation. Background Art

[0002] Speech-driven facial video generation technology, also known as audio-driven speaking face synthesis technology, aims to combine a single facial image and speech audio clips to generate realistic speaking videos. This technology has broad application prospects in virtual communications, digital entertainment, and multimedia content creation. In recent years, with the development of deep learning technology, this technology has also made significant progress.

[0003] Early audio-driven face synthesis methods focused on direct audio-visual mapping. Chen et al. (2018) and Zhou et al. (2019) established a basic framework for lip synchronization and expression generation. Recent developments have significantly improved the quality and controllability of synthesis. For example, SyncDiffusion (Zhao et al., 2024) introduced a diffusion-based framework that achieved significant temporal consistency and lip synchronization accuracy; HeyGen (Li et al., 2024) proposed a novel attention mechanism designed for processing dynamic facial features during speech; and SadTalker (Zhang et al., 2023) developed an emotion-aware generation pipeline that significantly enhanced the naturalness of facial expressions.

[0004] Recent research focuses on enhancing fine-grained control and generalization capabilities. DiffTalk (Wang et al., 2024) employs a conditional diffusion model to achieve precise control of facial attributes while maintaining speech synchronization. VideoReTalking (Xu et al., 2024) achieves remarkable results in cross-identity synthesis by decoupling speech content from speaker identity. VASA (Xu et al., 2024) further advances the field by introducing a view-adaptive architecture that effectively handles diverse head poses.

[0005] Despite significant progress in audio-driven speaking face synthesis, existing methods still have significant shortcomings, especially when dealing with dental areas:

[0006] (1) Lack of tooth structure stability: Existing methods fail to treat teeth as special rigid structures, resulting in unnatural changes in the size and shape of teeth in the generated videos, especially obvious instability between consecutive frames.

[0007] (2) Insufficient fidelity of tooth details: During the generation process, details such as tooth surface texture and tooth gaps are often blurred or completely lost, making the generated tooth area lack realism.

[0008] (3) Lack of tooth occlusion processing mechanism: During speaking, teeth are often partially occluded by lips. Existing methods are difficult to effectively handle the transition from image editing to image restoration, resulting in serious visual artifacts when teeth are partially occluded.

[0009] (4) Lack of a tooth-specific time consistency module: Existing technologies mostly use a general time consistency mechanism and fail to develop a special time consistency maintenance method for the particularity of the tooth area.

[0010] These issues severely impact the realism and user experience of audio-driven speaking face synthesis videos, especially in high-quality facial animation applications. Therefore, a new solution is urgently needed to specifically address the temporal consistency and visual realism of the dental region in audio-driven speaking face synthesis, thereby improving the overall quality of the generated videos and user experience. Summary of the Invention

[0011] To address the above-mentioned problems in the prior art, the present invention provides a method for enhancing the teeth region in speech-driven facial video generation. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0012] In a first aspect, the present invention proposes a method for enhancing the teeth region for speech-driven face video generation, comprising:

[0013] Obtaining the original video frames generated by the speech-driven image generation video model, and cropping the mouth area of ​​each original video frame to obtain the masks and contours of the teeth and lips, and generating the tooth area image frame to be corrected;

[0014] Inputting the tooth region image frame to be corrected into a pre-trained orthodontist network to perform tooth region enhancement correction to obtain a corrected tooth region image frame;

[0015] The orthodontist network includes a detail-oriented tooth aligner and a memory-oriented tooth stabilizer. The detail-oriented tooth aligner is used to extract features from the tooth region image frame to be corrected based on a dual-branch structure to obtain a detail feature map. The memory-oriented tooth stabilizer is used to use long-term and short-term memory to maintain the global structural consistency and local dynamic features of the teeth, and combines the detail feature map to obtain the corrected tooth region image frame.

[0016] The corrected tooth area image frames are stitched into the corresponding original video frames to synthesize the final face video frames.

[0017] In a second aspect, the present invention provides a teeth region enhancement device for speech-driven facial video generation, for implementing the method provided in the first aspect of the present invention, the device comprising:

[0018] A preprocessing module is used to obtain the original video frames generated by the speech-driven image generation video model and crop the mouth area of ​​each original video frame to obtain the masks and contours of the teeth and lips, and generate the image frames of the teeth area to be corrected;

[0019] The enhancement and correction module is configured with an orthodontist network, and the orthodontist network is used to perform tooth region enhancement and correction on the tooth region image frame to be corrected to obtain a corrected tooth region image frame;

[0020] The orthodontist network includes a detail-oriented tooth aligner and a memory-oriented tooth stabilizer. The detail-oriented tooth aligner is used to extract features from the tooth region image frame to be corrected based on a dual-branch structure to obtain a detail feature map. The memory-oriented tooth stabilizer is used to use long-term and short-term memory to maintain the global structural consistency and local dynamic features of the teeth, and combines the detail feature map to obtain the corrected tooth region image frame.

[0021] The synthesis module is used to stitch the corrected tooth area image frames into the corresponding original video frames to synthesize the final face video frames.

[0022] Beneficial effects of the present invention:

[0023] The tooth region enhancement method for speech-driven facial video generation provided by the present invention first preprocesses the original video frames generated by the speech-driven image generation video model to obtain the tooth region image frames to be corrected. Then, an orthodontist network including a detail-oriented tooth aligner and a memory-oriented tooth stabilizer is constructed and trained to perform tooth region enhancement correction on the tooth region image frames to be corrected. Finally, the corrected tooth region image frames are stitched into the corresponding original video frames to synthesize the final facial video frames. On the one hand, this method extracts comprehensive tooth features through the dual-branch fine-grained feature extraction mechanism of the detail-oriented tooth aligner, which can simultaneously preserve the micro-details of the teeth and adapt to complex deformations, thus solving the dual challenges of blurred tooth region details and structural deformation in existing methods. On the other hand, the dual memory mechanism of the memory-oriented tooth stabilizer, which combines long-term and short-term memory, can more effectively balance global structural stability and local natural changes, maintaining temporal consistency and visual realism while preserving the details of the tooth structure, significantly improving the overall quality of the generated video and user experience.

[0024] 2. The present invention introduces a quality assessment mechanism into the memory-guided tooth stabilizer to dynamically adjust the memory update strategy, ensuring that only high-quality tooth features are retained in the memory library, effectively preventing the accumulation of temporal inconsistencies caused by low-quality frames, and showing stronger robustness in the continuous speech process.

[0025] 3. This invention adopts a plug-in design, enabling seamless integration with a variety of existing audio-driven facial synthesis methods (such as AniPortrait, Audio2Head, EchoMimic, JoyHallo, and Hallo2). Experiments have shown that OrthoNet provides stable performance improvements over different baseline methods, with an average improvement of 2-3% in the FVD metric, demonstrating the excellent versatility and adaptability of this invention.

[0026] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic flow chart of a method for enhancing the teeth region in speech-driven facial video generation according to an embodiment of the present invention;

[0028] Figure 2 A schematic diagram of the algorithm framework of a teeth region enhancement method for speech-driven face video generation provided by an embodiment of the present invention;

[0029] Figure 3 A schematic diagram of preprocessing an original video frame according to an embodiment of the present invention;

[0030] Figure 4 This is a structural block diagram of a teeth area enhancement device for speech-driven facial video generation provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0032] The first aspect of the present invention provides a method for enhancing the teeth region for speech-driven face video generation. Figure 1 and Figure 2 , Figure 1 A schematic flow chart of a method for enhancing the teeth region in speech-driven face video generation according to an embodiment of the present invention is provided. Figure 2 Schematic diagram of the algorithm framework of the teeth region enhancement method for speech-driven face video generation provided by an embodiment of the present invention; the method includes:

[0033] Step 1: Obtain the original video frames generated by the speech-driven image generation video model and crop the mouth area of ​​each original video frame to obtain the masks and contours of the teeth and lips, and generate the image frames of the teeth area to be corrected.

[0034] The method proposed in this paper corrects the dental regions in videos generated using existing speech-driven image generation video models, or audio-driven facial synthesis methods, to enhance the realism of the dental regions in audio-driven speech-driven facial synthesis. Therefore, the generated raw video frames must first be preprocessed to obtain the dental region image frames to be corrected. It should be understood that the raw video frames to be processed here are facial images.

[0035] Specifically, see Figure 3 , Figure 3 The schematic diagram of preprocessing the original video frame provided by the embodiment of the present invention mainly includes the following steps:

[0036] 11) Perform facial key point recognition on the original video frame to accurately locate the mouth area and capture the mouth area; wherein, facial key points include the left and right endpoints of the mouth corners, the tip of the nose, and the chin.

[0037] 12) Create masks and outlines of teeth and lips based on the identified facial key points;

[0038] 13) Adjusting the resolution of the masks and contours of the teeth and lips, and performing fusion to generate an image frame of the tooth region to be corrected.

[0039] It should be noted that the key point recognition technology, creation of teeth and lips masks and contours, resolution adjustment, and fusion technology involved in the above steps can all be implemented using existing related technologies, and will not be introduced in detail in this embodiment.

[0040] Step 2: Input the tooth area image frame to be corrected into the pre-trained orthodontist network to perform tooth area enhancement correction to obtain the corrected tooth area image frame.

[0041] To enhance and correct dental regions, this example proposes a plug-in framework called "Orthodontist Network" (OrthoNet), specifically designed to enhance the realism of dental regions in audio-driven speaking face synthesis. Acting as a virtual orthodontist, OrthoNet seamlessly integrates with existing Audio2Video methods (such as AniPortrait, Audio2Head, EchoMimic, JoyHallo, and Hallo2), significantly improving the realism and temporal consistency of dental regions in generated videos.

[0042] This framework draws on the professional skills of orthodontists in aligning and stabilizing teeth, and mainly includes two core modules: a detail-guided tooth aligner and a memory-guided tooth stabilizer; among them, the detail-guided tooth aligner is used to extract features from the tooth area image frame to be corrected based on a dual-branch structure to obtain a detail feature map; the memory-guided tooth stabilizer is used to use long-term and short-term memory to maintain the global structural consistency and local dynamic characteristics of the teeth, and combined with the detail feature map to obtain the corrected tooth area image frame.

[0043] The detail-guided tooth aligner and memory-guided tooth stabilizer proposed in the present invention are respectively introduced in detail below.

[0044] 1. Detail-oriented tooth aligners

[0045] The detail-oriented tooth aligner module designed in this invention extracts precise tooth features through a two-branch approach, similar to how orthodontists use different tools to adjust the position and shape of teeth.

[0046] For more details, please see Figure 2 , wherein the fine-tuned tooth aligner specifically includes a spatial-depth transform convolution (DRM-Conv), a first maximum pooling layer (Max Pooling), an adaptive variable kernel convolution (PKConv), a second maximum pooling layer, a channel fusion module (Channel Fusion) and an hourglass network (Hour Glass); wherein,

[0047] The spatial-depth transformation convolution and the first maximum pooling layer constitute a first branch structure, which is used to process the image frame of the tooth area to be corrected to convert the information of the spatial dimension into the channel dimension while maintaining the fine structure of the tooth details, thereby obtaining a first feature map;

[0048] The second branch structure is composed of an adaptive variable kernel convolution and a second maximum pooling layer, which is used to process the image frame of the tooth area to be corrected. It uses position-specific adaptive weights and dynamic offsets to adapt to the changes in tooth shape according to the tooth mask to obtain a second feature map.

[0049] The channel fusion module is used to perform channel adaptive fusion on the first feature map and the second feature map to obtain a fused feature map;

[0050] The hourglass network is used to perform multi-scale feature extraction on the fused feature map to capture tooth features at different scales, achieve comprehensive feature representation, and obtain a detailed feature map.

[0051] Specifically, given the input feature map F in∈RC×H×W (where C represents the channel dimension, H and W represent the spatial dimensions), the dual-branch structure of the detail-oriented tooth aligner extracts comprehensive tooth features through two complementary paths and outputs them after multi-scale fusion.

[0052] The first branch is the space-depth transform convolution branch. This branch preserves key fine-grained details such as tooth edges and textures through space-to-depth transformation, followed by a specialized convolution operation, which is expressed as:

[0053] F drm =Conv(Cat(F in [i:N:scale,j:N:scale]),W drm );

[0054] Where, F drm represents the output of the spatial-depth transform convolution, Cat(·) represents the splicing operation along the channel dimension, i and j represent the spatial position index, N represents the spatial dimension of the feature map, scale is empirically determined to be 2 to maintain the best balance between computational efficiency and preservation of key edge details, W spd represents the learnable convolution weights designed for detail enhancement.

[0055] After the spatial-depth transform convolution processing, it undergoes the maximum pooling operation. The maximum pooling operation can extract the most significant features, reduce the spatial dimension, retain the structural information, and enhance the model's robustness to translation, thereby obtaining the features extracted by the spatial-depth transform convolution branch, which is recorded as the first feature map.

[0056] The present invention can effectively preserve and enhance the micro-detail features of the tooth area through a special convolution processing based on space-to-depth transformation of the space-depth transformation convolution branch.

[0057] Next comes the adaptive variable kernel convolution branch, also known as the second branch. This branch uses position-specific adaptive weights and dynamic offsets to adapt to changes in tooth shape based on the tooth segmentation mask. This is particularly well-suited for handling the partial visibility of teeth during speech.

[0058] In order to handle complex tooth shapes and occlusions, adaptive variable kernel convolution adopts an adaptive sampling mechanism, which is expressed as follows:

[0059]

[0060] Where, F pk (p) represents the output of the adaptive variable kernel convolution operation at position p, where p represents the spatial position, P n represents a sampling pattern with n points, w k (p) is the position-specific adaptation weight, Δpk Indicates the sampling offset, f offset (F in ,M teeth ) Based on the input feature F in and the segmentation mask M of the tooth region teeth Calculated dynamic offset function.

[0061] After the adaptive variable kernel convolution processing and the maximum pooling operation, the features extracted by the adaptive variable kernel convolution branch can be obtained, which is recorded as the second feature map.

[0062] The present invention uses adaptive convolution processing with a dynamic sampling mechanism of an adaptive variable kernel convolution branch to adjust the feature extraction strategy according to the tooth occlusion state.

[0063] Next is multi-scale feature fusion. The first feature map generated by the spatial-depth transform convolution branch and the second feature map generated by the adaptive variable kernel convolution branch are fused channel by channel, and the contribution of details and global structure is balanced through adaptive weights. The formula is expressed as:

[0064]

[0065] Where, F Out represents the output features after fusion, L represents the total number of scale levels, β l is a learnable scale-specific weight that optimizes the importance of each resolution level, Up(·) represents a bilinear upsampling operation to align feature resolutions, and α l ∈[0,1] dynamically balances the contribution of local details and global structure at each scale level l, and They represent the features of the spatial-depth transformation convolution and adaptive variable kernel convolution at the lth scale level respectively.

[0066] This multi-scale fusion ensures comprehensive feature extraction at different spatial scales, which is crucial for accurate tooth alignment planning.

[0067] Finally, the hourglass network, also known as the multi-scale feature extraction network, is used to first downsample and then upsample to capture tooth features at different scales, achieve comprehensive feature representation, and obtain the output of the detail-oriented tooth aligner, called the detail feature map.

[0068] The detail-oriented tooth aligner designed by the present invention innovatively combines a dual-branch architecture of space-depth transform convolution and adaptive variable kernel convolution. The former combines space-to-depth transform with dedicated convolution operations to accurately preserve micro details such as tooth edges and textures; the latter adaptively adjusts the feature extraction strategy according to the tooth occlusion status through a dynamic sampling mechanism, effectively coping with complex deformations during lip-tooth interaction. This complementary dual-branch design comprehensively solves the dual challenges of blurred tooth area details and structural deformation in existing methods. Compared with the standard convolution operations used in existing methods (such as SyncDiffusion, HeyGen, VASA, etc.), the feature extraction mechanism of the present invention shows obvious advantages in tooth edge clarity and surface texture fidelity.

[0069] 2. Memory-guided tooth stabilizer

[0070] To maintain temporal consistency while preserving tooth structural details, the present invention proposes a memory-guided tooth stabilizer that integrates long-term and short-term time dependencies, similar to the way retainers are used to stabilize teeth after orthodontic treatment.

[0071] For details, please refer to Figure 2 The memory-oriented tooth stabilizer specifically includes a long-term memory module (LongTerm Memory, LTM), a short-term memory module (ShortTerm Memory, STM), a memory fusion module (MemoryFusion) and a feature decoder; wherein,

[0072] The long-term memory module is used to maintain a long-term memory buffer based on the global structure of the tooth, and output a key vector K and a value vector V; the long-term memory buffer stores a first number of tooth region image frames;

[0073] The short-term memory module is used to maintain a short-term memory buffer based on the local dynamic features and output a query vector Q; the short-term memory buffer stores a second number of tooth region image frames; and the second number is less than the first number;

[0074] The memory fusion module is used to integrate the key vector K, the value vector V, the query vector Q and the detail feature map output by the detail-oriented tooth aligner based on a cross-attention mechanism to obtain a third feature map;

[0075] The feature decoder is used to process the third feature map to obtain a corrected tooth region image frame;

[0076] The corrected tooth region image frames are also used to dynamically update the tooth region image frames stored in the long-term memory buffer and the short-term memory buffer.

[0077] Optionally, as an implementation, the first number in this embodiment may be 30, and the second number may be 4, i.e., 30 image frames are stored in the long-term memory buffer, and 4 image frames are stored in the short-term memory buffer. It is understood that, at the initial moment, the image frames stored in the short-term memory buffer are from the tooth region image frames to be corrected generated based on the original video frames. When the short-term memory buffer has stored 4 image frames, the short-term memory module in the memory-guided dental stabilizer is activated, performs image processing, and outputs corrected tooth region image frames. The image frames stored in the long-term memory buffer are from the tooth region image frames to be corrected generated based on the original video frames and the corrected tooth region image frames output by the memory-guided dental stabilizer. When the long-term memory buffer has stored 30 image frames, both the short-term memory module and the long-term memory module in the memory-guided dental stabilizer are activated, and image processing is performed jointly.

[0078] For long-term memory modules, such as Figure 2 As shown, it includes a memory encoding unit and a memory updating unit;

[0079] The memory encoding unit is used to sequentially perform a 3D convolution (Conv3D) operation, a 3D maximum pooling (MaxPool3D) operation, and a feature compression (Feature Compression) operation on the tooth area image frames in the long-term memory buffer to obtain a fourth feature map. The 3D convolution operation is used to process 3D data including a time dimension and simultaneously extract features in the spatial and time dimensions to obtain a 3D feature map. The 3D maximum pooling operation is used to downsample the 3D feature map to reduce the data dimension while retaining significant features. The feature compression operation is used to further reduce the feature dimension.

[0080] The memory update unit processes the fourth feature map in sequence through a first-in-first-out queue (FIFO Queue), a feature selection operation, and a memory writing operation, and outputs a key vector K and a value vector V; wherein the first-in-first-out queue is used to implement a memory update mechanism; the feature selection operation is used to select high-quality tooth features based on quality assessment; and the memory writing operation is used to write the selected high-quality tooth features into a long-term memory cache.

[0081] Specifically, in the long-term memory module, for the 30 frames of tooth area image frames in the long-term memory buffer, 3D convolution analyzes the spatiotemporal features of the 30-frame tooth sequence to capture the long-term consistency pattern of the tooth structure. 3D maximum pooling retains important tooth structure information while reducing the computational burden by selecting the maximum value in the local area. Feature compression improves storage efficiency by reducing feature dimensions and removing redundant information. This is crucial for the efficient management of the long-term memory buffer to ensure that only key representations of the tooth structure are retained. When new high-quality tooth frames enter the first-in-first-out queue, the oldest frames are removed to maintain the temporal relevance of the memory. Feature selection selects high-quality tooth features through quality assessment to ensure that only high-quality representations that exceed the threshold are updated to the long-term memory. Finally, the selected high-quality tooth features are written to the long-term memory buffer through memory writing to update the global tooth representation.

[0082] Furthermore, for short-term memory modules, such as Figure 2 As shown, it includes a motion analysis unit and a query vector generation unit; wherein,

[0083] The motion analysis unit is used to perform feature extraction, long short-term memory convolution (ConvLSTM) and temporal fusion operations on the tooth area image frames in the short-term memory buffer to obtain a fifth feature map. The feature extraction operation is mainly used to extract time-related tooth features and capture local dynamic changes. The long short-term memory convolution operation is used to combine convolution and LSTM structures to process spatial and time series data. The temporal fusion operation integrates features from multiple time steps to generate a unified representation of local tooth dynamics.

[0084] The query vector analysis unit is used to perform adaptive variable kernel convolution (PKConv) operation, ReLU activation and layer normalization (ReLU+LayerNorm) operation, and feature projection (Feature Projection) operation on the fifth feature map in sequence, and output the query vector Q; among them, the adaptive variable kernel convolution operation is used to use the dynamically adjusted convolution kernel to process the input features to adapt to different tooth shapes and oral changes; ReLU activation and layer normalization operations are used to optimize feature representation; feature projection operation is used to map the processed features to a query space suitable for the cross-attention mechanism.

[0085] Specifically, in the short-term memory module, a convolutional long short-term memory network architecture is primarily used to capture inter-frame variations in tooth appearance within a four-frame sliding window of four tooth region images stored in the short-term memory buffer. Combining convolution operations with an LSTM structure to process both spatial and temporal data, this enables the model to remember the state of previous frames while adapting to changes in the current frame, achieving short-term temporal modeling. Subsequently, through specialized queries based on adaptive variable kernel convolutions, hidden states encoding local temporal dependencies are generated, which can adapt to different tooth shapes and oral changes. This adaptive approach is particularly important for processing complex tooth deformations during speech, enabling focused attention on relevant dental features and smooth transitions between frames. The short-term memory module adapts to dynamic speech patterns while maintaining local consistency. Furthermore, the ReLU function introduces nonlinearity, while layer normalization stabilizes the training process and accelerates convergence, jointly optimizing feature representation.

[0086] In addition, for the memory fusion module, such as Figure 2 As shown, it includes a cross attention unit and a feature fusion unit; wherein,

[0087] The cross attention unit is used to perform Q / K / V projection operation, multi-head attention operation and output projection operation on the key vector K, value vector V and query vector Q in sequence to obtain the sixth feature map; wherein, Q / K / V projection operation;

[0088] The multi-head attention operation is used to calculate attention in parallel through multiple "heads" to capture the relationship between different feature subspaces and effectively fuse long-term and short-term memory information; the output projection operation is used to transform the output of the multi-head attention back to the original feature space to generate a fused representation, which is recorded as the sixth feature map;

[0089] The feature fusion unit is used to perform channel concatenation, 1×1 convolution, and ReLU operations on the sixth feature map in sequence to obtain a converted feature map, and perform a residual add operation on the converted feature map and the detail feature map to obtain a third feature map; wherein, the channel concatenation operation is used to merge features from different sources along the channel dimension and retain their respective information; the 1×1 convolution and ReLU operations are used to achieve feature conversion and enhancement; the residual connection operation is used to add a skip connection to add the detail feature map directly to the converted feature map to alleviate the gradient vanishing problem and promote deep network training.

[0090] Specifically, in the memory fusion module, the vectors output by the long-term memory module and the short-term memory module are integrated through the cross-attention mechanism, and finally the corrected tooth area image frame is obtained through the feature decoder.

[0091] Furthermore, for the dynamic update of the long-term memory buffer and the short-term memory buffer, this embodiment adopts a quality-aware update strategy, which is as follows:

[0092] First, for each newly obtained corrected tooth region image frame, the tooth features are evaluated to obtain the corresponding quality estimation value;

[0093] Then, a threshold judgment is performed. When the quality estimation value is higher than the preset threshold, the current corrected tooth area image frame and the original tooth area image frame in the long-term memory buffer or the short-term memory buffer are combined for update, and the corresponding number of tooth area image frames are retained through the Top-K selection mechanism to ensure that the long-term memory buffer and the short-term memory buffer maintain high-quality tooth structure samples.

[0094] The quality assessment mechanism employed in this invention dynamically adjusts the memory update strategy, ensuring that only high-quality dental features are retained in the memory bank. This effectively prevents the accumulation of temporal inconsistencies caused by low-quality frames, demonstrating greater robustness during continuous speech.

[0095] The memory-guided tooth stabilizer designed in this invention innovatively incorporates a dual-stream architecture of long-term memory (LTM) and short-term memory (STM), simulating the professional methods used by orthodontists to maintain tooth stability using different types of retainers after treatment. The LTM module maintains a 30-frame feature buffer to store global tooth structure information. A carefully designed quality-aware update strategy ensures the quality of the memory. Specifically, the system evaluates the quality of tooth features in each new frame and only considers updating the memory when the quality exceeds a preset threshold. A Top-K selection mechanism retains the most representative, high-quality tooth structure samples, effectively preventing memory degradation caused by low-quality frames. Simultaneously, the STM module utilizes a convolutional long-short-term memory network architecture to maintain a four-frame sliding window, capturing local temporal dynamics and subtle changes. A specialized adaptive kernel convolution query generates hidden states that encode rich temporal dependencies. The two memories are intelligently fused through a cross-attention mechanism, and an adaptive blending factor dynamically balances global stability with local natural variations based on the quality of the current frame. This achieves a perfect balance between the stability of teeth as rigid structures and the natural micro-movements necessary during speech. After experimental verification, the present invention has an average improvement of 9.8% to 15.7% in TSM (tooth stability) indicators compared with existing methods, proving the significant advantages of this mechanism in maintaining tooth shape stability.

[0096] It should be further explained that, in this embodiment, a loss function is specifically designed when pre-training the orthodontist network.

[0097] On the one hand, in order to ensure the pixel-level accuracy between the generated tooth region Ig and the true Igt, the reconstruction loss function and the tooth perception loss function are designed.

[0098] Specifically, the reconstruction loss is recorded as L rec , whose expression is:

[0099]

[0100] Where, I g Represents the generated tooth area image, I gt represents the real tooth area image,

[0101] || ||1 and They represent the square of L1 norm and L2 norm respectively; α1 and α2 represent the balance weights of L1 and L2 respectively.

[0102] The reconstruction loss designed in this embodiment can ensure pixel-level accuracy and perceptual similarity between the generated image and the real image.

[0103] The tooth perception loss is recorded as L teeth , whose expression is:

[0104] L teeth =β·L struct +(1-β)·L detail ;

[0105]

[0106] Where β represents the balance parameter between structure loss and detail loss, and its value is 0.7; L struct Indicates structural loss, L detail Indicates detail loss, N t Represents the total number of pixels in the tooth area, M t represents the tooth region mask, φ l and φ h denote the low-level and high-level feature extractors of the VGG-19 network, respectively. and represent the generated image and real image at position p respectively.

[0107] On the other hand, the present invention also designs adversarial loss and temporal consistency loss to achieve temporal consistency.

[0108] Specifically, this embodiment mainly adopts the adversarial loss based on WGAN-GP, denoted as L adv , whose expression is:

[0109]

[0110] Where D() represents the discriminator, P g and P r is the distribution of generated images and real images, L gp is the gradient penalty term, λ gp Represents the weight of the gradient penalty term.

[0111] Furthermore, the temporal consistency loss is recorded as L temp , whose expression is:

[0112]

[0113] Where, represents the generated image of the t-th frame, w() represents the optical flow distortion operation, γ represents the weight of the memory consistency term, and h t represents the hidden state of the convolutional long short-term memory network ConvLSTM, f lstm () represents the LSTM state transition function.

[0114] The total loss for training is expressed as:

[0115] L total =λ1L rec +λ2L adv +λ3L temp +λ4L teeth ;

[0116] Where K total Represents the total loss, λ1, λ2, λ3, and λ4 represent the weight parameters of each loss function respectively.

[0117] By using the above loss function to train the constructed orthodontist network including the detail-oriented tooth aligner and the memory-oriented tooth stabilizer, the trained network can be used for tooth area enhancement correction.

[0118] Step 3: Stitch the corrected tooth area image frame into the corresponding original video frame to synthesize the final face video frame.

[0119] The tooth area enhancement method provided by the present invention for voice-driven face video generation, on the one hand, extracts comprehensive tooth features through the dual-branch fine-grained feature extraction mechanism of the detail-oriented tooth aligner, which can simultaneously retain the micro-details of the teeth and adapt to complex deformations, solving the dual challenges of blurred tooth area details and structural deformation in existing methods; on the other hand, through the dual memory mechanism combining the long and short-term memory of the memory-oriented tooth stabilizer, it can more effectively balance global structural stability and local natural changes, while maintaining temporal consistency and visual realism while maintaining the details of the tooth structure, significantly improving the overall quality of the generated video and user experience.

[0120] Furthermore, the present invention adopts a plug-in design, enabling seamless integration with a variety of existing audio-driven face synthesis methods, such as AniPortrait, Audio2Head, EchoMimic, JoyHallo, and Hallo2. Experiments have demonstrated that OrthoNet consistently delivers performance improvements over various baseline methods, with an average improvement of 2-3% in the Fréchet Video Distance (FVD) metric, demonstrating the present invention's excellent versatility and adaptability.

[0121] Based on the same inventive concept, the second aspect of the present invention also provides a teeth region enhancement device for speech-driven face video generation. Figure 4 , Figure 4 This is a structural block diagram of a teeth region enhancement device for speech-driven face video generation provided by an embodiment of the present invention. The device mainly includes:

[0122] A preprocessing module is used to obtain the original video frames generated by the speech-driven image generation video model and crop the mouth area of ​​each original video frame to obtain the masks and contours of the teeth and lips, and generate the image frames of the teeth area to be corrected;

[0123] The enhancement and correction module is configured with an orthodontist network, and the orthodontist network is used to perform tooth region enhancement and correction on the tooth region image frame to be corrected to obtain a corrected tooth region image frame;

[0124] The orthodontist network includes a detail-oriented tooth aligner and a memory-oriented tooth stabilizer. The detail-oriented tooth aligner is used to extract features from the tooth region image frame to be corrected based on a dual-branch structure to obtain a detail feature map. The memory-oriented tooth stabilizer is used to use long-term and short-term memory to maintain the global structural consistency and local dynamic features of the teeth, and combines the detail feature map to obtain the corrected tooth region image frame.

[0125] The synthesis module is used to stitch the corrected tooth area image frames into the corresponding original video frames to synthesize the final face video frames.

[0126] It should be noted that, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. Therefore, the device provided by the present invention can also achieve the same or similar beneficial effects as the method embodiment.

[0127] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A method for enhancing the teeth region in speech-driven face video generation, characterized in that: include: Obtaining the original video frames generated by the speech-driven image generation video model, and cropping the mouth area of ​​each original video frame to obtain the masks and contours of the teeth and lips, and generating the tooth area image frame to be corrected; Inputting the tooth region image frame to be corrected into a pre-trained orthodontist network to perform tooth region enhancement correction to obtain a corrected tooth region image frame; The orthodontist network includes a detail-oriented tooth aligner and a memory-oriented tooth stabilizer; the detail-oriented tooth aligner is used to extract features from the tooth region image frame to be corrected based on a dual-branch structure to obtain a detail feature map; the memory-oriented tooth stabilizer is used to utilize long-term and short-term memory to maintain the global structural consistency and local dynamic features of the teeth, and in combination with the detail feature map, obtain a corrected tooth region image frame; The corrected tooth region image frame is stitched into the corresponding original video frame to synthesize the final face video frame.

2. The method for enhancing the teeth region for speech-driven face video generation according to claim 1, characterized in that: The mouth area of ​​each original video frame is cropped to obtain the masks and contours of the teeth and lips, and generate the image frame of the teeth area to be corrected. Specifically, the following steps are performed: Performing facial key point recognition on the original video frame to accurately locate the mouth area and intercept the mouth area; wherein the facial key points include the left and right endpoints of the mouth corners, the tip of the nose, and the chin; Create masks and outlines of teeth and lips based on identified facial landmarks; The resolution of the masks and contours of the teeth and lips are adjusted, and fusion is performed to generate an image frame of the teeth region to be corrected.

3. The method for enhancing the teeth region for speech-driven face video generation according to claim 2, characterized in that: The detail-oriented tooth aligner specifically includes a space-depth transform convolution, a first maximum pooling layer, an adaptive variable kernel convolution, a second maximum pooling layer, a channel fusion module and an hourglass network; wherein, The spatial-depth transform convolution and the first maximum pooling layer constitute a first branch structure, which is used to process the tooth region image frame to be corrected to convert spatial dimension information into channel dimension while maintaining the fine structure of tooth details, thereby obtaining a first feature map; The adaptive variable kernel convolution and the second maximum pooling layer constitute a second branch structure, which is used to process the tooth region image frame to be corrected, use adaptive weights and dynamic offsets, adapt to tooth shape changes according to the tooth mask, and obtain a second feature map; The channel fusion module is used to perform channel adaptive fusion on the first feature map and the second feature map to obtain a fused feature map; The hourglass network is used to perform multi-scale feature extraction on the fused feature map to capture tooth features at different scales, achieve comprehensive feature representation, and obtain a detailed feature map.

4. The method for enhancing the teeth region for speech-driven face video generation according to claim 3, characterized in that: The memory-guided tooth stabilizer specifically includes a long-term memory module, a short-term memory module, a memory fusion module and a feature decoder; wherein, The long-term memory module is used to maintain a long-term memory buffer based on the global structure of the teeth, and output a key vector K and a value vector V; the long-term memory buffer stores a first number of tooth region image frames; The short-term memory module is used to maintain a short-term memory buffer based on the local dynamic features and output a query vector Q; the short-term memory buffer stores a second number of tooth region image frames; and the second number is less than the first number; The memory fusion module is configured to integrate the key vector K, the value vector V, the query vector Q, and the detail feature map output by the detail-oriented tooth aligner based on a cross-attention mechanism to obtain a third feature map; The feature decoder is used to process the third feature map to obtain a corrected tooth region image frame; The corrected tooth region image frame is further used to dynamically update the tooth region image frames stored in the long-term memory buffer and the short-term memory buffer.

5. The method for enhancing the teeth region for speech-driven face video generation according to claim 4, characterized in that: The long-term memory module specifically includes a memory encoding unit and a memory updating unit; The memory encoding unit is used to sequentially perform a 3D convolution operation, a 3D maximum pooling operation, and a feature compression operation on the tooth region image frames in the long-term memory buffer to obtain a fourth feature map; wherein the 3D convolution operation is used to process 3D data including a time dimension and simultaneously perform feature extraction on the spatial and time dimensions to obtain a 3D feature map; the 3D maximum pooling operation is used to downsample the 3D feature map to reduce the data dimension while retaining significant features; and the feature compression operation is used to further reduce the feature dimension; The memory update unit processes the fourth feature map in sequence through a first-in-first-out queue, a feature selection operation, and a memory write operation, and outputs a key vector K and a value vector V; wherein the first-in-first-out queue is used to implement a memory update mechanism; the feature selection operation is used to select high-quality tooth features based on quality assessment; and the memory write operation is used to write the selected high-quality tooth features into the long-term memory cache.

6. The method for enhancing the teeth region for speech-driven face video generation according to claim 4, characterized in that: The short-term memory module specifically includes a motion analysis unit and a query vector generation unit; wherein, The motion analysis unit is used to sequentially perform a feature extraction operation, a long short-term memory convolution operation, and a time series fusion operation on the tooth region image frames in the short-term memory buffer to obtain a fifth feature map; wherein the feature extraction operation is mainly used to extract time-related tooth features and capture local dynamic changes; the long short-term memory convolution operation is used to combine convolution and LSTM structures to process spatial and time series data; the time series fusion operation integrates features of multiple time steps to generate a unified representation of local tooth dynamics; The query vector analysis unit is used to perform adaptive variable kernel convolution operation, ReLU activation and layer normalization operation, and feature projection operation on the fifth feature map in sequence, and output a query vector Q; wherein the adaptive variable kernel convolution operation is used to use a dynamically adjusted convolution kernel to process input features to adapt to different tooth shapes and oral changes; the ReLU activation and layer normalization operations are used to optimize feature representation; and the feature projection operation is used to map the processed features to a query space suitable for the cross-attention mechanism.

7. The method for enhancing the teeth region for speech-driven face video generation according to claim 4, characterized in that: The memory fusion module specifically includes a cross attention unit and a feature fusion unit; wherein, The cross attention unit is used to perform a Q / K / V projection operation, a multi-head attention operation, and an output projection operation on the key vector K, the value vector V, and the query vector Q in sequence to obtain a sixth feature map; wherein the Q / K / V projection operation; The multi-head attention operation is used to calculate attention in parallel through multiple "heads" to capture the relationship between different feature subspaces and effectively fuse long-term and short-term memory information; the output projection operation is used to convert the output of the multi-head attention back to the original feature space to generate a fused representation, which is recorded as the sixth feature map; The feature fusion unit is used to perform channel splicing operations, 1×1 convolution and ReLU operations on the sixth feature map in sequence to obtain a converted feature map, and perform a residual connection operation on the converted feature map and the detail feature map to obtain a third feature map; wherein the channel splicing operation is used to merge features from different sources along the channel dimension to retain their respective information; the 1×1 convolution and ReLU operations are used to achieve feature conversion and enhancement; the residual connection operation is used to add the detail feature map directly to the converted feature map by adding a skip connection.

8. The method for enhancing the teeth region for speech-driven face video generation according to claim 4, characterized in that: The long-term memory buffer and the short-term memory buffer are dynamically updated using a quality-aware update strategy, specifically including: For each newly obtained corrected tooth region image frame, performing a quality assessment on the tooth features to obtain a corresponding quality estimation value; When the quality estimation value is higher than a preset threshold, the current corrected tooth area image frame and the original tooth area image frame in the long-term memory cache or the short-term memory cache are updated, and a corresponding number of tooth area image frames are retained through the Top-K selection mechanism to ensure that the long-term memory cache and the short-term memory cache maintain high-quality tooth structure samples.

9. The method for enhancing the teeth region for speech-driven face video generation according to claim 1, characterized in that: When pre-training the orthodontist network, the loss function used is: L total =λ1L rec +λ2L adv +λ3L temp +λ4L teeth ; Where K total Represents the total loss, λ1, λ2, λ3, and λ4 represent the weight parameters of each loss function respectively; L rec represents the reconstruction loss, which is expressed as: Where, I g Represents the generated tooth area image, I gt represents the real tooth region image, || ||1 and They represent the square of L1 norm and L2 norm respectively; α1 and α2 represent the balance weights of L1 and L2 respectively; L adv represents the adversarial loss based on WGAN-GP, which is expressed as: Where D() represents the discriminator, P g and P r is the distribution of generated images and real images, L gp is the gradient penalty term, λ gp Represents the weight of the gradient penalty term; L temp represents the time consistency loss, which is expressed as: Where, represents the generated image of the t-th frame, w() represents the optical flow distortion operation, γ represents the weight of the memory consistency term, and h t represents the hidden state of the convolutional long short-term memory network ConvLSTM, f lstm ( ) represents the LSTM state transition function; L teeth represents the tooth perception loss, and its expression is: L teeth =β·L struct +(1-β)·L detail ; Where β represents the balance parameter between structure loss and detail loss, and its value is 0.7; L struct Indicates structural loss, L detail Indicates detail loss, N t Represents the total number of pixels in the tooth area, M t represents the tooth region mask, φ l and φ h denote the low-level and high-level feature extractors of the VGG-19 network, respectively. and represent the generated image and real image at position p respectively.

10. A teeth region enhancement device for speech-driven face video generation, used to implement the method according to any one of claims 1 to 9, characterized in that: The device includes: A preprocessing module is used to obtain the original video frames generated by the speech-driven image generation video model and crop the mouth area of ​​each original video frame to obtain the masks and contours of the teeth and lips, and generate the image frames of the teeth area to be corrected; an enhancement and correction module, configured with an orthodontist network, wherein the orthodontist network is used to perform tooth region enhancement and correction on the tooth region image frame to be corrected to obtain a corrected tooth region image frame; The orthodontist network includes a detail-oriented tooth aligner and a memory-oriented tooth stabilizer; the detail-oriented tooth aligner is used to extract features from the tooth region image frame to be corrected based on a dual-branch structure to obtain a detail feature map; the memory-oriented tooth stabilizer is used to utilize long-term and short-term memory to maintain the global structural consistency and local dynamic features of the teeth, and in combination with the detail feature map, obtain a corrected tooth region image frame; The synthesis module is used to stitch the corrected tooth area image frame into the corresponding original video frame to synthesize the final face video frame.

Images