A face action migration model optimization method, system and device based on frame-level consistency checking

Abstract

The application discloses a face action migration model optimization method and system based on frame-level consistency verification and a device, and is applied to the technical field of data processing. A face action migration neural network is built, identity features and mouth action are decoupled, and an optimization architecture is built in combination with static form prior and dynamic key points. A five-dimensional frame-level consistency discrimination network is designed to complete quality scoring and abnormality identification. Multi-class training samples are constructed relying on contrast learning, and model optimization is carried out in combination with sorting and consistency loss. A local adapter fine-tuning mode is adopted, the identity backbone is frozen, only the mouth, time sequence and quality related modules are updated, and multi-class supervision loss is fused. Relying on small batch incremental learning, high-frequency defect samples are used to carry out closed-loop iteration to continuously correct problems such as time sequence jitter and abnormal mouth shape, and after multiple rounds of verification iteration, a digital human action migration model with stable time sequence and excellent details is finally obtained.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, system and device for optimizing a face motion migration model based on frame-level consistency verification. Background Technology

[0002] Currently, various facial motion transfer technologies are widely used in the field of digital humans. The mainstream approach involves selecting a source person's driving video and a target person's facial image or video, then using an open-source facial motion transfer model to generate a frame-by-frame video of the target person's lip movements, and directly using the generated results as digital human assets. Commonly used technical solutions at this stage include LivePortrait, VideoReTalking, and various TalkingHead-type transfer solutions such as facial speech animation generation models. These models can complete the transfer of facial movements and lip movements based on keypoint-driven and spatiotemporal generation methods, and are currently the mainstream choice for creating 2D digital human assets.

[0003] Existing facial motion transfer models struggle to maintain stable lip and tooth display during long-term frame-by-frame generation, easily exhibiting issues such as tooth flickering, misalignment, and lip contour distortion. Furthermore, lip movements corresponding to the same phoneme and viewpoint often show inconsistencies, disrupting visual continuity. Frame-by-frame video frames are prone to motion jumps, facial and lip jitter, and unnatural transitions in lip opening and closing rhythms and facial postures between frames, resulting in playback smoothness that fails to meet the standards for professional digital human footage.

[0004] Mainstream solutions focus solely on image generation, neglecting to design dedicated frame-level quality inspection, anomaly identification, and automated repair processes for the transfer learning results. Defective frames and abnormal segments cannot be automatically identified and corrected by the system, requiring manual frame-by-frame screening. Due to the lack of automated quality control, significant manpower is required for frame-by-frame inspection, localized modifications, and rework adjustments to select qualified materials and correct image defects, substantially increasing the labor costs of material production and hindering the efficiency of large-scale, batch production of digital human materials. Existing solutions cannot utilize the defective data generated during the generation process; failed video frames and segments are directly discarded and cannot be converted into training data for iterative optimization of the transfer learning model. This leads to recurring model defects and hinders continuous performance improvement. Summary of the Invention

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: A method for optimizing a face motion transfer model based on frame-level consistency verification includes: constructing a face motion transfer optimization model based on a neural network, decoupling mouth motion generation from identity feature representation, and establishing a machine learning optimization architecture that includes static morphological priors and dynamic keypoint constraints; designing a frame-level consistency discrimination neural network, constructing a multi-dimensional quantitative scoring model based on temporal stability, consistency of mouth shapes within the same category, consistency of regional textures, consistency of identity features, and consistency of geometric constraints, and outputting anomaly classification and quality ranking results; and constructing a contrastive learning and incremental optimization paradigm, using abnormal frames as anchor samples, high-quality repaired frames and stable samples as positive samples, and perturbed and degraded samples as negative samples to construct a ranking. Loss and consistency constraint loss are used; a local adapter fine-tuning mechanism is adopted, freezing the model identity encoding backbone and only updating the parameters of the mouth action branch, temporal smoothing adapter, and quality prediction head with small steps, and integrating key point supervision loss, temporal continuity loss, region consistency loss and feature preservation loss; based on the mini-batch incremental learning strategy, high-frequency error samples are used for iterative optimization, and the temporal jitter, detail collapse and lip shape inconsistency defects of the transfer model are continuously corrected through closed-loop self-supervised learning; through multiple rounds of model iteration and quality verification, the temporal robustness and detail generation consistency of the neural network in face action transfer are improved, and an optimized action transfer model for digital human materials is obtained.

[0006] A face motion transfer model optimization system based on frame-level consistency verification is provided. The system is used to execute executable instructions to perform the aforementioned face motion transfer model optimization method based on frame-level consistency verification.

[0007] Its beneficial effects are as follows: Based on a neural network architecture that decouples identity and action, a hierarchical training strategy is adopted to freeze the identity encoding backbone, performing lightweight iterations only on local modules related to mouth, timing, and quality. A five-dimensional frame-level consistency discrimination network is constructed, coupled with quantitative scoring and anomaly classification mechanisms to achieve automated quality inspection of video frames; anchor samples and positive / negative sample groups are built based on defective frames, and multiple loss functions are designed using contrastive learning. A small-batch incremental learning mode is adopted, feeding back detected and identified defective samples for closed-loop self-supervised training of the model, continuously correcting issues such as timing jitter, tooth flickering, and inconsistent mouth shapes. The entire solution connects the entire chain of transfer generation, quality inspection, repair, sample feedback, and model iteration, achieving automated and continuous optimization of the face action transfer model, suitable for large-scale production scenarios of 2D digital human materials.

[0008] This application effectively improves issues such as lip trembling, teeth flickering, inconsistent homophone lip movements, and identity drift caused by frame-by-frame generation through multi-dimensional frame-level verification and multi-loss constraints. The temporal coherence and detail consistency of the images are significantly improved. The entire process of quality inspection, repair, and sample collection is automated, eliminating the workload of manual frame-by-frame screening and modification, and greatly improving the efficiency of digital human material production. Self-supervised incremental training is conducted using defective samples generated from business operations, eliminating the need for additional manual annotation and enabling closed-loop continuous optimization of model performance. A global module freezing strategy is adopted, ensuring that the facial identity and inherent style of the target person are not altered during training, balancing optimization effects with the model's original capabilities. Attached Figure Description

[0009] Figure 1 A flowchart of a face motion transfer model optimization method based on frame-level consistency verification provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a face motion migration model optimization system based on frame-level consistency verification, provided in an embodiment of the present invention. Detailed Implementation

[0010] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention. Figure 1 This paper describes a face motion migration model optimization method based on frame-level consistency verification according to an exemplary embodiment of this application.

[0011] In this application embodiment, a face motion transfer model optimization method based on frame-level consistency verification is provided, such as... Figure 1 As shown: S101, construct a face motion transfer optimization model based on neural networks, decouple mouth motion generation from identity feature representation, and establish a machine learning optimization architecture that includes static morphological priors and dynamic key point constraints.

[0012] In one implementation, combining the identity-action decoupling design of the face motion transfer model, mouth feature constraint rules, and local fine-tuning requirements, and relying on the hierarchical training mechanism of the base model, a globally fixed module and a locally updated module are distinguished. An open-source face transfer network with the characteristics of identity representation and mutual decoupling of facial expressions and actions is selected as the overall base model. This type of model can use mainstream architectures such as keypoint-driven, spatiotemporal diffusion, and locally editable architectures, and can independently parse facial identity information and dynamic facial expressions and actions, meeting the basic conditions for modular training and optimization. The entire base model is divided into two levels, a globally fixed module and a locally updated module, according to functional positioning and training update rules. The two levels adopt a sequential layout in the network structure, and the data flows unidirectionally from front to back.

[0013] The global fixation module consists of a face backbone encoder, an identity feature branch, and a global facial style branch. The face backbone encoder is responsible for extracting features from the complete face image, compressing and representing basic visual information. The identity feature branch uses mainstream face feature extraction methods, determining identity status by calculating the cosine similarity of feature vectors. The calculation method is: Identity Consistency Index = 1 - Cosine Distance of Feature Vectors. The threshold for this index is set to 0.15; an index value exceeding the threshold indicates an identity shift. The global facial style branch uniformly retains the inherent style information of the target person, such as skin tone, contours, and overall facial proportions. Throughout the entire training and inference phases, all network parameters in the global fixation module remain locked, without gradient calculation or parameter backpropagation. It retains the complete facial identity information and basic visual features extracted from the target person's reference images and videos, avoiding identity drift and facial style changes during training.

[0014] The local update module includes a mouth feature branch, a lip keypoint regression head, a tooth consistency branch, a temporal smoothing adapter, and a quality prediction head. This layer is the core area for iterative optimization of model parameters. The mouth feature branch specifically extracts facial mouth region data, extracting static features such as mouth width, lip thickness, lip peak shape, and mouth width-to-height ratio, while also analyzing dynamic features such as mouth opening amplitude and opening / closing rhythm. The lip keypoint regression head predicts coordinates for 6, 20, or 40 sets of keypoints on the lips, combining the Euclidean distance between adjacent frames and the variance of the three-frame window to calculate lip temporal stability. The threshold for this indicator is set to 0.18; exceeding this threshold indicates a lip jitter problem. The tooth consistency branch extracts the tooth mask region and calculates the tooth consistency index by comprehensively considering the mean difference in regional brightness, texture gradient variance, and contour Hausdorff distance. The corresponding threshold is set to 0.25 to control issues such as flickering, misalignment, and breakdown in the tooth region. The temporal smoothing adapter is used to constrain the amplitude of motion changes between consecutive frames, stipulating that the maximum change in mouth opening per frame should not exceed 0.25, thereby suppressing inter-frame jump phenomena. The quality prediction head calculates the comprehensive score of a single frame using a multi-dimensional weighted index method. The calculation rule is: Comprehensive Score = 1.0 - (lip temporal index × 0.25 + mouth shape consistency index × 0.15 + teeth consistency index × 0.25 + identity consistency index × 0.15 + geometric consistency index × 0.20). The pass / fail threshold for the comprehensive score is set at 0.65. A score below this value is judged as an unqualified frame, thus completing the single-frame quality assessment.

[0015] The network data flow follows a sequential logic. After the global fixed module completes feature extraction, it uses identity features and global facial style features as input data and transmits them completely to the subsequent local update module. Under the premise that the identity features are not tampered with, the local update module completes mouth motion generation, detail optimization, temporal correction, and quality evaluation, ensuring the stable transmission of the target person's identity features across all network layers throughout the process. During model training, only the local update module performs parameter updates; the global fixed module remains frozen. Iterative optimization of mouth motion, tooth details, inter-frame temporal relationships, and output quality is achieved solely through the local module, correcting transfer defects while completely avoiding the risk of damaging the person's identity and facial style.

[0016] Using static mouth features of the target person as prior and dynamic parameters of facial key points in the source video as dual constraints, the feature extraction rules and feature fusion logic are clearly defined. The system is configured with two types of core input constraints, corresponding to the inherent mouth shape of the target person and the dynamic actions of the source video, respectively. Both types of data serve as the pre-input of the face transfer network, supporting the model to complete the generation logic of decoupling identity and action. The first type is the static mouth feature prior of the target person. This type of data is extracted from a single reference image or a complete reference video of the target person and is used to define the person's unique mouth shape standard. The specific extracted content includes the overall mouth width in the closed state, the thickness of the upper and lower lips, the shape of the lip peaks, the probability of teeth being exposed in the normal state, and the coordinates of the left and right corners of the mouth. All static features are numerically solidified to form a unique shape benchmark for the person. For example, the standard value of the mouth width of a single person is fixed, and the probability range of teeth being exposed in the normal state is set between 0.2 and 0.4. The mouth size and tooth state generated by the subsequent model must be constrained against this benchmark. The parameters such as mouth width, mouth height, and lip thickness are allowed to fluctuate within ±10% of the benchmark value. If the value exceeds the range, it is judged as a shape abnormality. This set of features is bound to the entire video sequence and will not change with frame changes, serving as a global morphological constraint throughout the entire model inference and training process.

[0017] The second category consists of dynamic parameters of facial key points in the source video, obtained frame-by-frame from the driving video. Frame-by-frame, coordinates of 68 key points on the entire face and 20 key points on the lips are extracted. Based on this coordinate data, multiple dynamic indicators are further calculated, including the instantaneous opening and closing of the mouth, the frame-by-frame rhythm of opening and closing, facial pitch, yaw, and roll—three types of local posture angles. These dynamic parameters change sequentially with the video frames, with each frame corresponding to an independent set of key point coordinates and motion parameters, used to convey the trend of the source person's facial expressions and movements. In posture determination, the normal range for head posture angle deviation is set within 10°; deviations exceeding this range are considered deviations from the driving standard.

[0018] Both types of constraint data require unified temporal alignment. A three-level alignment method using timestamps and frame numbers is employed, with the source driving video frame as the absolute temporal reference. The video's standard frame rate is set to 25fps or 30fps, and audio is processed by 10ms frame shifts. Single-frame audio data, phoneme tags, viewpoint IDs, source video keypoint dynamic parameters, and target character's mouth static features are bound to the same frame number, constructing a one-to-one correspondence between frames, audio, viewpoint, dynamic parameters, and static features, ensuring complete temporal matching for each set of network input data.

[0019] After temporal alignment, the combined data is fed into a face transfer neural network, where static and dynamic features are fused. The fusion process uses the static mouth shape of the target person as a base, superimposing dynamic motion features from the source video. This preserves the unique lip shape, teeth, and corners of the mouth of the target person while replicating the mouth opening and closing, facial rotation, and other motion changes that drive the video. The fusion logic is accompanied by multiple constraint rules, controlling the variation in mouth opening and closing amplitude between frames to within 0.25 to avoid abrupt changes in motion. Mouth parameters corresponding to the same phoneme and viewpoint are distanced using the average of historical stable samples, with a corresponding judgment threshold of 0.22, ensuring consistency in mouth shapes for similar pronunciations. Through feature fusion, the model balances morphological uniqueness and motion tracking, achieving a synchronized matching between the target person's inherent mouth shape and the source person's dynamic motion, reducing mouth shape distortion, motion jumps, and morphological disproportion issues from the source.

[0020] A neural network architecture was constructed by combining an identity encoding freeze strategy, incremental update rules for the mouth branch, and multi-dimensional loss constraints. During the formal training phase, a hierarchical training rule was implemented, with control scopes divided according to module function and parameter update permissions. The global identity encoding branch and the global feature layer were set to a frozen state. This part includes the face backbone encoder, identity feature extraction unit, and facial global style unit; the corresponding network weights and bias parameters were locked throughout the process and did not participate in gradient calculation or backpropagation. Only the mouth-related local branches, the temporal smoothing adapter, and the quality prediction head were open in the trainable region. These units, as functional enhancement modules, are specifically responsible for mouth motion generation, inter-frame temporal optimization, and image quality judgment. All parameter iterations and updates are completed within this scope, thus avoiding issues such as changes in person identity and overall facial style distortion during training from an architectural perspective.

[0021] The entire training process is supported by a complete multi-dimensional loss constraint system. Each loss item has a clear division of labor and works together to control the model's output. Its specific components and functions are as follows: The system uses the coordinate difference between the lip keypoints and the reference keypoints in the transfer result as the calculation basis to constrain the lip point positions from shifting, ensuring that the mouth movement trajectory remains consistent with the driving source; it calculates the overall shape of the mouth, the distance between the upper and lower lips, and the mouth's aspect ratio to limit deformations such as stretching, compression, and malformation; and it combines the average brightness of the tooth area, texture gradient, and contour distance for comprehensive calculation, with a threshold set to 0. A loss term of 0.25 is used to determine tooth flickering, misalignment, or damage when the index exceeds this value, and this loss term is used to complete the constraint. The identity consistency index is calculated using the cosine relationship of facial feature vectors, with the formula being: Identity Consistency Index = 1 - Cosine Distance of Feature Vectors, corresponding to an anomaly threshold of 0.15, effectively suppressing identity drift. The feature difference between adjacent frames is calculated, limiting the mouth opening and closing variation in a single frame to no more than 0.25, avoiding abrupt changes in action and image transitions between frames. The overlap between the predicted mask and the standard mask is compared to ensure accurate mouth region segmentation and prevent region boundary confusion. All loss terms are used in combination according to preset weights to jointly constrain the model output and correct various generation defects item by item.

[0022] The entire network forms a closed-loop forward inference and backward iterative link, with a clear data flow and parameter update process. The first stage is data input, which synchronously connects the original driving video and the target person's reference image to the network front end. The second stage performs global feature extraction, using the frozen global identity encoding branch and global feature layer to extract the target person's unique identity features and basic facial visual features. The third stage completes feature fusion, fusing the previously extracted static features of the target person's mouth and the dynamic features of key points in the source video, integrating morphological and action constraint information. The fourth stage generates output results, with the fused feature data fed into the trainable local branch, temporal adapter, and quality prediction head, outputting the final facial mouth action image and corresponding quality score. The fifth stage calculates the loss error, calling the six types of loss functions mentioned above and calculating the deviation value of the current output against the standard data. The sixth stage performs parameter backpropagation, with the error signal only transmitted to the mouth local branch, temporal adapter, and quality prediction head, updating only the parameters of that part, while the parameters of the globally related modules remain frozen. The entire process loops continuously, with each round of inference and calculation optimizing the capabilities of local modules based on errors. While always preserving the identity of the target person and their basic facial features, it continuously optimizes the mouth shape, teeth condition, inter-frame smoothness, and overall generation quality.

[0023] The model integrates hierarchical division criteria, inter-module connection methods, feature extraction execution specifications, dual-constraint access and linkage logic, hierarchical training execution schemes, multiple loss combinations, and full training control requirements to form a unified model building and operation specification. In terms of module hierarchy, it distinguishes between globally fixed modules and locally updated modules. The globally fixed module includes the face backbone encoder, identity feature branch, and facial global style branch, locking network parameters throughout the process. The locally updated module consists of the mouth feature branch, lip keypoint regression head, tooth consistency branch, temporal smoothing adapter, and quality prediction head, serving as the main body for parameter iteration. Modules are connected sequentially from front to back, with feature data output from the global module being transmitted unidirectionally to the subsequent local modules, ensuring a fixed data flow path.

[0024] The feature extraction process follows a unified standard. Static features such as mouth width, lip thickness, lip peak shape, tooth visibility probability, and mouth corner position are extracted for the target individual. For the source video, 68 full-face key points and 20 lip key points are extracted frame-by-frame, and dynamic parameters such as mouth opening and head posture are calculated. The two types of features are bound and aligned according to a unified timeline composed of the source video frame number, audio frame, and viewpoint ID. The video uses a standard frame rate of 25fps or 30fps, and the audio is set with a 10ms frame shift to ensure complete temporal matching of multiple data types.

[0025] Layered training clearly defines execution boundaries. The global identity encoding branch and global feature layer are frozen throughout both training and inference phases, with only local modules open for parameter updates. This fundamentally avoids identity drift and facial style changes during training. The loss system is used in combination according to predetermined rules, including keypoint supervision loss, lip contour loss, tooth region loss, identity preservation loss, temporal continuity loss, and mouth masking loss. Each loss corresponds to different control objectives: tooth consistency threshold is 0.25, identity consistency threshold is 0.15, lip temporal stability threshold is 0.15, inter-frame mouth opening / closing variation is limited to no more than 0.25, and the comprehensive score threshold is set to 0.65. All these indicators collectively constrain the model's output performance.

[0026] Training management uses a unified configuration of parameters, employing the AdamW optimizer with a base learning rate of 2e-4, a batch size of 4 or 8, and 5 to 10 training epochs per iteration. Gradient clipping is enabled, limiting the maximum norm of the gradient to 1.0, and a weight decay coefficient of 1e-5 is used to control the training amplitude and prevent parameter oscillations and model overfitting.

[0027] Based on the aforementioned complete set of specifications, the face motion transfer network was instantiated and deployed. A unified method for initializing network parameters was set, and the entire training and inference operation mechanism was determined by combining hierarchical update logic, loss calculation logic, and data flow logic. The final optimized face motion transfer model can completely retain the inherent features of the target person's face, such as facial proportions, and follows the motion rhythm and key point trajectory of the source video, stably outputting consistent mouth shape, tooth state, and inter-frame transition effects. This model can be directly applied to 2D digital human material production scenarios. The produced video material meets all quantitative standards for material entry into the database, with a frame-level pass rate of over 99%, an audio-visual matching degree of no less than 0.90, and an automatic repair frame rate controlled within 5%, fully adapting to the actual needs of digital content production.

[0028] S102, design a frame-level consistency discrimination neural network, and build a multi-dimensional quantitative scoring model based on temporal stability, lip shape consistency of the same category, regional texture consistency, identity feature consistency, and geometric constraint consistency, and output anomaly classification and quality ranking results.

[0029] In one implementation, based on the actual business needs of automated quality inspection of face migration video frames, a dedicated frame-level consistency discrimination network and a supporting quantitative scoring system are built according to a complete execution process of multi-dimensional evaluation, score fusion, anomaly detection, and hierarchical labeling. This network is embedded as an independent functional module in the entire material production chain, possessing dedicated data input and output paths. During the data access phase, the network input simultaneously receives multiple types of time-aligned business data. The entire set of data uses the source-driven video frame number as a unified time-series benchmark. The video uses a standard frame rate of 25fps or 30fps, and the audio is framed according to a 10ms frame shift, obtaining frame-by-frame view position information through phoneme recognition mapping. The final accessed data includes the face migration output video frame, corresponding audio features, frame-by-frame view position labels, source video reference facial key point sequences, and prior static features of the target person's mouth. All data is bound one-to-one according to frame number, ensuring that various reference information corresponds to the migration frame at the same time, forming a complete input chain.

[0030] The frame-level consistency discrimination network is divided into five evaluation dimensions: temporal stability of lip keypoints, homophone co-viewpoint lip shape consistency, tooth region consistency, facial identity stability, and reference keypoint geometric consistency. All detection indicators are uniformly normalized to a value range of 0 to 1, with higher values ​​indicating a higher degree of abnormality. Each dimension is configured with an independent calculation method and judgment threshold. For lip keypoint temporal stability, 20 coordinates of the lips are selected, and the results are calculated by combining the Euclidean distance between adjacent frames and the variance of the three-frame sliding window. The judgment threshold is set to 0.18. Exceeding the threshold indicates that the lips are jittering or drifting. For homophone co-viewpoint lip shape consistency, the average value of historical normal mouth parameters under co-viewpoint is extracted, and the distance between the current frame parameter and the average value is calculated. The judgment threshold is set to 0.22. Exceeding the threshold indicates inconsistent lip shape performance. For tooth region consistency, the mean difference of brightness in the tooth mask region, texture gradient variance, and contour Hausdorff distance are comprehensively calculated. The judgment threshold is set to 0.25. Exceeding the threshold is considered as tooth flickering, misalignment, or collapse.

[0031] Facial identity stability is assessed by calculating cosine similarity using facial feature vectors. The calculation rule is: Identity Consistency Index = 1 - Cosine Distance of Feature Vectors, with a threshold of 0.15. Frames exceeding this threshold are considered to have identity drift. Geometric consistency is also considered, incorporating differences in mouth corner position, upper and lower lip distance, mouth aspect ratio, and head pose. A weighted average of these factors is used, with a threshold of 0.20. Frames exceeding this threshold indicate deviations from the driving trajectory of mouth movements. After single-dimensional detection, a score fusion calculation is performed. The comprehensive score is calculated as: Comprehensive Score = 1.0 - (0.25 × Lip Temporal Index + 0.15 × Mouth Shape Consistency Index + 0.25 × Tooth Region Index + 0.15 × Identity Index + 0.20 × Geometric Index). The weights in the formula are fixed. The comprehensive score ranges from 0 to 1, with higher scores indicating better image quality. A globally unified passing score of 0.65 is used; video frames with comprehensive scores below this value are marked as unqualified.

[0032] After score calculation, anomaly detection and classification are performed. Anomalies are prioritized from highest to lowest as follows: identity anomalies, tooth region anomalies, geometrical following anomalies, lip temporal jitter, and co-view inconsistencies. When multiple anomalies exist in a single frame, only the highest priority anomaly label is retained. Anomalies are also categorized based on the number of consecutive frames: isolated single-frame anomalies are marked as repairable frames; two or more consecutive frames are marked as risky segments; and more than five consecutive frames are marked as unrepairable segments. The network's final output includes the judgment result for each frame, anomaly type label, quality score, and segment classification information.

[0033] The network's output directly connects to the automatic repair module and the quality acceptance module. Frames marked as repairable and risky segments are transferred to the automatic repair stage for targeted processing. Unrepairable segments and content that still fails to meet standards after repair are sent to a low-quality sample pool for subsequent model optimization. Video frames that pass acceptance are directly stored in the media library. Through data exchange between the front and back modules, deep linkage is achieved between the quality inspection stage and the entire business chain of migration generation, automatic repair, sample return, and media storage, ensuring the orderly operation of the entire face motion migration material production process.

[0034] Five consistency evaluation directions were defined: temporal stability of lip key points, homophone / co-optical position consistency, tooth region consistency, facial identity consistency, and reference key points. Evaluation rules, normalization requirements, and corresponding judgment thresholds were established for each dimension. In accordance with the requirements of automated quality inspection of face migration video frames, five core evaluation dimensions were identified: temporal stability of lip key points, homophone / co-optical position consistency of lip shape, tooth region consistency, facial identity stability, and geometric consistency of reference key points. The detection data for all dimensions were standardized and mapped to a numerical range of 0 to 1. Higher values ​​indicate more prominent problems. A specific anomaly judgment threshold was set for each dimension, and the calculation results were used to determine whether a single frame contained corresponding defects.

[0035] The first dimension is the temporal stability of lip keypoints. During detection, 20 sets of lip feature points are selected. First, the Euclidean distance between adjacent feature points is calculated, and then the local variance is calculated by combining data from three consecutive frames to obtain the overall value of this indicator. The anomaly detection threshold for this dimension is set to 0.18. When the calculated result is greater than 0.18, it is determined that the current frame has issues such as lip trembling or keypoint drift.

[0036] The second dimension is the consistency of lip movements within the same phoneme and viewpoint. First, the lip parameters of all historical normal frames under the same phoneme and viewpoint are statistically analyzed and averaged. Then, the distance between the current frame's lip parameters and the average is calculated to obtain the corresponding index. The anomaly threshold for this dimension is set at 0.22. If the index value exceeds the threshold, it indicates inconsistency in the visual presentation for the same pronunciation and lip movement category.

[0037] The third dimension is tooth region consistency. First, the tooth detection area is delineated using a mask. Then, the mean difference in pixel brightness, texture gradient variance, and contour Hausdorff distance within the area are calculated sequentially. These three results are then fused to obtain a comprehensive index. The anomaly threshold for this dimension is set at 0.25. Values ​​exceeding this threshold are considered to indicate flickering, misalignment, or image corruption in the tooth region.

[0038] The fourth dimension is facial identity stability. It uses facial feature vector comparison, and the calculation formula is: Identity Consistency Index = 1 - Cosine Distance of Facial Feature Vector. The anomaly threshold for this dimension is set at 0.15. When the index is greater than 0.15, it indicates that the facial features of the person in the image have changed, indicating an identity drift problem.

[0039] The fifth dimension is the geometric consistency of the reference keypoints. The differences in mouth corner position, upper and lower lip distance, mouth aspect ratio, and head pose between the current frame and the reference frame are calculated sequentially, and then weighted and fused with weights of 0.3, 0.3, 0.2, and 0.2 respectively to obtain the final metric. The anomaly threshold for this dimension is set to 0.20; exceeding this threshold indicates that the mouth shape and facial pose deviate from the motion trajectory of the driving video.

[0040] The five evaluation dimensions operate independently, each completing data calculation, normalization, and threshold comparison, and then outputting its corresponding detection conclusion. All subsequent single-dimensional results will participate in the comprehensive score calculation, providing complete data support for video frame quality grading, abnormal segment marking, and subsequent repair processing.

[0041] The results from each dimension are integrated according to predetermined weights to calculate a comprehensive score for each frame, with 0.65 serving as the passing threshold. After calculating and normalizing the individual indicators for the five dimensions, a weighted calculation is performed according to pre-set weight ratios to obtain the comprehensive consistency score for each frame. The weights for the five dimensions are as follows: temporal stability of lip keypoints (0.25), homophone-co-optical position lip shape consistency (0.15), tooth region consistency (0.25), facial identity stability (0.15), and reference keypoint geometric consistency (0.20).

[0042] The formula for calculating the overall consistency score is: Overall Score = 1.0 (0.25 × Lip Timing Index + 0.15 × Mouth Shape Consistency Index + 0.25 × Tooth Region Index + 0.15 × Identity Index + 0.20 × Geometric Index); All individual indicators and the overall score are uniformly within the range of 0 to 1. The higher the score, the better the overall quality and consistency of the face migration frame. The system sets a globally unified passing score of 0.65 as the evaluation threshold, comparing the calculated single-frame overall score with this standard. When the overall score is greater than or equal to 0.65, the frame is judged as a normal frame and can directly enter the subsequent quality acceptance stage; when the overall score is less than 0.65, the frame is judged as a non-compliant frame, marked, and transferred to the automatic repair stage for further processing. This calculation and judgment logic runs automatically throughout the entire process, providing a quantitative basis for video frame quality grading and subsequent process scheduling.

[0043] Based on anomaly priority and frame segmentation rules, non-compliant frames are classified and labeled, distinguishing between repairable frames, risky segments, and unrepairable segments, and outputting an anomaly list and quality ranking results. After completing the comprehensive score determination for a single frame and identifying non-compliant frames, the system performs classification and labeling according to preset anomaly priority rules. The anomaly levels are sorted from high to low as identity anomaly, tooth region anomaly, geometrical following anomaly, lip temporal jitter, and co-view inconsistency. When multiple anomaly indicators are triggered simultaneously in a single frame, the system only retains the highest priority anomaly type as the official label for the current frame, and does not add other lower-level anomalies, thus ensuring clear and consistent anomaly definitions. Among them, identity anomaly corresponds to a feature vector cosine distance calculation result greater than 0.15, tooth region anomaly corresponds to a comprehensive index greater than 0.25, geometrical following anomaly corresponds to a weighted deviation value greater than 0.20, lip temporal jitter corresponds to a 3-frame window variance and adjacent frame distance combination index greater than 0.18, and co-view inconsistency corresponds to a mouth shape parameter mean distance greater than 0.22. All thresholds are fixed execution standards.

[0044] After anomaly type labeling is completed, the system performs segment division based on the continuous state of the frame sequence. Using the original video frame number as the sorting basis, the abnormal frames are distinguished according to their continuity: frames with only a single abnormal frame are uniformly marked as repairable frames; segments with two or more consecutive abnormal frames are defined as risky segments; when the number of consecutive abnormal frames exceeds 5, they are directly classified as unrepairable segments, and these segments will not enter the regular repair process.

[0045] The entire judgment logic is executed automatically, ultimately generating two types of output. The first type is a full list of anomaly types, recording the frame number, anomaly category, and anomaly level for each frame. The second type is the global frame quality ranking result, which is based on the comprehensive consistency score in the range of 0 to 1. The higher the score, the better the picture quality.

[0046] The above outputs are respectively connected to the automatic repair module and the low-quality sample pool. Data marked as repairable frames and risky segments are directly transferred to the automatic repair stage for local optimization according to the corresponding repair strategy; unrepairable segments and content that still does not meet the standards after repair are uniformly collected into the low-quality sample pool for subsequent structured sample construction and incremental model training. The entire set of annotation and classification rules runs through the entire chain of quality inspection, repair, and sample return, providing clear data basis for each subsequent business action.

[0047] The system integrates all detection and judgment rules to form a unified and implementable quality inspection execution standard. The standard comprehensively covers five consistency evaluation directions, anomaly thresholds for each dimension, comprehensive score calculation weights, anomaly labeling priorities, and frame segmentation standards. The five evaluation directions include temporal stability of lip keypoints, homophone-to-visual-position lip shape consistency, tooth region consistency, facial identity stability, and reference keypoint geometric consistency. All detection indicators are uniformly normalized to a numerical range of 0 to 1. Specifically, the threshold for lip keypoint temporal stability is set at 0.18, the threshold for homophone-to-visual-position lip shape consistency is set at 0.22, the threshold for tooth region consistency is set at 0.25, the threshold for facial identity stability is set at 0.15, and the threshold for reference keypoint geometric consistency is set at 0.20.

[0048] The overall score is calculated using a fixed weighting, and the formula is: Overall Score = 1.0 (0.25 × lip timing index + 0.15 × mouth shape consistency index + 0.25 × tooth region index + 0.15 × identity index + 0.20 × geometric index), with 0.65 set as the passing score globally. Anomaly labeling priorities, from highest to lowest, are: identity anomaly, tooth region anomaly, geometrical following anomaly, lip timing jitter, and co-view inconsistency. When multiple anomalies occur simultaneously in a single frame, only the highest priority label is retained. Frame segments are clearly defined: isolated anomalies in a single frame are considered repairable frames; two or more consecutive frames with anomalies are considered risky segments; and segments with anomalies exceeding five consecutive frames are directly considered unrepairable segments.

[0049] The entire specification serves as the operational basis for the frame-level consistency discrimination network, driving the network to complete the entire automated process. The network reads the face migration frame by frame in chronological order, combining synchronized audio, viewpoint, reference key points, and prior data on the target person's static mouth, sequentially completing single-dimensional index calculation, comprehensive score calculation, anomaly identification, and level classification. The quality inspection process is deeply integrated with the entire material production chain. After inspection and rating, video frames and segments are distributed according to the results. Frames deemed normal are directly transferred to the quality acceptance stage; content marked as repairable frames and risky segments is sent to the automatic repair module for local optimization; unrepairable segments and content that still fails to meet standards after repair are uniformly collected into a low-quality sample pool. The collected defective samples are further processed into structured sample units for subsequent comparative learning and incremental model optimization. The entire quality inspection system connects the migration generation, automatic repair, quality acceptance, sample return, and model iteration stages, forming a closed-loop operation mode to continuously ensure the production efficiency and output quality of digital human migration materials.

[0050] S103 constructs a contrastive learning and incremental optimization paradigm, using abnormal frames as anchor samples, high-quality repaired frames and stable samples as positive samples, and perturbation and degradation samples as negative samples, and constructs ranking loss and consistency constraint loss.

[0051] In one implementation, combining the actual business goal of iterative optimization of the face motion transfer model, the system invokes the deployed low-quality sample collection mechanism, standardized sample grouping specifications, and loss function construction rules to unify the sample partitioning standards throughout the entire process. The entire sample collection process interfaces with the output data of the frame-level consistency detection module and the automatic repair module, using only various abnormal images generated in the face motion transfer process as sample sources. This ensures that the samples accurately correspond to the existing defects in the model, laying a solid foundation for subsequent sample group construction, loss calculation, and model parameter iteration.

[0052] The system uses defective frames and continuous abnormal defect segments identified during quality inspection as raw data. It categorizes these data into three core samples based on functional and quality differences: anchor samples, positive samples, and perturbation negative samples. Positive samples are further subdivided into locally redrawn positive samples and stable reference positive samples. The first category, anchor samples, directly selects defective frames and abnormal segments that have passed quality acceptance. These samples directly reflect the problems with the current model generation and serve as the core reference for the entire sample system. All comparative learning and constraint training revolve around anchor samples. Each anchor sample includes a corresponding comprehensive score, anomaly type label, and defect severity score. Segment-level anchor samples simultaneously record the start and end frame numbers of the anomaly and the number of consecutive frames.

[0053] The second category is positive samples, which include two different generation methods. Local redrawing positive samples use anchor samples as a base, adhering to identity preservation rules and facial region segmentation requirements. The content of the non-mouth areas of the face remains unchanged. Guided by mouth masks, corresponding phoneme tags, viewpoint tags, and reference facial keypoint sequences, local editing is used to redraw and optimize the mouth area. During the redrawing process, the mouth shape and teeth state must match the static features of the target person. Parameters such as mouth width and lip thickness are controlled within ±10% of the baseline value, and the head posture deviation does not exceed 10°, ultimately generating a high-quality restored image. Stable reference positive samples are obtained from a material library and a collection of historical normal frames. The search scope is limited to the previous and subsequent normal frames, frames with the same phoneme, frames with the same viewpoint, and historical high-quality template frames of the current target person. During the search, the person's identity, audiovisual information, mouth opening / closing state, and head posture are matched sequentially. The mouth opening / closing difference is required to be less than 0.2, and the geometric distance of keypoints to be less than 0.2. The selected images have standard shapes and smooth movements, serving as the optimal reference.

[0054] The third category is perturbation negative samples, which are obtained by procedurally processing the original anchor samples. Multiple degrading perturbations are actively applied to the anchor samples to artificially simulate typical model defects such as mouth misalignment, tooth flickering, local texture damage, inter-frame temporal jumps, and disordered contour boundaries, deliberately lowering the image quality. The perturbation operations are kept at a uniform intensity to ensure that the overall performance of the negative samples is significantly worse than that of the anchor samples, clearly defining the generation effects that the model needs to avoid.

[0055] After all samples are segmented, a unified alignment process is performed. Using the source-driven video frame number and timestamp as a common temporal reference, the video uses a standard frame rate of 25fps or 30fps, and the audio is segmented with a 10ms frame shift to ensure complete consistency in face bounding box position, image scale, head pose, and temporal semantics among anchor samples, both types of positive samples, and perturbation negative samples. Single-frame samples and continuous segment samples are combined according to grouping rules to form standardized sample units. Each unit synchronously records full-dimensional information such as audio segments, phonemes, view positions, keypoints, anomaly labels, and quality scores, which are then directly input into the loss calculation module to support the implementation of the contrastive learning paradigm.

[0056] The system first establishes standardized structured sample units for various defective frames and abnormal segments collected from the low-quality sample pool. Each sample unit fully incorporates all-dimensional related data, ensuring data traceability and computability. The unit stores the corresponding audio segment, audio timestamp, phoneme text, viewpoint number, 68 key points of the face and 20 key points of the lips from the source video reference, face migration generated image, mouth binary mask, lip key point coordinates, head pitch, yaw, and roll posture parameters, abnormality type label, defect severity score, single-frame comprehensive consistency score, and corresponding stable reference frame number. The defect severity score ranges from 0.2 to 1.0, categorized by defect level: minor abnormalities (0.2-0.4), general abnormalities (0.4-0.6), severe abnormalities (0.6-0.8), and fatal abnormalities (0.8-1.0). The comprehensive consistency score follows a predetermined calculation method, with a passing threshold of 0.65.

[0057] After completing the sample unit construction, the training samples are formally divided into different types. The migration anomaly frames and continuous anomaly segments marked in the frame-level consistency detection are uniformly designated as anchor samples. Anchor samples are the core reference benchmarks for the entire sample set. All comparative learning and constraint training logic revolves around anchor samples. At the same time, the comprehensive score, anomaly label and severity score corresponding to the sample are bound to it for subsequent loss calculation and weight allocation.

[0058] Two types of positive samples are generated based on anchor samples, both adhering to the fundamental rule of not altering the target person's identity. The first type is a local redraw positive sample, which utilizes a local editing model for image optimization. During the editing process, all areas of the face except the mouth are locked, and only the mouth area is redrawn. The editing stage uses a mouth mask to define the operation range, combined with the trajectory of reference key points in the same frame, current phoneme and view position information as guidance. The redrawn mouth parameters need to match the static features of the target person, with the fluctuation range of mouth width, lip thickness, and other size parameters relative to the baseline value controlled within ±10%, the head posture angle deviation not exceeding 10°, and the mouth opening and closing difference less than 0.2. The final result is a locally redraw positive sample with a neat image and normal details.

[0059] The second category is stable reference positive samples, extracted from the material library and historical videos using multi-condition screening rules. The search scope is limited to the video sequence of the current target person, prioritizing normal frames adjacent to the anchor sample, and then expanding to historical stable frames with the same phonemes and viewpoints across the entire network, as well as high-quality template frames already in the library. During screening, identity matching, viewpoint and phoneme matching, mouth opening and closing state matching, and head posture matching are performed sequentially. It is required that the difference in mouth opening and closing between the reference frame and the anchor sample is less than 0.2, the geometric distance of key points is less than 0.2, and the head posture angle difference is less than 10°. Finally, images with standard shapes and smooth movements are selected as stable reference positive samples.

[0060] Degraded negative samples are generated based on anchor samples, and common defects in the model are simulated by programmatically applying perturbations. Specific perturbation types include misalignment of key points in the mouth, abnormal brightness and texture in the tooth area, broken mouth contour boundaries, and temporal jumps in consecutive frames. All perturbation operations are kept at a uniform intensity. The quality of the processed image is significantly lower than that of the original anchor samples, thus clarifying the generation effects that need to be avoided in model training.

[0061] For segments with anomalies appearing in multiple consecutive frames, segment-level sample groups are constructed according to temporal rules. A sliding method with a fixed temporal window of 5 frames and a step size of 2 frames is used for combination, and the segment overlap rate is set to 60%. Samples are extracted with the abnormal interval as the center, which not only preserves the defect features of a single frame, but also captures the action transition rules between frames in a short period of time, thus adapting to the training requirements of temporal loss.

[0062] After all types of samples are assembled, a unified alignment operation is performed. The entire set of samples uses the source-driven video frame number and timestamp as a common temporal reference. The video uses a standard frame rate of 25fps or 30fps, and the audio is segmented into frames with a 10ms frame shift. The positions of all face bounding boxes, image scaling, head pose angles, and the temporal semantics corresponding to phonemes and view positions are uniformly calibrated to ensure that anchor samples, the two types of positive samples, and degraded negative samples remain completely consistent in spatial and temporal dimensions. This provides a basis for fair comparison among the samples and a reliable data prerequisite for subsequent distance calculation and loss function operation.

[0063] A quality ranking loss is constructed based on the quality differences between samples, and a consistency constraint loss is constructed by combining mouth key points, teeth, and temporal dimensions. Based on the differences in visual quality, detail performance, and temporal stability among various types of samples, a complete loss function system is built. The entire system is divided into two major parts: quality ranking loss and multi-dimensional consistency constraint loss. The two types of losses are calculated together to provide supervision signals for model parameter iteration.

[0064] First, a quality ranking loss is constructed. The constraint logic is determined by combining the actual quality levels of anchor samples, two types of positive samples, and perturbation negative samples. The sample quality, from high to low, is: stable reference positive samples, locally redrawn positive samples, anchor samples, and perturbation negative samples. This loss introduces an interval threshold parameter, with a fixed value of 0.15. The calculation logic ensures that the model outputs quality scores for high-quality samples that are significantly higher than those for ordinary defective samples, while the scores for ordinary defective samples are higher than those for degraded perturbation samples. The corresponding calculation rule is to first calculate the score difference between positive samples and anchor samples, and the score difference between anchor samples and negative samples, and then combine this with the preset interval threshold for constraint judgment. When the score difference between samples is less than the threshold, a loss value is generated, thereby guiding the model to actively distinguish the generation effects of different quality levels and avoid low-quality output. When the image quality difference between anchor samples and perturbation negative samples is too small to achieve effective differentiation, the constraint conditions can be simplified to only require that the scores of all positive samples are higher than those of the anchor samples, ensuring that the ranking logic works normally.

[0065] Secondly, a multi-dimensional consistency constraint loss is constructed, corresponding to the three core detection directions: lip keypoints, tooth regions, and inter-frame temporal sequences. Each constraint adopts the judgment criteria and numerical thresholds of the previous frame-level detection. For the lip keypoint consistency constraint, the Euclidean distance between the model output keypoint and the standard coordinates is calculated using the 20-point coordinate reference of the lip. The normal allowable upper limit of this index is 0.18. Once it exceeds this limit, it is judged as lip jitter or point position offset, and the loss function will amplify the deviation value to drive the model to correct the mouth movement trajectory and morphological distortion. For the tooth region consistency constraint, the mean difference in brightness of the tooth mask region, the texture gradient variance, and the contour Hausdorff distance are integrated for comprehensive calculation. The comprehensive index judgment threshold is 0.25. When the calculation result exceeds the threshold, the loss term simultaneously exerts a constraint effect to correct problems such as tooth flickering, misalignment, and edge collapse. To address the inter-frame temporal consistency constraint, the difference in mouth opening and closing degree between adjacent frames is calculated frame by frame, and the maximum change in mouth opening and closing degree in a single frame is limited to 0.25. At the same time, the overall variance of the three-frame window is used for auxiliary judgment to suppress sudden changes in action and screen jumps in consecutive frames, thus ensuring the smoothness of video playback.

[0066] All loss terms are weighted and fused according to fixed weights: quality ranking loss is weighted at 0.7, lip keypoint constraint loss at 1.0, tooth region constraint loss at 1.2, and temporal consistency constraint loss at 0.8. The weighted total loss serves as the core supervision quantity for model iteration. The loss calculation module directly interfaces with the previously constructed sample group, sequentially reading image data, keypoint parameters, quality scores, and temporal information from single-frame and segment-level samples to complete full-dimensional loss calculations. The calculated loss results are unidirectionally transmitted to the model trainable modules, namely the mouth action branch, temporal smoothing adapter, and quality prediction head, to complete backpropagation and parameter updates in conjunction with the AdamW optimizer. The entire calculation chain is seamlessly integrated with the sample grouping, local model fine-tuning, and closed-loop incremental training process, continuously correcting various existing defects in the model based on quantized loss, and gradually improving the overall effect of facial action transfer.

[0067] By integrating the sample construction process, sample pairing rules, and loss calculation system, a contrastive learning paradigm oriented towards incremental model optimization is formed. This involves integrating sample classification standards, structured sample unit usage specifications, single-frame and segment-level sample generation processes, multi-dimensional sample alignment constraints, and full-category loss calculation logic to build a standardized contrastive learning execution system. All execution rules are matched to the overall business process of face motion transfer material production and the model incremental optimization architecture.

[0068] At the sample usage level, all training data is based on previously constructed structured sample units. Each unit uniformly stores information such as audio segments, phonemes, view positions, reference keypoints, transitional frames, mouth masks, anomaly labels, defect severity, and comprehensive scores. Defect severity is divided into four intervals: 0.2~0.4, 0.4~0.6, 0.6~0.8, and 0.8~1.0. The comprehensive score follows the established calculation method, with a passing threshold of 0.65. Unit data can be directly extracted to construct single-frame sample groups and segment-level sample groups. Single-frame samples are paired based on independent anomalous frames. Segment-level samples are generated using a 5-frame temporal window, a 2-frame sliding step, and a 60% overlap rate, focusing on covering continuous anomalous intervals while considering both single-frame defect correction and short-term temporal transition pattern learning.

[0069] The sample alignment process follows a unified standard, using the source-driven video frame number and timestamp as a common timing reference. The video uses a standard frame rate of 25fps or 30fps, and the audio is framed with a 10ms frame shift. Anchor samples, two types of positive samples, and perturbation negative samples within the same group must be completely consistent in face bounding box coordinates, image scaling, head pitch / yaw / roll angles, phonemes, and viewpoint labels. The head posture deviation is allowed to be no more than 10°, and the mouth opening difference is controlled within 0.2, ensuring that different samples have a basis for lateral comparison and avoiding the loss of calculation results due to spatial and temporal differences.

[0070] The loss calculation is performed using a standardized formula and weighting. The system includes quality ranking loss, lip keypoint constraint loss, tooth region constraint loss, and inter-frame temporal constraint loss. The quality ranking loss has an interval threshold of 0.15. The core logic is to ensure that the model output scores of stable reference positive samples and locally redrawn positive samples are successively higher than those of anchor samples, and that the anchor sample score is higher than that of perturbed negative samples. If the quality difference between the two types of samples is too small, the rule is simplified, requiring only that the positive sample score be higher than the anchor sample. The lip keypoint constraint is calculated based on the Euclidean distance of 20-point coordinates, with an anomaly threshold of 0.18. The tooth region constraint is calculated by fusing brightness difference, texture variance, and contour distance, with an anomaly threshold of 0.25. The inter-frame temporal constraint limits the maximum change in mouth opening between adjacent frames to 0.25. The weights of each loss are set as follows: quality ranking loss 0.7, lip keypoint loss 1.0, tooth region loss 1.2, and temporal loss 0.8. All losses are weighted and summed, serving as the core supervision for model parameter updates.

[0071] The entire comparative learning paradigm is deeply embedded in the closed loop of face transfer material production. The upstream data interface directly connects to a low-quality sample pool, automatically collecting various defective frames and abnormal fragments filtered out in the detection and repair stages, and completing sample extraction, grouping, and alignment processing. The downstream link directly connects to the model incremental optimization module, and the grouped sample groups and calculated loss values ​​are input into the training process according to a predetermined timing and calling rules. The system clearly defines the entire data flow path: from defective samples to collection, structured encapsulation, sample grouping, loss calculation, and signal output, the triggering timing and data interaction form of each stage are standardized.

[0072] By leveraging the comparison of quality differences between samples and multi-dimensional loss constraints, self-supervised optimization of the model is achieved. During training, the global identity encoding branch and global feature layer parameters of the base model are kept frozen, and parameter iteration is performed only on the mouth feature branch, lip keypoint regression head, tooth consistency branch, temporal smoothing adapter, and quality prediction head. The entire paradigm requires no additional manually labeled data, and only uses defective samples generated in the business process to complete model training, continuously correcting problems such as mouth tremors, tooth flickering, inconsistent lip shapes, and inter-frame jumps. Ultimately, it forms a complete comparative learning operation scheme that can be stably deployed and adapted to the iterative upgrade of face transfer models, supporting the continuous operation of the material production closed loop.

[0073] S104 employs a local adapter fine-tuning mechanism, freezes the model identity encoding backbone, and only updates the parameters of the mouth action branch, temporal smoothing adapter, and quality prediction head with small steps, while fusing keypoint supervision loss, temporal continuity loss, regional consistency loss, and feature preservation loss.

[0074] In one implementation, the face transfer base model selected in this study possesses the core capability of decoupling identity features from facial expressions and movements. It can independently analyze the inherent attributes of a person's face and dynamic lip movements, adapting to the overall modular training scheme. The entire model is divided into two main levels based on its functional purpose and parameter update permissions. The modules are connected in a serial manner, with data transmitted unidirectionally from front to back. The connection logic between levels and modules remains fixed.

[0075] The front-end layer is the identity encoding backbone, which integrates a face backbone encoder, an identity feature branch, and a global facial style branch. The face backbone encoder is responsible for compressing and extracting visual features from the complete face image. The identity feature branch identifies the unique identity of the individual based on feature vector comparison logic. The calculation method is identity consistency index = 1 - cosine distance of the face feature vector. The threshold for this index is set to 0.15; exceeding this threshold indicates an identity shift. The global facial style branch specifically stores fixed visual information such as the target individual's skin color, facial contours, and facial feature proportions. Throughout the entire model training and inference cycle, all network weights and bias parameters within the identity encoding backbone are locked and do not participate in gradient calculations or backpropagation operations. This fundamentally safeguards the target individual's facial identity and basic visual features, preventing identity drift and facial style tampering during training.

[0076] The backend layer consists of local functional units, which sequentially include a mouth motion branch, a temporal smoothing adapter, and a quality prediction head. These three types of units are the core areas of this round of parameter iteration. The mouth motion branch specifically handles dynamic motion commands and is responsible for generating lip shapes and opening / closing effects that conform to the driving logic. The temporal smoothing adapter mainly manages motion changes between consecutive video frames, limiting the maximum change in mouth opening / closing in a single frame to 0.25, thereby suppressing inter-frame jumps and image jitter. The quality prediction head uses a predetermined scoring rule to calculate the comprehensive score for each frame. The calculation formula is: Comprehensive Score = 1.0 - (0.25 × Lip Temporal Index + 0.15 × Mouth Shape Consistency Index + 0.25 × Tooth Region Index + 0.15 × Identity Index + 0.20 × Geometric Index). The passing score for the comprehensive score is set at 0.65, thus completing the single-frame image quality judgment. This group of local units has open parameter update permissions and is specifically designed to optimize three types of issues: mouth shape, inter-frame smoothness, and image quality.

[0077] The model employs an adapter-based local fine-tuning scheme, implementing a small-step parameter update strategy. During training, the AdamW optimizer is used consistently, with a base learning rate of 2e-4 and a weight decay coefficient of 1e-5. Gradient clipping is enabled, limiting the maximum norm of the gradient to 1.0 to constrain parameter update magnitude and prevent parameter oscillations caused by gradient anomalies. Training batch sizes are set to 4 or 8, with each iteration lasting 5 to 10 epochs, maintaining a short-cycle iteration pattern throughout. This strategy only applies to the backend mouth action branch, the temporal smoothing adapter, and the quality prediction head; the frontend identity encoding backbone remains parameter-frozen and unmodified during iterative training, thus optimizing model defects while fully preserving the original feature extraction capabilities of the base model.

[0078] The training process employs four types of collaborative supervised losses, forming a complete loss constraint system. The first type is keypoint supervised loss, which uses the standard 20-point lip coordinates from the source video as a reference to calculate the Euclidean distance between the model's output keypoints and the standard coordinates. The normal upper limit for this metric is 0.18, and the loss value increases with coordinate deviation, guiding the model to correct issues such as lip point offset and mouth shape distortion. The second type is temporal continuity loss, which calculates the mouth opening / closing difference between adjacent frames, combining the variance of the three-frame window to comprehensively judge the inter-frame state, adhering to the constraint that the single-frame opening / closing change does not exceed 0.25, eliminating abrupt changes in motion and image jitter in consecutive frames. The third type is region consistency loss, calculated for the tooth mask region, fusing the mean difference in brightness, texture gradient variance, and contour Hausdorff distance to obtain a comprehensive value, with a judgment threshold of 0.25, used to constrain details such as tooth flickering, misalignment, and edge distortion. The fourth type is feature preservation loss. It reads the global features output from the identity encoding backbone, calculates the matching degree between the locally generated region and the global features, and generates a loss when the matching result is lower than the standard value, ensuring that the mouth region and the overall facial features are consistent. The four types of losses are calculated simultaneously and serve as a supervision basis for model parameter updates.

[0079] The entire data and error flow chain has a clear set of rules. The model input data includes static features of the target person's mouth, key points of the reference face in the source video, frame-by-frame transitions, phoneme labels, and viewpoint labels. All input data is first fed into the frozen identity encoding backbone to complete global identity and visual feature extraction. The extracted feature data is then passed to the mouth action branch. The mouth action branch combines constraint data to generate basic lip-sync images, and the output is fed into the temporal smoothing adapter for inter-frame optimization. The optimized images are finally fed into the quality prediction head for quality scoring. Four types of loss functions calculate the deviation value corresponding to the output of each unit. The generated error signal is only propagated back to the mouth action branch, temporal smoothing adapter, and quality prediction head for backpropagation and parameter updates. The front-end identity encoding backbone does not receive error signals and does not modify any parameters. This chain is deeply integrated with the entire material production process and is specifically designed to serve the iterative correction of defective samples.

[0080] A standardized local adapter fine-tuning execution process is formed by integrating the model's hierarchical structure, module connection methods, parameter freezing and update rules, training hyperparameters, loss calculation logic, and data flow rules. This process is embedded in the overall closed loop of transfer generation, quality inspection, repair, and sample re-entry. Upstream, it receives structured training sample sets and outputs updated local module weights after multiple iterations. In the model storage stage, only the weight files of the mouth action branch, temporal smoothing adapter, and quality prediction head are saved, while the original parameters of the identity encoding backbone remain unchanged. During actual deployment and inference, the frozen identity encoding backbone and the fine-tuned local units are run together. This not only fully preserves the facial identity and inherent style of the target person but also optimizes mouth actions, inter-frame timing, and image quality by relying on the iterative local capabilities, ultimately outputting a transfer video that meets the requirements for using digital human materials.

[0081] S105, based on a mini-batch incremental learning strategy, uses high-frequency error samples for iterative optimization and continuously corrects the defects of temporal jitter, detail collapse and lip-sync inconsistency in the transfer model through closed-loop self-supervised learning.

[0082] In one implementation, combining the core business objectives of repairing defects and iterating performance in the face motion transfer model, a small-batch incremental learning mode is adopted overall, along with local parameter update control requirements and a closed-loop iteration logic across the entire chain, to build a standardized model training execution process. This process is deeply integrated into the complete production system of digital human transfer materials, enabling data exchange and motion linkage between upstream and downstream modules, with the core operating direction of continuously optimizing model performance and improving the quality of finished materials.

[0083] The upstream process directly connects to the system's built-in low-quality sample pool. This pool uniformly collects defective frames and abnormal segments that have been deemed unqualified after frame-level inspection, automatic repair, and quality acceptance. All content entering the database is pre-packaged into structured sample units, each containing information such as audio segments, phoneme tags, viewpoint numbers, 68 full-face key points, 20 lip key points, mouth mask, head pose, anomaly type, defect severity, and single-frame comprehensive score. The defect severity is divided into four ranges: 0.2 to 0.4 for minor anomalies, 0.4 to 0.6 for moderate anomalies, 0.6 to 0.8 for severe anomalies, and 0.8 to 1.0 for fatal anomalies. The passing score for the comprehensive score is fixed at 0.65. The system extracts high-frequency defect samples from the sample pool according to screening rules to form the training batch for this round, prioritizing fatal and severe anomaly samples. Simultaneously, corresponding training weights are assigned based on the sample severity; the higher the defect level, the larger the weight value, enhancing the model's ability to learn and correct high-frequency, high-risk problems.

[0084] The training phase employs a hierarchical model update rule, utilizing a base model architecture that decouples identity and action, dividing the network into global and local modules. The global module includes the face backbone encoder, identity feature branch, and global facial style branch. All parameters in this part are locked throughout the training process and do not participate in gradient calculation or backpropagation. The identity stability index is calculated as: Identity Consistency Index = 1 - Cosine Distance of Face Feature Vector, with a threshold of 0.15. Freezing the global module ensures the preservation of the target person's facial identity, skin color, and facial proportions, fundamentally preventing identity drift during training. Only the mouth action branch, temporal smoothing adapter, and quality prediction head are open as trainable modules, and all parameter updates are limited to these ranges, employing a small-step iterative training approach. The AdamW optimizer was used for training, with a base learning rate of 2e-4, a batch size of 4 or 8, a weight decay coefficient of 1e-5, and gradient clipping enabled to limit the maximum norm of the gradient to 1.0. Each training iteration consisted of 5 to 10 training epochs. Lightweight updates were used to prevent overfitting, parameter oscillations, and other problems in the model.

[0085] The training process employs a complete multi-dimensional loss constraint system, including keypoint supervision loss, lip contour loss, tooth region loss, identity preservation loss, temporal continuity loss, mouth masking loss, and quality ranking loss. Each loss is assigned a fixed weight of 1.0, 1.0, 1.2, 0.5, 0.8, 0.5, and 0.7, respectively. The threshold for the temporal stability index of lip keypoints is 0.18, the threshold for the consistency index of tooth regions is 0.25, the maximum limit for the change in mouth opening and closing between frames is 0.25, and the temporal continuity loss constrains the amplitude of actions in adjacent frames. After the loss calculation is completed, an error signal is generated. This signal is only fed back to the locally trainable modules to update parameters; the global module does not receive any error data.

[0086] After a single round of incremental training, the process automatically flows to downstream business modules, restarting the complete material production chain. First, the updated model is called to perform facial motion transfer inference, generating entirely new video frames of the target person. Then, frame-level consistency detection is performed, based on five dimensions: lip keypoint temporal sequence, homophone co-viewpoint, tooth region, facial identity, and reference geometry. The result is calculated using the formula: Overall Score = 1.0. The quality score for each frame is calculated using (0.25 × lip temporal index + 0.15 × mouth shape consistency index + 0.25 × tooth region index + 0.15 × identity index + 0.20 × geometric index), with 0.65 as the passing standard. After detection, frames are categorized into repairable frames, risky segments, and unrepairable segments based on anomaly priority and segment division rules. For content meeting the criteria, two automatic repair strategies are applied: local interpolation and historical frame rollback. The repaired content then re-enters the quality acceptance module, with acceptance criteria including a frame-level pass rate ≥99%, audio-visual matching degree ≥0.90, and an automatic repair frame percentage ≤5%.

[0087] After acceptance testing, data is distributed. Normal footage that meets all criteria is directly stored in the footage library. Frames and segments that still fail after repair or are deemed unrepairable are reassigned to a low-quality sample pool as the data source for the next round of training. This entire process forms a closed-loop iterative model, continuously collecting new defective samples and repeatedly conducting small-batch incremental training. With multiple rounds of iteration, the model's ability to correct issues such as timing jitter, tooth flickering, inconsistent lip movements, and local morphological distortions gradually improves, resulting in continuous improvement in the overall frame-level pass rate, timing smoothness, and detail consistency of the footage.

[0088] The entire training process is deeply integrated with every stage of material production. The timing of each stage, including sample collection, model training, inference generation, quality inspection and repair, and acceptance and distribution, is fixed and the data interaction rules are clear. No additional manual intervention is required. The model is continuously iterated and optimized through an automated closed loop, which fully supports the large-scale and high-quality production of 2D digital human migration materials.

[0089] The system extracts high-frequency error samples according to sample selection rules to form training batches. Following a model hierarchical strategy, the identity encoding branch and global feature layer are frozen, and only the mouth branch, temporal adapter, and quality prediction head are used for parameter updates. The system constructs training batches based on preset sample selection rules, limiting the selection to a low-quality sample pool. This pool contains video frames and continuous abnormal segments deemed unqualified after frame-level consistency detection, automatic repair, and quality acceptance. All samples are pre-structured and encapsulated, with each sample unit recording frame number, anomaly type, defect severity, overall consistency score, reference keypoints, and phoneme viewpoints. The system statistically analyzes the frequency of occurrence of all samples, prioritizing the extraction of high-frequency defect samples such as temporal jitter, broken teeth, inconsistent lip movements, and identity drift to form the current training batch. Simultaneously, training weights are assigned based on defect severity, with higher weights given to fatal anomalies (0.8-1.0) and severe anomalies (0.6-0.8) to enhance the model's learning ability for high-incidence, high-risk issues. The batch size is adapted to the training hardware, with 4 or 8 samples per batch to match subsequent training batch configurations. During training, the system employs a layered architecture based on the decoupling of identity and action, dividing the entire network into two main structures: a global layer and a local layer. These two layers maintain a sequential connection, with data flowing unidirectionally from the global layer to the local layer. Layer functions and parameter permissions are clearly defined.

[0090] The global layer consists of an identity encoding branch, a face backbone encoder, and a global facial feature layer. The face backbone encoder is responsible for extracting global visual features from the input face image and video frames, compressing and representing image information. The identity encoding branch relies on the face feature vector to complete identity representation, and uses the formula identity consistency index = 1 - cosine distance of face feature vector to determine the identity matching status. The threshold for this index is 0.15; when the calculated result exceeds the threshold, it indicates identity drift. The global facial feature layer uniformly stores the inherent visual features of the target person, such as facial contours, skin color, and facial proportions. During the forward inference and backpropagation stages of the entire incremental training, all network weights and bias parameters in the global layer are frozen and do not participate in gradient calculation or parameter updates. This locks the target person's facial identity and overall visual features at the architectural level, completely avoiding the problems of identity tampering and facial style changes during training.

[0091] The local layer comprises the mouth branch, a temporal smoothing adapter, and a quality prediction head. This layer is the only trainable region for this round of parameter iteration. The mouth branch is responsible for parsing dynamic keypoints and phoneme position information from the source video, combining this with static prior knowledge of the target person's mouth to generate the corresponding lip shape. The output must meet the constraints that parameters such as mouth width and lip thickness fluctuate by no more than ±10% relative to the baseline value, and head pose deviation is less than 10°. The temporal smoothing adapter specifically manages motion changes in consecutive video frames, limiting the maximum change in mouth opening and closing between adjacent frames to 0.25, to suppress issues such as frame jumps and image jitter. The quality prediction head calculates the overall score for each frame according to a fixed formula: Overall Score = 1.0 (0.25×lip timing index + 0.15×mouth shape consistency index + 0.25×teeth region index + 0.15×identity index + 0.20×geometric index), with a comprehensive score passing threshold of 0.65, to complete the quantitative evaluation of image quality.

[0092] The entire training process limits the parameter update range, with all gradient signals only propagated back to the three types of modules at the local level. Targeted optimization is performed solely on mouth motion generation effects, inter-frame temporal stability, and image quality assessment capabilities. The AdamW optimizer is used for training, with a base learning rate of 2e-4 and a weight decay coefficient of 1e-5. Gradient clipping is enabled, limiting the maximum norm of the gradient to 1.0. A short-cycle iteration pattern of 5 to 10 rounds is employed, using small steps to fine-tune local parameters, optimizing model defects while ensuring the original capabilities of the base model remain intact.

[0093] The system calculates losses based on a pre-defined multi-dimensional loss function, and strengthens constraints on high-risk samples by incorporating sample weights to guide network parameter iteration. During model iteration training, the system calls pre-configured loss functions for unified calculation. The entire loss system consists of keypoint supervision loss, temporal continuity loss, region consistency loss, and feature preservation loss. These four types of losses correspond to different optimization directions, specifically constraining the model's output performance. Keypoint supervision loss is mainly used to correct mouth shape. During calculation, the standard 20-point coordinates of the lips in the source video are used as a reference to calculate the Euclidean distance between the model's output lip keypoints and the standard keypoints. The normal upper limit for this metric is set to 0.18. When the calculated result exceeds this value, it indicates that the lip points have shifted and the mouth shape has become distorted. The loss value will increase accordingly to guide the model to correct the mouth contour and motion trajectory, ensuring that the mouth shape conforms to the requirements of the driving action.

[0094] The temporal continuity loss specifically manages the effects of inter-frame changes, primarily constraining the mouth movement state between consecutive video frames. During computation, the difference in mouth opening and closing is calculated frame by frame, and the overall variance of a 3-frame sliding window is used to comprehensively determine the magnitude of the change. It is explicitly stipulated that the maximum change in mouth opening and closing in a single frame must not exceed 0.25. If there are abrupt changes in action or opening / closing amplitude between frames, this loss will be increased accordingly to suppress issues such as image jitter and abrupt movements, ensuring the smoothness of the entire video playback. The region consistency loss focuses on the local details of teeth, lips, etc. During computation, a dedicated detection region for teeth is first defined using a mask. The mean difference in pixel brightness, texture gradient variance, and contour Hausdorff distance of the region are calculated sequentially, and the integrated results are used as the evaluation criteria. The anomaly threshold for this indicator is set to 0.25. When issues such as tooth flickering, misalignment, or edge distortion occur, this loss takes effect, forcing the model to maintain the stability of local details and avoiding detail defects.

[0095] Feature preservation loss is used to protect the overall features of the person. It is calculated based on the identity feature vector, and the calculation rule is: Identity consistency index = 1 - cosine distance of face feature vector, with a corresponding judgment threshold of 0.15. This loss is connected to the global features output by the identity encoding backbone, and the feature matching degree between the mouth generation region and the entire face is compared. If identity drift or facial style change occurs, the loss value increases accordingly to ensure that the mouth generation content remains consistent with the inherent facial features of the target person.

[0096] After calculating the four basic losses individually, the system differentiates the training weights for each training sample based on its accompanying defect severity score. Defect severity is divided into four intervals: 0.2 to 0.4 for minor anomalies, 0.4 to 0.6 for moderate anomalies, 0.6 to 0.8 for severe anomalies, and 0.8 to 1.0 for fatal anomalies. For severe and fatal anomalies such as identity drift and broken teeth, higher weight coefficients are assigned. For example, severe anomaly samples are weighted at 1.2, and fatal anomaly samples at 1.5; minor and moderate anomaly samples are weighted at 0.8 and 1.0 respectively. This weighting prioritizes learning methods to correct high-risk defects, strengthening the model's ability to constrain high-frequency, severe problems.

[0097] All losses are weighted and fused according to fixed weights, with the keypoint supervision loss having a weight of 1.0, the temporal continuity loss having a weight of 0.8, the region consistency loss having a weight of 1.2, and the feature preservation loss having a weight of 0.5. The final fused loss value serves as the core supervision signal for network parameter iteration. This error signal is only propagated backward to three types of trainable modules: the mouth branch, the temporal smoothing adapter, and the quality prediction head. This drives the network to adjust its internal parameters layer by layer, gradually reducing the deviation between the model output and the standard effect, and continuously guiding the network to generate face transfer images that are morphologically standard, temporally smooth, detailed, and identity consistent.

[0098] A short-cycle, multi-round training model is employed, with each round restarting the entire training process, including transfer learning, frame detection, automatic repair, and quality acceptance. The entire training workflow uses this short-cycle, multi-round iterative model, configuring all training parameters based on hardware computing power and model characteristics to control parameter update magnitude from the source and avoid overfitting and parameter oscillations. The AdamW optimizer is used for parameter updates during training, with a base learning rate set to [value missing]. This value balances the optimization efficiency and stability of local modules; the batch size for a single training run is set to 4 or 8, employing a small batch strategy to reduce the impact of a single set of data on the parameters; simultaneously, a gradient pruning mechanism is enabled, limiting the maximum norm of the gradient to 1.0 to constrain the gradient fluctuation range and prevent drastic parameter changes. A single complete iteration consists of 5 to 10 training epochs. In each epoch, only the mouth branch, temporal smoothing adapter, and quality prediction head undergo lightweight parameter adjustments. The global-level identity encoding branch, face backbone encoder, and facial global style branch maintain parameter freeze throughout the process and do not participate in gradient calculation or backpropagation.

[0099] After a single training round is completed, the overall process is not terminated. Instead, the entire face motion transfer workflow is automatically restarted to complete model performance testing and new defect data collection. First, the updated model is called, and data such as the target person's mouth static features, source video key points, and phoneme viewpoint labels are input to generate new face transfer video frames in batches. The generated results are simultaneously fed into a frame-level consistency detection stage, using five detection dimensions for quantitative verification: lip keypoint temporal stability threshold 0.18, same phoneme same viewpoint mouth shape consistency threshold 0.22, tooth region consistency threshold 0.25, facial identity stability threshold 0.15, and reference keypoint geometric consistency threshold 0.20. The comprehensive score is calculated as: Comprehensive Score = 1.0 (0.25×lip timing index + 0.15×mouth shape consistency index + 0.25×teeth region index + 0.15×identity index + 0.20×geometric index) Using 0.65 as the passing score, and combining anomaly priority and frame continuity rules, repairable frames, risky segments, and unrepairable segments are divided.

[0100] For detected abnormal content, the system initiates an automatic repair process, employing two repair methods: piecewise linear interpolation and historical frame rollback. The repair interval is controlled within 5 consecutive frames. After repair, linear weighted fusion is performed on the first and last frames of the interval to eliminate transition artifacts. Repaired footage enters the quality acceptance phase, with acceptance criteria including frame-level pass rate ≥99%, audio-visual matching degree ≥0.90, automatic repair frame ratio ≤5%, and mouth parameter deviation not exceeding ±10% of the baseline value, among other quantitative indicators. Footage that meets the acceptance criteria is directly stored in the footage library; defective frames and abnormal segments that fail acceptance are uniformly collected into a low-quality sample pool according to rules, with frame number, abnormality type, defect severity, comprehensive score, reference key points, audio-visual information, etc., simultaneously recorded, completing a new round of defect data collection. The newly collected defect data is merged with historical samples, and the frequency of various problems is statistically analyzed. High-frequency defect samples are selected as core data for the next round of training, providing material support for subsequent multiple rounds of incremental iterations, allowing the model to gradually optimize performance in a closed loop of continuous verification and correction.

[0101] The system continuously collects new defect samples and iteratively conducts incremental training to gradually improve defects such as temporal jitter, tooth breakage, and inconsistent lip shapes, achieving closed-loop self-supervised optimization. After each round of incremental training, model parameter updates, and completion of facial motion transfer inference, the system performs full-dimensional frame-level consistency detection on newly generated video frames, identifying various defective content based on established rules. The detection uses five evaluation dimensions, with each indicator normalized to the 0-1 range: lip keypoint temporal stability threshold 0.18, homophone co-viewpoint lip shape consistency threshold 0.22, tooth region consistency threshold 0.25, facial identity stability threshold 0.15, and reference keypoint geometric consistency threshold 0.20. The comprehensive score is calculated using the following formula: ,in, The overall consistency score for frame t; For key time-series indicators of the lips; As an indicator of lip-shape consistency in the same visual position; As an indicator of tooth region consistency; For identity consistency indicators; This is the geometric consistency index for key points. A score of 0.65 is used as the passing mark; frames with a comprehensive score below this value are uniformly judged as defective frames or abnormal segments.

[0102] For identified defective content, the system classifies it according to frame continuity: isolated single frames are repairable frames, two or more consecutive frames are risky segments, and more than five consecutive frames are classified as unrepairable segments. All defective content is no longer discarded directly, but rather its information is entered according to the structured sample unit specification. Each unit fully records information such as frame number, audio segment, phoneme and viewpoint labels, reference keypoints, moving images, mouth mask, head pose, anomaly type, and defect severity. Defect severity is divided into four levels: 0.2–0.4 for minor anomalies, 0.4–0.6 for moderate anomalies, 0.6–0.8 for severe anomalies, and 0.8–1.0 for fatal anomalies. All newly added defective samples are uniformly collected and continuously added to the overall training sample set, achieving dynamic expansion of the sample library.

[0103] After the sample set is updated, the system starts a new round of incremental training, still following the execution rule of small steps in multiple rounds and local updates. The AdamW optimizer is used for training, and the base learning rate is set to... Weight decay coefficient The maximum norm of the gradient is limited to 1.0, the batch size is 4 or 8, and each iteration consists of 5 to 10 training epochs. During training, the global hierarchy, comprising the face backbone encoder, identity feature branch, and global facial style, is frozen; only the mouth branch, temporal smoothing adapter, and quality prediction head are updated. The loss system used for training includes keypoint supervision loss, temporal continuity loss, region consistency loss, and feature preservation loss. Training weights are assigned based on the severity of sample defects: fatal aberrations have a weight of 1.5, severe aberrations 1.2, moderate aberrations 1.0, and minor aberrations 0.8, enhancing the model's learning ability for high-risk defects.

[0104] Through multiple rounds of iterative iteration, the model gradually corrects various issues through repeated training, inference, detection, and sample feedback. To address temporal jitter, the model uses temporal continuity loss to constrain mouth opening and closing variations between adjacent frames, keeping the single-frame opening and closing difference within 0.25, and combines this with 3-frame window variance to suppress abrupt changes in action between frames. For issues like tooth flickering, misalignment, and distortion, the model relies on tooth region consistency loss, using a threshold of 0.25 to control the brightness, texture, and contour features of the tooth region. To address inconsistencies in lip movements for homophones and co-viewpoints, the model uses lip movement consistency indicators to ensure that the deviation between the image parameters for the same pronunciation and lip movement category and the historical stable sample mean does not exceed 0.22, gradually unifying lip movement performance.

[0105] The entire process uses defect sample feedback as a link and lightweight model iteration as the core to build a complete closed-loop self-supervised operation system. While old defects are gradually fixed, newly generated problems are promptly transformed into training materials, driving continuous model optimization. After multiple rounds of iteration, the model's ability to suppress various typical defects steadily improves, and mouth movements, tooth details, inter-frame smoothness, and consistency of human identity are comprehensively improved. Ultimately, the model's defect repair and comprehensive capability iteration are completed, and the optimized model can stably produce face transfer videos that meet the various quantitative standards of digital human materials.

[0106] S106 improves the temporal robustness and detail generation consistency of neural networks in face motion transfer through multiple rounds of model iteration and quality verification, resulting in an optimized motion transfer model for digital human materials.

[0107] In one implementation, relying on the established incremental training, sample feedback, and local model update workflow, the system continuously performs multiple rounds of iterative work. Within each training cycle, a lightweight parameter update method is used to optimize the model. Each iteration consists of 5 to 10 training rounds, combined with the AdamW optimizer. Base learning rate, The weight decay coefficient is set, and gradient clipping is enabled with the maximum norm of the gradient limited to 1.0. Small batch configurations of 4 or 8 groups are used. Throughout the process, parameter adjustments are only performed on the mouth motion branch, the temporal smoothing adapter, and the quality prediction head, while the parameters of global modules such as the identity encoding backbone, the face backbone encoder, and the global facial style branch are kept frozen. After a single round of lightweight training, the system immediately initiates full-dimensional frame-level quality verification.

[0108] The verification process employs an established frame-level consistency detection system, uniformly using the source-driven video frame sequence number as the global temporal benchmark. The video uses a standard frame rate of 25fps or 30fps, and the audio is segmented into frames with a 10ms frame shift. Multiple data types, including the face migration output frame, audio features, phoneme labels, viewpoint numbers, source video reference key points, and static prior information about the target person's mouth, are bound frame-by-frame to ensure complete temporal synchronization of all verification data. The verification covers five core detection dimensions, with each indicator normalized to the 0-1 range and fitted with fixed anomaly thresholds: 0.18 for lip key point temporal stability, 0.22 for lip shape consistency with the same phoneme and viewpoint, 0.25 for tooth region consistency, 0.15 for facial identity stability, and 0.20 for reference key point geometric consistency. Facial identity stability is calculated using the formula: Identity Consistency Index = 1 - Cosine Distance of Facial Feature Vector. Geometric consistency is referenced by using weights of 0.3, 0.3, 0.2, and 0.2 to fuse four types of deviation values: corner of mouth, lip distance, mouth proportion, and head posture.

[0109] The overall score for a single frame is calculated using the following formula: Overall Score = 1.0 - (0.25) Lip time sequence index +0.15 Mouth shape consistency index +0.25 Tooth region index +0.15 +0.20 (Geometric indicators) are used, with a global passing score of 0.65. Combining the scores, anomaly priority, and frame continuity rules, normal frames, repairable frames, risky segments, and unrepairable segments are distinguished. Simultaneously, quantitative data such as frame-level pass rate, audio-visual matching degree, mouth shape compliance rate, and the proportion of automatically repaired frames are statistically analyzed for the entire footage. This data serves two purposes: firstly, to evaluate the optimization effect of the current round of the model, and secondly, to determine whether the iteration process should continue or terminate.

[0110] The system sets clear iteration termination criteria. The hard indicators for material inclusion are: frame-level pass rate ≥99%, audio-visual matching degree ≥0.90, and automatic repair frame ratio ≤5%. Simultaneously, parameters such as mouth width and lip thickness must deviate from the static prior by no more than ±10%. After multiple iterations, if all indicators meet the inclusion criteria and the improvement in indicators for consecutive iterations is less than a preset threshold, the model optimization is considered complete, and the incremental iteration process terminates. If the verification results show that the material still has defects such as timing jitter, tooth flickering misalignment, inconsistent lip shapes for the same phoneme, or identity drift, then the abnormal frames and segments detected in this round are automatically collected, and information is entered according to the structured sample unit specification. Each unit completely records the frame number, audio, phoneme visual position, key points, abnormality type, and defect severity in the 0.2~1.0 range. New high-frequency defect samples are added to the training sample set, and the small-batch incremental training, multi-loss constraint calculation, and local parameter update process is restarted. Throughout the entire process, the principle of small steps and multiple rounds of training was adhered to, and the identity coding backbone was frozen throughout the process to prevent problems such as damage to identity features and overfitting of the model from occurring during training.

[0111] During multiple rounds of iteration and verification, targeted optimization work was carried out in two main directions. The first part focused on enhancing the model's temporal robustness. This involved using temporal continuity loss to constrain inter-frame variations, ensuring the maximum change in mouth opening / closing between adjacent frames did not exceed 0.25, and combining this with a 3-frame sliding window variance to suppress short-term action abrupt changes. In the training phase, in addition to single-frame samples, segment-level sample groups with 5-frame windows, 2-frame strides, and 60% overlap were used to allow the model to fully learn the action transition patterns within short timeframes. In the post-inference processing stage, linear weighted fusion was performed on the first and last frames of the repair interval. The weighting coefficients transitioned linearly from 0 to 1 and then back to 0, further eliminating inter-frame gaps and jumps, ensuring smooth video playback.

[0112] The second part focuses on optimizing local details such as the mouth and teeth. Using supervised keypoint loss and referencing the 20-point standard coordinates of the lips, the Euclidean distance is controlled within 0.18 to correct mouth shape distortion and point position offset. Relying on tooth region consistency loss, constraints are applied by comprehensively considering the brightness difference of the tooth mask area, texture gradient variance, and contour Hausdorff distance, keeping the index within 0.25 to address issues such as tooth flickering, edge distortion, and misalignment. By comparing the historical stable sample mean of homophones and co-viewpoints, and setting the lip shape parameter distance threshold to 0.22, consistent image performance is ensured for the same pronunciation and lip shape category. Simultaneously, feature preservation loss is used to control the identity consistency index within 0.15, ensuring that the generated mouth area remains consistent with the overall facial features of the target person, thus fundamentally avoiding identity drift issues. Four types of losses are combined with fixed weights: key point supervision loss 1.0, temporal continuity loss 0.8, regional consistency loss 1.2, and feature preservation loss 0.5. Training weights are assigned in combination with the severity of sample defects: fatal anomalies (0.8~1.0) weight 1.5, severe anomalies (0.6~0.8) weight 1.2, general anomalies (0.4~0.6) weight 1.0, and minor anomalies (0.2~0.4) weight 0.8, to enhance the ability to correct high-risk defects.

[0113] After all iterations are completed and multiple rounds of quality checks have passed, the system separately saves the training weights of the mouth action branch, the temporal smoothing adapter, and the quality prediction head. The identity encoding backbone and the global feature layer retain their original parameters without modification. During formal deployment and inference, the frozen global module and the locally optimized module are run together. When the model is running, it sequentially receives source person video footage, target person facial reference images, temporally aligned phoneme viewpoint labels, and reference keypoint data. Relying on the decoupled network structure, it accurately replicates the mouth actions of the source person while fully preserving the inherent features of the target person's facial identity, facial proportions, skin color, etc., and outputs a temporally smooth, detailed, and consistent lip-sync face transfer video. The optimized model can be directly used in practical applications such as 2D digital human material production and virtual character driving, and the output fully meets the established database specifications and is suitable for large-scale production.

[0114] like Figure 2 As shown, a face motion transfer model optimization system based on frame-level consistency verification includes: Model architecture building module 201 is used to build a facial motion transfer neural network to decouple identity features from mouth movements and integrate static mouth shape priors and dynamic key point constraints to build a machine learning optimization architecture. The frame-level detection and scoring module 202 is used to build a consistency discrimination network. It completes quantitative scoring based on five dimensions: time sequence, lip shape, region, identity, and geometry, and outputs the anomaly type and quality ranking results. The sample and loss construction module 203 is used to build a comparative learning paradigm, divide anchor samples, positive samples, and perturbation negative samples, and construct ranking loss and consistency constraint loss. The local fine-tuning module 204 is used to enable the adapter fine-tuning scheme, freeze the identity coding backbone, update only the mouth branch, the time-series adapter, and the quality prediction head, and fuse multiple types of supervised losses to complete the parameter update. Incremental Iteration Module 205 adopts a small-batch learning approach and relies on high-frequency defect samples to carry out closed-loop self-supervised training to correct problems such as timing jitter, detail collapse, and abnormal lip movements. The iterative verification module 206 is used to conduct quality verification after multiple rounds of training, continuously improve the model's temporal robustness and detail generation capabilities, and output an optimized digital human motion transfer model.

[0115] A computing device includes a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute any face motion migration model optimization method based on frame-level consistency verification.

[0116] The methods and / or embodiments in this application can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by a processing unit, it performs the functions defined in the methods of this application.

[0117] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0118] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application.

Claims

1. A method for optimizing a face motion transfer model based on frame-level consistency verification, characterized in that, include: A face motion transfer optimization model based on neural networks is constructed to decouple mouth motion generation from identity feature representation and establish a machine learning optimization architecture that includes static morphological priors and dynamic key point constraints. Design a frame-level consistency discrimination neural network, and build a multi-dimensional quantitative scoring model based on temporal stability, lip shape consistency of the same category, regional texture consistency, identity feature consistency, and geometric constraint consistency to output anomaly classification and quality ranking results; A contrastive learning and incremental optimization paradigm is constructed, with abnormal frames as anchor samples, high-quality repaired frames and stable samples as positive samples, and perturbation and degradation samples as negative samples. Ranking loss and consistency constraint loss are constructed accordingly. A local adapter fine-tuning mechanism is adopted to freeze the model identity encoding backbone and only update the parameters of the mouth action branch, temporal smoothing adapter, and quality prediction head with small steps. The key point supervision loss, temporal continuity loss, region consistency loss and feature preservation loss are integrated. Based on the mini-batch incremental learning strategy, high-frequency error samples are used for iterative optimization. Closed-loop self-supervised learning is used to continuously correct the defects of temporal jitter, detail collapse and inconsistent lip movements in the transfer model. Through multiple rounds of model iteration and quality verification, the temporal robustness and detail generation consistency of neural networks in face motion transfer are improved, resulting in an optimized motion transfer model for digital human materials.

2. The face motion transfer model optimization method based on frame-level consistency verification according to claim 1, characterized in that, A neural network-based facial motion transfer optimization model is constructed, decoupling mouth motion generation from identity feature representation. A machine learning optimization architecture incorporating static morphological priors and dynamic keypoint constraints is established, including: Combining the identity-action decoupling design of the face action transfer model, mouth feature constraint rules, and local fine-tuning requirements, and relying on the hierarchical training mechanism of the base model, the global fixed module and the local update module are distinguished. Using the static features of the target person's mouth as a priori and the dynamic parameters of the key facial points in the source video as dual constraints, the feature extraction rules and feature fusion logic are clearly defined. By combining the identity encoding freezing strategy, the mouth branch incremental update rule, and the multi-dimensional loss constraint requirements, a neural network overall architecture is built. Summarize the model branching, feature extraction methods, constraint rules, and training control requirements to complete the construction of the face motion transfer optimization model.

3. The face motion transfer model optimization method based on frame-level consistency verification according to claim 1, characterized in that, Design a frame-level consistency discrimination neural network. Based on temporal stability, lip-sync consistency within the same category, regional texture consistency, identity feature consistency, and geometric constraint consistency, construct a multi-dimensional quantitative scoring model to output anomaly classification and quality ranking results, including: Based on the requirements for quality inspection of face migration video frames, and following the execution principles of multi-dimensional evaluation, score fusion, anomaly detection, and hierarchical labeling, a frame-level consistency discrimination network and quantitative scoring system are built. Five consistency evaluation directions are defined: lip key point sequence, homophones / co-visual positions, tooth regions, facial identity, and reference key points. Evaluation rules, normalization requirements and corresponding judgment thresholds are set for each dimension. The results of each dimension are fused according to the predetermined weights, and the comprehensive score of a single frame is calculated, with 0.65 as the passing threshold. Based on the anomaly priority and frame segmentation rules, unqualified frames are classified and labeled to distinguish between repairable frames, risky segments, and unrepairable segments, and an anomaly list and quality ranking results are output. By integrating evaluation dimensions, thresholds, weights, and grading rules, automated consistency verification, anomaly identification, and quality quantification output of video frames can be achieved.

4. The face motion transfer model optimization method based on frame-level consistency verification according to claim 1, characterized in that, A contrastive learning and incremental optimization paradigm is constructed, using anomalous frames as anchor samples, high-quality repaired frames and stable samples as positive samples, and perturbed and degraded samples as negative samples. Ranking loss and consistency constraint loss are constructed, including: Based on the iterative optimization requirements of the face transfer model, and relying on the rules for defect sample collection, sample grouping and loss construction, anchor samples, two types of positive samples and perturbation negative samples are divided according to the established standards. Based on the structured sample units, single-frame and fragment-level sample groups are generated. Migrating abnormal frames are used as anchor samples to generate local redraw positive samples and stable reference positive samples, respectively. At the same time, perturbation is applied to the original samples to obtain degraded negative samples. A quality ranking loss is constructed based on the quality differences between samples, and a consistency constraint loss is constructed by combining mouth key points, teeth, and temporal dimensions. By integrating the sample construction process, sample pairing rules, and loss calculation system, a comparative learning paradigm oriented towards incremental model optimization is formed.

5. The face motion transfer model optimization method based on frame-level consistency verification according to claim 4, characterized in that, Based on a mini-batch incremental learning strategy, iterative optimization is performed using high-frequency error samples. Closed-loop self-supervised learning continuously corrects the temporal jitter, detail degradation, and lip-sync inconsistency defects in the transfer model, including: In response to the needs of face transfer model defect repair and iterative optimization, a model training execution process is built based on small-batch incremental learning, local parameter updates and closed-loop iterative logic. High-frequency error samples were extracted according to the sample selection rules to form training batches. Following the model hierarchical strategy, the identity encoding branch and global feature layer were frozen, and only the mouth branch, time adapter, and quality prediction head were enabled to participate in parameter updates. Loss calculation is performed based on a pre-defined multi-dimensional loss function, and high-risk samples are subjected to enhanced constraints by combining sample weights to guide the iteration of network parameters; A small-step, multi-round training mode is adopted to perform short-cycle iterations. After each round of training, the entire chain of migration generation, frame detection, automatic repair, and quality acceptance is restarted. We continuously collect new defect samples and iteratively conduct incremental training to gradually improve the defects of model timing jitter, tooth breakage, and inconsistent mouth shape, thus completing closed-loop self-supervised optimization.

6. A face motion transfer model optimization system based on frame-level consistency verification, characterized in that, The system is used to execute executable instructions to perform the face motion migration model optimization method based on frame-level consistency verification as described in any one of claims 1 to 5.

7. An electronic device, characterized in that, include: First processor; The processor also includes a memory for storing executable instructions of the first processor; wherein the first processor is configured to execute the face motion migration model optimization method based on frame-level consistency verification as described in any one of claims 1 to 5 by executing the executable instructions.

8. A computing device, the device comprising a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein, When the computer program instructions are executed by the processor, the device is triggered to execute the face motion migration model optimization method based on frame-level consistency verification as described in any one of claims 1 to 5.

Images