An audio-visual lip-synching video generation method, system and device based on a discrete code prediction model

Abstract

The application discloses a kind of audio linkage lip video generation method, system and equipment based on discrete code prediction model, applied in data processing technical field, audio, face lip video and mouth shape cluster code table data are collected;Three-stage sample grading is realized by teacher model output, discrimination and boundary mask, and error code suppression, timing control and rendering rule are set with supporting differentiated supervision strategy.Extract frame-level audio features and mouth shape information processed by WENet;Determine optimization strategy in combination with pronunciation, video and computing power conditions, configure label supervision, error code correction and rendering scheduling parameters.Learn audio and mouth mapping based on AudioEncoder, timing smoothing module and 512x16 vector quantizer, complete space-time correlation of audio features, top-k candidate code, frame state and template.Relying on hierarchical supervision, error code suppression, dynamic programming path routing, spike suppression, target code correction and ROI rendering algorithm, generate lip video by fusing candidate confidence, cross-frame cost and template features.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, system, and device for generating audio-linked lip-sync video based on a discrete code prediction model. Background Technology

[0002] With the rapid development of multimedia and digital human technologies, audio-driven lip-sync video technology has been widely applied in various scenarios such as virtual avatar live streaming, film and television dubbing, and intelligent interaction. Currently, the mainstream implementation solution in the industry is clustering code table technology. This solution has a mature overall process, relatively low engineering implementation difficulty, and is also the most widely used technical route at present. The conventional implementation process of the existing clustering code table solution is as follows: First, collect synchronously recorded audio and facial lip-sync video materials; second, extract local mouth area images from the video footage to complete lip-sync sample collection; then, classify the massive lip-sync samples using a clustering algorithm to generate discrete codes and corresponding lip-sync center templates, constructing a dedicated clustering code table; in the inference application stage, based on the input audio features, match the cluster centers frame by frame, select the corresponding template, complete video rendering, and finally output the lip-sync synchronized video.

[0003] While this basic scheme can achieve the basic effect of audio-driven lip-syncing, it has revealed several significant shortcomings in practical applications: The cluster centers are an averaged and compressed representation of massive lip-syncing samples, failing to accurately adapt to fine-grained lip-syncing changes under different pronunciations and speaking speeds. Directly using cluster centers as output during the inference phase easily leads to low lip-syncing-pronunciation matching, a deficiency particularly pronounced in complex pronunciation scenarios. Traditional schemes complete matching decisions on a single frame as an independent unit, lacking temporal constraints on consecutive frames. This easily causes frequent switching of discrete codes, jagged edges in single and double frames, and continuous shaking at the end of the video, ultimately resulting in flickering lips and stiff, unnatural movements. Traditional schemes often employ a "keep / discard" binary label filtering mode, simply dividing usable and discarded samples. This fails to differentiate the use of samples of high, medium, and low quality. Medium-confidence samples cannot be effectively trained, and low-quality samples lack targeted interference suppression, ultimately limiting the model's generalization ability and operational stability. Some discrete error codes can gain a competitive advantage in specific speech contexts and local segments. Even if the model outputs standard target code candidates, the final result will still be dominated by error codes. Moreover, it is difficult to completely correct these errors by simply relying on post-processing, directly causing long-term deviations in the lip movements corresponding to key pronunciations. Traditional rendering methods often directly read the entire template video for image blending, which involves a large number of video decoding and random reading operations, resulting in high I / O overhead and long processing time per frame. This makes it difficult to meet the engineering requirements of large-scale video generation and high-concurrency deployment. Summary of the Invention

[0004] To solve the above-mentioned technical problems, the present invention provides the following technical solution: A method for generating audio-linked lip-shape videos based on a discrete code prediction model includes: acquiring audio, facial lip-shape videos, and associated data from an initial lip-shape clustering code table; and generating videos based on a teacher model (top1), a discrimination margin, and a transition boundary mask. Mask implements a three-stage sample confidence level classification, coupled with differentiated hard / soft label supervision strategies. It sets error code orientation and attraction suppression rules, multi-condition triggering mechanisms, temporal jump cost thresholds, and ROI rendering scheduling rules. The original sample labels are graded and the code value competition relationship is optimized. A supervision dataset adapted for discrete code prediction model training is constructed. Frame-level audio semantic features, lip-sync discrete code sequences, candidate code confidence, and sample quality assessment information are extracted from WENet encoded outputs, frames with shape (T, 20, 256) and completed silence replacement. Discrete code prediction and temporal optimization strategies are determined based on pronunciation detail complexity, video segment length, inter-frame jitter, and hardware computing resources. Label supervision type, error code correction coefficient, window stability constraints, template caching, asynchronous prefetching, and pipeline rendering scheduling parameters are set according to the code value competition rules, inter-frame state changes, abnormal glitches, and rendering load. Aud is adaptively trained according to the strategy. A discrete code prediction model consisting of ioEncoder, temporal smoothing, and a 512×16 vector quantizer is used to learn the mapping relationship between audio features and lip-sync discrete codes, the inter-frame code value switching logic, and the ROI template matching rules. The model spatiotemporally associates the audio feature sequence, the five top-k candidate discrete codes retained during the inference stage, the inter-frame state information, and the local lip-sync template. Relying on three-stage hierarchical supervised training, error code pairwise-attractor directional competition suppression, dynamic programming path selection, spike suppression, event-triggered stabilization, target code directional correction, temporal sequence path selection optimization, and ROI local rendering algorithms, the model calculates the mapping relationship between audio features, lip-sync discrete codes, inter-frame temporal states, and lip-sync images. It combines candidate code confidence, cross-frame jump cost composed of emission cost, transition cost, and switching penalty, and template local features for weighted fusion to generate temporally stable and lip-sync accurate audio-driven lip-sync videos.

[0005] An audio-linked lip-sync video generation system based on a discrete code prediction model is disclosed. The system is used to execute executable instructions to perform the aforementioned audio-linked lip-sync video generation method based on a discrete code prediction model.

[0006] Its beneficial effects are as follows: First, it collects audio, facial lip-shape video, and associated data from an initial lip-shape clustering code table; combining multiple dimensions such as lip-shape accuracy and temporal continuity, it implements a three-stage sample grading system based on the teacher model's top-1 ranking, discriminative power, and switching boundary mask, and provides a differentiated supervision strategy, setting error code suppression, temporal control, and ROI rendering scheduling rules. It constructs a supervised dataset adapted for training the discrete code prediction model, extracting frame-level audio features (T, 20, 256) and lip-shape discrete code sequences from WENet-encoded frames with silence replacement; combining pronunciation complexity, video length, frame jitter, and hardware computing power, it determines discrete code prediction and temporal optimization strategies, and configures label supervision, error code correction, window constraints, and rendering scheduling parameters. Based on the architecture of AudioEncoder, temporal smoothing module and 512×16 vector quantizer, the audio-lip mapping relationship is learned, and audio features, 5 sets of top-k candidate codes, frame state and lip-sync template are spatiotemporally associated and frame sequence is synchronized. Relying on hierarchical supervision, error code suppression, dynamic programming path selection, peak suppression, target code correction and ROI rendering algorithms, candidate confidence, cross-frame transition cost and template features are integrated to generate temporally stable and lip-sync accurate audio-driven lip-sync video.

[0007] This application aims to employ a three-stage hierarchical supervision mechanism to distinguish between high, medium, and low confidence samples and match them with soft and hard labels. By fully utilizing various sample types, it enhances the model's generalization ability and training stability. It adds a Pairwise-attractor-oriented error code suppression and multi-condition target code orientation correction strategy to effectively solve the error code preemption problem and significantly improve the accuracy of lip-sync matching. Temporal optimization is achieved through codebook embedding spatial dynamic programming. An event triggering mechanism based on windows and transition thresholds suppresses inter-frame spikes and jitter, resulting in a more natural and smooth lip-sync transition. The application uses ROI templates combined with caching, asynchronous prefetching, and pipelined rendering to reduce decoding and I / O overhead, significantly improving video generation efficiency and making it more suitable for batch deployment in engineering projects. Attached Figure Description

[0008] Figure 1 A flowchart of an audio-linked lip-sync video generation method based on a discrete code prediction model provided in this embodiment of the invention; Figure 2 This is a schematic diagram of a module for an audio-linked lip-sync video generation system based on a discrete code prediction model, provided in an embodiment of the present invention. Detailed Implementation

[0009] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention. Figure 1This application describes an audio-linked lip-sync video generation method based on a discrete code prediction model, according to an exemplary embodiment of the present application.

[0010] In this application embodiment, an audio-linked lip-sync video generation method based on a discrete code prediction model is described, such as... Figure 1 As shown: S101, acquire the associated data of audio, face lip shape video and initial lip shape clustering code table.

[0011] In one implementation, audio serves as the core input driving lip-shape changes, comprehensively covering various pronunciation forms such as daily conversation, formal reading aloud, short sentence broadcasting, and continuous speech, while also distinguishing diverse speech speeds and tones, including smooth, rapid, soft, and loud. The acquisition equipment primarily uses high-fidelity recording devices to capture continuous speech signals throughout the entire process, completely preserving all acoustic details such as pitch, timbre, and syllable transitions, without pre-modification such as noise reduction or gain adjustment, ensuring the integrity and usability of the original speech information. After audio acquisition, equal time segmentation is performed according to the standard video frame rate. Commonly used video frame rates in the industry include 25 frames per second and 30 frames per second, corresponding to dividing the entire speech segment into audio segments of 0.04 seconds and 0.033 seconds each, strictly ensuring that the duration of a single audio segment is completely equivalent to the duration of the corresponding single video frame.

[0012] During actual data collection, four typical material categories can be selected: daily conversational audio, three-minute standard reading transcripts, single-word short sentence broadcasts, and rapid connected speech. Each category is configured with three speaking speeds: slow, normal, and fast, to enrich the types of audio samples. All audio segments are strictly sorted according to their original playback time and stored as separate original audio signal files. These audio segments will serve as the raw input for the audio feature extraction module, directly invoked when extracting frame-level semantic features using WENet. This establishes a one-to-one business correspondence with concurrent lip-sync samples and discrete code data, ensuring basic matching of the audio and lip-sync data.

[0013] Video acquisition focuses on capturing complete dynamic images of human faces. The shooting angle is fixed at a frontal, eye-level view, and the indoor lighting environment is kept constant to avoid interference from factors such as changes in light and shadow, image shake, and background clutter. The acquired raw video is then decomposed frame by frame at the same standard frame rate as the audio. For example, after decomposition at 30 frames per second, one second of video generates thirty independent still images. The mouth area within the face is labeled for each image, clearly defining the boundary between the global face area and the local mouth shape area, accurately delineating the ROI location, and providing clean raw images for subsequent lip shape sample extraction and local template creation.

[0014] During actual recording, multiple test subjects with different facial features were selected to record facial videos in various states, including normal speaking, whispering, exaggerated pronunciation, and natural conversation. The videos were then split frame by frame and arranged in playback order. Each complete video file was paired with a simultaneously recorded audio file, forming audio-video pairs. These paired data serve as the foundation for subsequent lip-sync sample clustering, sample confidence level classification, and error code statistical analysis. The original video information was preserved throughout the process without any cropping, filtering, or other secondary processing.

[0015] The initial lip-shape clustering code table is generated based on all the split video lip-shape samples, following the implementation logic of traditional clustering code tables. First, lip-shape image samples are extracted in batches from the local lip-shape regions annotated in all video frames. A single sample set can collect tens of thousands of frames of mouth images corresponding to different pronunciations. Then, a clustering algorithm is used to merge and divide all lip-shape samples, grouping lip shapes with highly similar visual morphology and contour features into the same category. Based on practical application requirements, the entire set of lip-shape samples can be clustered into 512 basic categories. Each category is assigned a unique discrete numerical code (0~511) and matched with a dedicated standard lip-shape template image. The initial lip-shape clustering code table is composed of all discrete codes and their corresponding templates.

[0016] During the code table generation process, feature data such as the lip shape contour, opening and closing amplitude, and sample distribution range corresponding to each discrete code are recorded simultaneously to ensure that each set of discrete codes is uniquely mapped to the lip shape template, avoiding chaotic one-to-many or many-to-one relationships. This clustering code serves as the labeling benchmark for the entire scheme. Subsequent work, such as determining sample confidence levels, screening and statistically analyzing error codes, analyzing code value competition patterns, and setting weight parameters, will all use this code table as a unified reference.

[0017] S102 implements a three-stage sample confidence level classification based on the teacher model top1, the discrimination margin, and the transition mask. It is equipped with a differentiated hard / soft label supervision strategy, and sets error code orientation to attract and suppress rules, multi-condition triggering mechanism, time-series transition cost threshold, and ROI rendering scheduling rules.

[0018] In one implementation, a multi-dimensional constraint analysis technique is used to integrate the overall technical requirements, encompassing lip-sync accuracy, temporal continuity, training sample validity, and rendering efficiency. This is combined with the top-1 candidate weights output by the teacher model, the margin between top-1 and top-2 candidates, and the transition mask to generate a three-segment sample confidence grading basis, error code-oriented competition constraint index, inter-frame jump control threshold, and ROI rendering link scheduling benchmark data. The overall analysis revolves around four core requirements: lip-sync accuracy, temporal continuity, training sample validity, and rendering efficiency. In audio-driven lip-sync video scenarios, lip-sync accuracy measures the degree of matching between a single frame's lip shape and the clustering standard template; temporal continuity evaluates the smoothness of lip-sync switching between consecutive frames; training sample validity distinguishes the value of different frame data for model training; and rendering efficiency measures the overall operational overhead of video decoding, template reading, and image compositing. The analysis work is based on synchronously collected audio, facial lip shape video, and lip shape clustering code table. Multiple sets of candidate weight data output by the teacher model during the training phase are introduced. The discrimination is obtained by calculating the difference between the first candidate weight and the second candidate weight. At the same time, a switching boundary mask is generated according to the change of discrete code in adjacent frames. All information is combined to sort out four major categories of judgment criteria and execution indicators: sample classification, error code control, inter-frame jump, and template rendering.

[0019] In actual execution, the system first retrieves frame-by-frame paired audio data and facial lip-shape images, comparing the current lip shape with the standard templates in the clustering library frame by frame. It then counts the frequency and amplitude of lip-shape switching in consecutive frames, distinguishing between high-value usable samples, ordinary samples, and invalid low-quality samples. Simultaneously, it tracks the time consumed by template reading and image decoding during the entire video rendering process. The teacher model outputs multiple sets of candidate weights for each frame, defining the weight ranked first in a single frame as the first candidate weight, the weight ranked second as the second candidate weight, and subtracting the two to obtain the discrimination score. The ranking object is all discrete lip-shape codes within a single frame, based on the weight values ​​of each discrete code output by the teacher model, arranged in descending order of weight; the higher the weight value, the higher the ranking. These are used to calculate the first candidate weight, second candidate weight, and discrimination score, serving as the basis for determining the sample confidence level. When the system detects a change in the discrete lip-shape code between adjacent frames, it extends two frames forward and two frames backward from the location of the code value change, uniformly marking it as a switching boundary area and generating a unique mask identifier. Based on the above statistical results, weighted data, and regional identifiers, the criteria for classifying samples into three categories—high confidence, medium confidence, and low confidence—are defined. The identification and evaluation indicators for target codes and error codes are clarified, and the control standards for normal transitions of inter-frame discrete codes are set. At the same time, basic scheduling rules for facial region of interest templates are formulated, providing a unified and quantifiable data benchmark for subsequent full-link rule configuration.

[0020] Specifically, the target code is the discrete code of the real standard lip shape corresponding to the pronunciation of a single frame of audio. It is obtained by clustering and labeling of high-quality real human lip shape videos with synchronous alignment. It is the correct lip shape code that should theoretically be output for this frame. In the ideal state of the teacher model, the target code should be the candidate code with the highest ranking and weight.

[0021] Error codes are discrete interference codes that do not match the pronunciation of the current frame's audio, are easily confused with the target code, and have high prediction weights. These include similar codes with similar pronunciations and high visual similarity in lip shape; interference codes with weights close to the target code in time-series switching boundaries and blurred pronunciation frames, which preemptively occupy high rankings; the system will trigger pairwise-attractor pairwise competitive suppression based on the ranking, weight difference, and duration of the target code and error codes, thereby reducing the prediction weight of error codes.

[0022] The standard target code is a baseline target code standardized and normalized from the entire dataset. It has undergone audio and video synchronization calibration and multi-sample mean smoothing correction to eliminate annotation bias caused by single-person shooting shakiness and local image defects. It serves as a unified benchmark reference code for three-stage hierarchical supervised training and dynamic programming path selection. During the inference phase of dynamic programming path finding, it serves as a benchmark reference for the optimal temporal path, and temporal transition cost penalties are imposed on candidate paths that deviate from this code sequence.

[0023] This system unifies and integrates three-stage sample grading rules, error code-oriented attraction and suppression logic, inter-frame temporal control standards, and facial region of interest rendering requirements. It establishes a one-to-one correspondence between the sample grading supervision mechanism, error code management mechanism, frame temporal constraint mechanism, and rendering scheduling mechanism, addressing four key objectives: lip-sync accuracy, temporal continuity, training sample effectiveness, and rendering efficiency. This constructs a rule-based execution strategy covering the entire process from training data preprocessing, supervised model training, inference code value optimization, to video compositing and rendering. The entire strategy differentiates between training and inference scenarios, clearly defining the effective scope, linkage relationships, and priorities of different mechanisms to ensure coordinated operation of all modules.

[0024] During the rule matching and strategy building phase, the mechanism integration was completed sequentially according to the business process. To address the issues of lip-sync errors during location switching and loss of information in medium-sized samples caused by traditional binary label filtering, a three-stage sample grading rule was adopted as the core supervision scheme during the training phase, with differentiated supervision methods configured for boundary frames and steady-state frames. To address the problem of error codes continuously crowding out standard target codes and lip-sync output deviations in certain pronunciation scenarios, error code targeting and suppression logic was implemented, limiting the mechanism to only take effect within a specified sample range to avoid altering the training labels of high-confidence samples. To address the frequent code value jumps and single-frame glitches caused by frame-by-frame decision-making during inference, a multi-layered temporal control standard was implemented, constraining inter-frame changes from multiple dimensions including path selection, spike repair, and event stabilization. To address the high I / O overhead caused by reading entire frame templates, a dedicated rendering scheduling logic for regions of interest was designed to streamline the data reading range. The entire strategy is arranged according to the business sequence of data preprocessing—model training—audio inference—temporal optimization—video rendering. Each mechanism is independent and triggered as needed, allowing for complete implementation in both model training and online inference application scenarios.

[0025] Based on the rule-based customized execution strategy, core configuration requirements are extracted, including top1, margin, The sample confidence features constitute the input dimension, the competition state between the target code and the error code is the core parameter, and the temporal performance of the frame sequence is the judgment criterion. This enables the scientific setting of three sample levels, multi-condition triggering rules for error codes, temporal jump thresholds, and ROI rendering logic. The first candidate weight, discrimination, and switching boundary mask output by the teacher model are used as the core basis for judging the confidence level of the samples. The ranking, weight difference, and continuous occupation duration of the target code and the error code are used as the judgment conditions for the code value competition state. The code value switching frequency and jump amplitude within the frame sequence are used as the verification standard for temporal performance. Combining the discrete code prediction model architecture, audio feature format, and candidate code output rules, the sample level division, error code triggering rules, inter-frame jump threshold, and rendering logic are configured in a refined manner. At the same time, the effective mask and boundary constraints of each rule are clarified. Among them, the current ranking refers to the ranking position of the target code and the ranking position of the error code in the aforementioned global ranking results. By retrieving the ranking of the two types of codes in the global weight ranking, and combining the weight difference and continuous occurrence duration of the two, the code value competition state is judged so as to trigger the multi-condition suppression rule of error code.

[0026] Based on the top-k soft targets output by the teacher model, the first candidate weight (top1), the second candidate weight (top2), and the discrimination margin (margin = top1 - top2) are calculated for each frame. A transition mask is constructed: when the discrete code changes between adjacent frames, the boundary region is marked by extending the boundary region by 2 frames before and after the change point. High-confidence samples are determined to have top1 ≥ 0.41 and margin ≥ 0.05 and are not in the transition mask, and are subject to joint supervision using hard, soft, and temporal methods. Medium-confidence samples are determined to have top1 ≥ 0.30 and margin ≥ 0.015 but do not meet the requirements. (High-confidence samples have a complete set of constraints), using only soft / temporal supervision with the hard label set to -1; low-confidence samples do not meet the soft condition and do not participate in the hard / soft / temporal main loss. Switching zone The samples (samples using only soft labels and temporal loss, without hard supervision, are divided into two categories: switching region and stable region, with fixed loss weights) have a soft weight of 0.15 and a temporal weight of 0.0; the stable region... The sample soft weight is 0.62, and the temporal weight is 0.38.

[0027] The error code targeting suppression mechanism only applies to mid-confidence samples and frames whose original label is the standard target code. It also imposes multiple constraints: the top-ranked element in the current frame must be an error code; the standard target code must exist within the candidate sequence; the weight difference between the error code and the target code must not exceed 0.12; and the number of frames where the error code consecutively occupies the top position must not exceed four. When all conditions are met, the weight adjustment process is initiated. This rule also supports optional configurations that exclude switching boundary regions and fragment segments.

[0028] A unified audio extraction standard is adopted. For 16kHz raw audio, WENet encoding, VAD timestamp alignment, and silence feature replacement are used to process the final output frame-level semantic features of size (T, 20, 256). During the model training phase, 3 sets of candidate codes are retained per frame, and during the inference phase, 5 sets of candidate codes are output per frame by default. All candidate codes are used as the basic data for temporal optimization.

[0029] The timing threshold rules are set as follows: a single frame or two consecutive frames with a code value jump amplitude of 16 are judged as an abnormal spike; within a window centered on the current frame and with a radius of 6 frames, frames with a code value switching count greater than or equal to 4 and a jump amplitude of 56 are judged as a high-risk event area. Different abnormal types correspond to differentiated processing logic. The rendering logic is set as follows: around the region of interest template, the entire set of operating logic, including template indexing, local caching, asynchronous prefetching, and pipelined encoding, is configured, and the scheduling strategy is adjusted according to the concurrency of different business operations.

[0030] Combining requirements for lip-sync accuracy, image timing stability, and engineering efficiency, quantitative calibration and optimization adjustments were made to various rule parameters. This clarified the judgment boundaries and differentiated supervision methods for high / medium / low-quality samples, the triggering conditions and weight scaling parameters for error code suppression, various threshold ranges for timing transitions, and the operational mechanisms for ROI template caching / asynchronous prefetching / pipeline rendering, forming a complete end-to-end operational rule system. Focusing on three core objectives—lip-sync accuracy, image timing smoothness, and engineering deployment efficiency—four categories of parameters—sample supervision, error code control, timing constraints, and rendering scheduling—were individually debugged and optimized. All quantized values, computational rules, effective ranges, and linkage logic were clarified. Simultaneously, a three-stage model architecture consisting of AudioEncoder, timing smoothing module, and vector quantizer was used to complete the adaptation and connection between rules and the model, ultimately forming a complete rule system covering the entire training, inference, and rendering process, ready for direct deployment and application.

[0031] High-confidence samples retain all hard, soft, and temporal labels for model training calculations. Medium-confidence samples within the transition mask region have a soft label weight of 0.15 and a temporal label weight of 0, retaining only transition features without strong temporal constraints. Medium-confidence samples within the stable region (not within the transition mask) have a soft label weight of 0.62 and a temporal label weight of 0.38, supplementing steady-state feature constraints. Low-confidence samples are directly masked and do not participate in any loss calculations. This configuration preserves the dynamic features of frame transitions while maintaining steady-state lip-sync accuracy.

[0032] After error code suppression is triggered, the original weights of the error codes are multiplied by 0.90 to reduce the weights. The reduced weight values ​​are then fully fed back into the standard target code. Normalization is then performed on all candidate weights in the current frame to ensure that the total weights remain within the normal range. The entire process is executed only within the preset mask and will not modify the training labels of high-confidence samples, thus avoiding damage to the steady-state features already learned by the model.

[0033] Based on baseline (Taking the complete set of error code suppression configuration parameters as an example) =45 (Standard target code number 45, the correct lip shape discrete code for pronunciation matching); =472 (Error code number 472 to be suppressed, an interference code that is similar to the target code lip shape and is easy to preempt the high confidence ranking); =0.90 (Error code weight decay coefficient); Triggering condition ( The following belongs to And not ; original tag ∈ Not in transition mask ( (This is a region constraint and not within the transition mask switching boundary); located in the fragment segment ( (This is a segment constraint; the current frame belongs to a speech fragment segment). The competition strength is determined as follows: =True (The top-ranked global confidence index of the current frame must be error code 472); =True (Standard target code 45 must be included in the top-k candidate list); =0.12 (Error Code Weight) (Target code weight ≤ 0.12). =4 (the number of frames with consecutive error codes as top 1 is ≤4).

[0034] Error code contention suppression only applies to soft-only frames where the original label belongs to the target code. Taking target code 45 and error code 472 as an example, when the top-ranked error code is the target code, the target code appears in the top-k candidate list, the weight difference between the error code and the target code does not exceed 0.12, and the consecutive dominant length of the error code does not exceed 4 frames, a weight adjustment operation is performed. This mechanism is a targeted de-attracting strategy, only processing error codes that meet the conditions, rather than indiscriminately reducing the weight of all candidate codes in the sequence.

[0035] For single-frame and double-frame spikes with a jump amplitude of 16, stable code values ​​on both sides are used for backfilling and repair. Boundary masks are used to skip normal audio switching areas to prevent over-correction. For high-risk event areas within a 6-frame radius window, the area is expanded outward by 4 frames before an enhanced spike suppression strategy is implemented. Additional tail-segment stabilization logic is added to the end of the sequence to reduce video tail jitter. Timing optimization uses dynamic programming for routing based on the five candidate codes output by inference, and calculates the globally optimal code sequence by comprehensively considering transmission cost, target deviation cost, transition cost, and handover penalty.

[0036] The discrete code prediction model consists of an audio encoder, a temporal smoothing module, and a 512×16 vector quantizer. Audio features are mapped to 16-dimensional latent variables via a multilayer perceptron before temporal smoothing and codebook matching are performed. The data format and dimensions of all rule outputs match the model's input and output formats one-to-one, ensuring smooth feature flow and code value transmission. A region-of-interest (ROI) dedicated rendering mode is enabled, relying on template caching to reduce redundant decoding operations. Subsequent frame templates are asynchronously prefetched during video playback, and local image fusion and audio-visual synthesis are completed through a pipelined approach, significantly reducing I / O overhead.

[0037] S103 performs hierarchical processing on the original sample labels and optimizes the code value competition relationship, constructs a supervised dataset adapted for training the discrete code prediction model, and extracts the WENet encoded output, frame-level audio semantic features with shape (T,20,256) and completed silence replacement, lip-sync discrete code sequences, candidate code confidence, and sample quality evaluation information.

[0038] In one implementation, based on sample confidence differences, code value competition status, label adaptability, and model training stability requirements, the original lip-shape sample labels are reconstructed and optimized for code value competition using the top-1, discriminative margin, and transition mask output by the teacher model. This is achieved by leveraging a three-stage supervision rule and an error code-oriented de-attraction mechanism to generate a hierarchical label system, multi-condition error code control rules, sample quality rating standards, and WENet audio feature extraction specifications. Combining single-frame sample confidence levels, adjacent discrete code competition, label adaptability, and overall model training stability requirements, the original lip-shape sample labels are reclassified and their content adjusted using the first candidate weight, the discriminative margin between the first and second candidates, and the boundary markers generated by inter-frame discrete code switching, all based on the teacher model's inference output. To address the frequent issue of error codes crowding out standard target codes during training, competing discrete codes are simultaneously optimized and corrected. This step comprehensively applies a three-stage supervision rule and an error code-oriented de-attraction mechanism, sequentially building a hierarchical label system, error code control execution rules, sample quality rating standards, and a unified audio feature extraction specification based on the WENet network.

[0039] In actual sample processing, multiple sets of candidate weight data are first generated for each frame of lip-sync samples based on the teacher model. The difference between the first and second candidate weights is calculated numerically and used as the basis for candidate differentiation. When a change in the discrete code of the lip-sync is detected between adjacent frames, a switching boundary region is defined two frames before and after the code value change position, and a unique identifier is generated to distinguish between steady-state frames and dynamically switching frames. Combining the weight values, the size of the distinction, and the boundary identifiers, the original single label is split and reconstructed, changing the traditional label usage mode of either retaining or discarding labels, and matching the corresponding label application form for samples of different quality. To address the problem of error codes continuously occupying the first position in the pronunciation scenario and the inability of standard target codes to be output normally, directional control logic is enabled to clarify the recognition range and adjustment conditions of error codes and target codes. At the same time, the entire audio data parsing, encoding, and post-processing process is unified, and the feature output format and size are determined, so that the label usage rules, discrete code control logic, sample evaluation standards, and audio extraction specifications are matched with each other to form a unified standard, laying a solid foundation for the subsequent construction of the entire training dataset.

[0040] The overall architecture is based on two core operating mechanisms: three-stage hierarchical supervision and error code-oriented attraction. The entire training data processing flow is layered and broken down into four independent working modules according to the business sequence: sample confidence determination, label type matching, code value competition correction, and multi-class feature extraction. The work content, execution logic, data transfer direction, and interaction rules between modules are defined one by one, ultimately forming an overall execution plan in which label classification, code value optimization, and audio feature extraction are carried out in parallel and in an orderly manner.

[0041] The entire training data processing workflow operates sequentially in a fixed order, with each module closely integrated upstream and downstream, and data transferred unidirectionally without overlap. The sample confidence determination module, as the first execution unit, specifically determines the quality level of a single-frame sample based on candidate weights, distinguishing values, and switching boundary markers. The results after determination are directly transmitted to the label type matching module, which selects the appropriate combination of hard labels, soft labels, and temporal labels based on the sample level. After label configuration, the data enters the code value competition correction module, which identifies the competition state between target codes and error codes within the frame and adjusts the weights when preset conditions are met. Finally, the feature extraction module uniformly parses audio, lip-sync discrete codes, candidate confidence scores, and other content, outputting standardized feature data. Each of the four modules performs its own function, with the output of the previous module serving as the input for the next. The entire data flow is clear, allowing for both individual unit debugging and coordinated operation, fully adapting to the full-cycle training requirements of discrete code prediction models.

[0042] With the core objectives of improving the overall utilization of training samples, ensuring the accuracy of lip-sync discrete code prediction, enhancing error code suppression, and standardizing dataset format, this study combines sample features corresponding to different pronunciation scenarios and video segments to define the dividing lines between high-confidence, medium-confidence, and low-confidence samples, and clarifies the activation rules and specific weight values ​​for hard labels, soft labels, and temporal labels. Simultaneously, it refines the activation conditions and weight scaling ratios for error code control, and standardizes the extraction methods, judgment thresholds, and final data output format for frame-level audio semantic features, lip-sync discrete codes, and candidate codes.

[0043] When classifying samples, fixed values ​​are used as the criterion. If the weight of the first candidate in a single frame is greater than or equal to 0.41, the candidate discrimination is greater than or equal to 0.05, and the frame position is not within the switching boundary region, the sample is classified as a high-confidence sample, and hard labels, soft labels, and temporal labels are simultaneously enabled for model loss calculation. If a sample does not meet the high-confidence condition, but the weight of the first candidate is greater than or equal to 0.30 and the candidate discrimination is greater than or equal to 0.015, it is classified as a medium-confidence sample. For this type of sample, hard labels are disabled, and only soft labels and temporal labels are used. Specifically, for medium-confidence samples within the switching region, the soft label weight is set to 0.15 and the temporal label weight is set to 0; for medium-confidence samples within the stable region, the soft label weight is set to 0.62 and the temporal label weight is set to 0.01. Samples with a first candidate weight less than 0.30 or a candidate discrimination less than 0.015 are classified as low-confidence samples, and these samples are not included in any loss calculation.

[0044] For error code control, it only applies to frames that are of medium confidence and whose original label is the standard target code. It also requires that the current candidate first position is an error code, the standard target code exists in the candidate list, the weight difference between the error code and the target code does not exceed 0.12, and the number of frames in which the error code occupies the first position consecutively does not exceed 4 frames. After all conditions are met, the error weight is multiplied by 0.90 to complete the reduction. The reduced weight value is added to the target code, and then all candidate weights are normalized.

[0045] Audio data is uniformly sampled at 16kHz. After WENet encoding, speech activity timestamp alignment, and silence feature replacement, the output is a fixed-size frame-level semantic feature. During model training, three sets of candidate discrete codes are retained per frame, and during inference, five sets of candidate discrete codes are retained per frame. All feature and code value data are output according to a unified dimension to ensure that the dataset format is consistent and standardized.

[0046] This standard integrates sample grading strategies, code value optimization schemes, and audio feature extraction rules to construct training dataset construction control information. This includes a three-stage sample confidence grading rule, an error code orientation and de-attracting execution process, WENet frame-level semantic features, and multi-dimensional feature extraction standards. The integrated standard covers the entire process of raw sample processing, label rewriting, code value correction, and feature output. When creating training datasets in batches, staff can complete all operations such as sample selection, label configuration, error code adjustment, and feature extraction according to the unified rules. The entire standard is fully compatible with the discrete prediction model, which consists of an audio encoder, a temporal smoothing module, and a 512×16 vector quantizer. After audio features are input into the model, the audio encoder first converts the feature dimensions into 16-dimensional continuous latent variables. Then, the temporal smoothing module enhances the inter-frame continuity through one-dimensional convolution. Finally, the vector quantizer calculates the Euclidean distance based on 512 codebooks, outputting the lip-sync discrete code in the range of 0 to 511 and the corresponding candidate sequences. The dataset output data format, feature dimensions, and code value forms all correspond one-to-one with the model input requirements, providing stable data support for model training. During model training, the three-stage supervision and error code suppression mechanism continuously play a role, avoiding problems such as label contamination and error code preemption from the data source, steadily improving the model's learning effect, ensuring the accuracy of the mapping relationship between audio and discrete lip-shape codes, and providing a reliable foundation for subsequent audio-driven lip-shape video inference, temporal optimization, and image rendering.

[0047] S104 combines the complexity of pronunciation details, video segment length, inter-frame jitter, and hardware computing resources to determine discrete code prediction and timing optimization strategies. Based on code value competition rules, inter-frame state changes, abnormal glitch characteristics, and rendering load, it sets label supervision type, error code correction coefficient, window stability constraints, template caching, asynchronous prefetching, and pipeline rendering scheduling parameters.

[0048] In one implementation, four types of basic information are uniformly sorted and integrated: the complexity of audio pronunciation details, the length of video segments, the jitter state between frames, and the computing power of the device hardware. Combining the overall architecture and timing control logic of the three-segment discrete code prediction model (audio encoder + timing smoothing module + 512×16 vector quantizer) of this solution, and matching the model's operating specifications, a complete data processing flow is established before discrete code prediction and timing optimization work begins, ensuring that the front-end input data and data flow format fully adapt to the operational requirements of each level of the model.

[0049] Before formally formulating optimization strategies, a comprehensive assessment of all business data and the operating environment was conducted. For audio data, different pronunciation scenarios were differentiated, including everyday short dialogues, long text readings, and continuous rapid pronunciation. The complexity of pronunciation details was determined based on syllable density and the frequency of pronunciation transitions. For accompanying facial lip-sync videos, they were categorized into short and long clips based on usage duration. The video frames were inspected frame by frame, and the frequency and distribution of anomalies such as single-frame glitches, continuous jitter, and significant bit value jumps were statistically analyzed. Simultaneously, hardware resource parameters such as the device's processor, memory, and video memory were read to confirm the overall machine's load capacity and computational limits in real time.

[0050] The audio encoder in this scheme flattens and operates on audio features of size (T, 20, 256). The temporal smoothing module enhances inter-frame continuity through one-dimensional convolution, and the vector quantizer outputs discrete codes in the range of 0 to 511 based on 512 codebooks. The entire pre-processing workflow is adapted to the aforementioned model hierarchy, data format, and computational characteristics, unifying audio slicing rules, feature output formats, and video frame grouping methods. This ensures that all front-end processing results conform to the model's operating standards, laying the groundwork for subsequent strategy design and parameter configuration.

[0051] Based on the pronunciation characteristics of the audio itself, the competition rules between discrete lip-sync codes, the state changes of consecutive frames, and the characteristics of abnormal glitches in the image, a multi-dimensional comprehensive analysis framework is built. The inherent attributes of video segments and real-time load data from the video rendering process are integrated into the analysis system, establishing a mechanism for parameter linkage verification, abnormal state identification, and dynamic hardware load adaptation, achieving dynamic matching of state, parameters, and hardware.

[0052] The entire analysis framework covers four dimensions: audio, lip-sync sequence, video image, and hardware resources. It combines the syllable switching patterns of different pronunciations to determine the normal switching logic of the corresponding discrete lip-sync codes, while identifying abnormal states such as erroneous codes persistently crowding out the standard target code, frequent inter-frame jumps, and isolated spikes in a single frame. It also combines video length and the number of concurrent applications to determine the read / write and decoding load of the rendering process. When the short segment code value differs from the left and right stable values ​​and the jump amplitude exceeds the threshold of 16, spike suppression is triggered, and the switching boundary region is constrained by a protective mask.

[0053] The following linkage rules are set during operation: when complex pronunciation or increased inter-frame jitter is detected, the timing constraint parameters are automatically increased to strengthen the correlation constraints between frames; when hardware computing power is insufficient or the rendering load is high, non-core timing operations are simplified and the number of template prefetches is reduced to lower the overall computing pressure without affecting lip-sync accuracy. This mechanism can identify operational anomalies in real time and dynamically adjust the corresponding parameters to avoid the problems of decreased effect and overload of device resources caused by fixed parameters in different scenarios.

[0054] With the core optimization goals of improving the accuracy of lip-sync discrete code prediction and eliminating inter-frame jitter and abnormal glitches in the image, a one-to-one correspondence system of operation strategies and various configuration parameters is built by comprehensively considering multiple types of information such as pronunciation complexity, video segment type, discrete code competition relationship, timing anomaly performance, and rendering load, so that different application scenarios can be matched with exclusive strategy combinations and parameter combinations.

[0055] Different operational strategies and corresponding quantitative parameters are applied to different business scenarios. For complex audio content with dense syllables and frequent connected speech, the focus is on enhancing discrete code prediction capabilities, strictly enforcing error code contention control logic, keeping the weight difference between the error code and the target code within 0.12, and limiting the number of consecutive frames where the error code dominates to 4 frames. For longer video dubbing and virtual human broadcasting, temporal constraints are strengthened, window stabilization rules are enabled, and long-sequence cumulative jitter is suppressed. For spikes with single-frame jump amplitudes reaching 16, and high-risk temporal regions with ≥4 switches and ≥56 jumps within a 6-frame window, spike suppression and event-enhanced stabilization strategies are implemented separately, along with corresponding threshold parameters. When there are many batch rendering tasks and high system I / O and computational loads, template caching and asynchronous prefetching parameters are adjusted to reduce unnecessary data interactions. Each operational strategy is bound to fixed thresholds, coefficients, window ranges, and other parameters to achieve precise correspondence between scenarios, strategies, and parameters, ensuring optimal performance in different scenarios. Through three steps—scene feature identification, lip-code operation status verification, and hardware computing load assessment—operational conditions suitable for the current business environment are selected. Inconsistencies between timing parameters, error code parameters, and rendering parameters are identified and conflicts are avoided, ultimately resulting in a deployable overall strategy for discrete code prediction and timing optimization. Strategy selection and verification consists of three stages. First, the current business scenario is identified, distinguishing between different usage scenarios such as daily interaction, film and television production, and live streaming interaction, to initially define suitable strategy directions. Second, the discrete lip-code sequence output by the model is verified, checking the code value switching frequency, error code ratio, and sequence continuity to determine the suitability of the existing strategy. Finally, the remaining computing power, memory usage, and rendering load of the device are assessed to select solutions that can be supported by the hardware.

[0056] During the verification process, the focus is on investigating parameter conflicts. For example, high-level timing constraints increase computational load and cannot be used with low-computing-power devices; error code correction logic only applies to medium-confidence sample frames and cannot be mixed with fully supervised rules for high-confidence samples; the boundary protection mask for spike suppression needs to be isolated from the normal code value switching rules to avoid erroneous correction of normal pronunciation switching. After completing full-dimensional verification and conflict elimination, the final strategy is determined, and the solution can be directly imported into the system for formal operation.

[0057] The determined optimization strategy is imported into the discrete code prediction module and the timing control module. Combining error code contention rules, single-frame and continuous spikes, video tail jitter and other inter-frame anomaly features, as well as the operational pressure of video rendering, the configuration of all parameters, including label supervision type, error code correction coefficient, timing window stability constraint, template caching, asynchronous prefetching, and pipeline rendering, is completed in sequence to form an operation and management scheme covering the entire process.

[0058] After the strategy is imported into the corresponding module of the model, the parameters are configured item by item according to the established rules. The label supervision adopts a three-stage system: high-confidence samples are simultaneously enabled with hard labels, soft labels, and time-series labels; in the switching region, the soft label weight of medium-confidence samples is 0.15 and the time-series label weight is 0; in the stable region, the soft label weight of medium-confidence samples is 0.62 and the time-series label weight is 0.38; low-confidence samples do not participate in the loss calculation.

[0059] The error code correction process strictly follows the following judgment conditions: only when the error code ranks first in the middle confidence frame, the target code exists in the candidate list, the weight difference between the two is ≤0.12, and the error code has been dominant for ≤4 consecutive frames, the error code weight is multiplied by 0.90, the reduced weight is added back to the target code, and then global normalization is performed.

[0060] The timing-related parameters are set according to the anomaly type as follows: when the code value jump amplitude reaches 16, basic spike suppression is triggered; within a window with a radius of 6 frames centered on the current frame, when the number of switching is ≥4 and the jump is ≥56, the range is expanded outward by 4 frames to enable enhanced stabilization; additional tail-end anti-shake logic is added to the end of the video, while retaining the boundary protection mask so as not to interfere with normal audio switching.

[0061] The rendering process uniformly enables local template caching and asynchronous prefetching, employing a pipelined approach to complete local image fusion and audio-visual synthesis, reducing repetitive decoding and random read operations. The entire parameter set perfectly matches the input and output formats of the audio encoder, timing smoothing module, and vector quantizer, spanning the entire process from audio feature extraction, discrete code prediction, timing optimization, to video rendering, forming a complete control system to guide stable and efficient system operation.

[0062] S105 adaptively trains a discrete code prediction model consisting of AudioEncoder + temporal smoothing + 512×16 vector quantizer according to the strategy. It learns the mapping relationship between audio features and lip-sync discrete codes, the inter-frame code value switching logic and ROI template matching rules under the model. It completes the spatiotemporal association of audio feature sequence, 5 sets of top-k candidate discrete codes retained in the inference stage, inter-frame state information and local lip-sync template.

[0063] In one implementation, relying on an overall architecture consisting of an AudioEncoder, a temporal smoothing module, and a 512×16 vector quantizer, the model autonomously learns the mapping rules between audio features and discrete lip-sync codes. It also masters the inherent logic of code value switching between consecutive frames and the matching rules between audio features and local lip-sync templates. The entire model operates along a fixed link, and during the training phase, iteratively optimizes parameters using standardized supervised data to gradually improve mapping accuracy and logical adaptability.

[0064] The entire model is divided into three main functional modules, which are interconnected and work collaboratively. The audio encoder, as the first processing unit, takes as input frame-level audio data after WENet encoding, VAD timestamp alignment, and silence replacement processing. The original audio sampling frequency is set to 16kHz, and after data normalization, each data set consists of 20 time steps superimposed with 256-dimensional features. The encoder first flattens these features, converting them into 512-dimensional data. Then, it performs feature compression layer by layer through a multi-layer fully connected network, successively transforming from 512-dimensional to 128-dimensional and then from 128-dimensional to 16-dimensional, finally outputting a uniform 16-dimensional continuous latent variable. Each layer of the network is equipped with normalization, activation functions, and random deactivation strategies to suppress overfitting while compressing features, thus completing the deep extraction and feature condensation of the original audio features.

[0065] The temporal smoothing module receives the 16-dimensional latent variables output by the encoder and uses grouped one-dimensional convolution operations to smooth the short-term feature sequence. The number of groups is set to 16, and the operation is performed with a dedicated convolution kernel. This method can effectively enhance the feature continuity between adjacent frames, weaken the interference caused by abrupt changes in features in a single frame, and make the feature change trend output by the model closely match the actual state of smooth lip movements during real speech, avoiding abrupt lip movements in the future.

[0066] Finally, the vector quantizer is configured with 512 independent codebooks, each with a 16-dimensional dimension, perfectly consistent with the latent dimension output by the preceding modules. This module compares the similarity between the current latent variable and all code vectors by calculating Euclidean distance, selecting the discrete lip-sync code with the highest matching degree. It also outputs multiple candidate codes and their corresponding weights, sorted by confidence level. During model training, 3 candidate codes are retained per frame, while during inference, 5 candidate codes are output per frame. All candidate data serve as the foundation for subsequent temporal path calculations.

[0067] During long-term iterative training, the model continuously optimizes the mapping accuracy from audio features to discrete lip-shape codes by relying on a three-segment supervision rule and an error code-oriented suppression mechanism. Simultaneously, it continuously learns the code value switching patterns under different pronunciation scenarios, clearly distinguishing reasonable inter-frame changes such as normal syllable switching and artificial speech rate variations, as well as abnormal jitter features such as single-frame spikes and high-frequency jumps. For local lip-shape template matching rules, the model uses a 512-class clustered lip-shape pattern as a benchmark to establish a binding relationship between each discrete code and the corresponding facial region of interest template. This ensures that when the model outputs any set of discrete codes, it can accurately retrieve the corresponding lip-shape template resource, performing pre-matching for image rendering.

[0068] After model learning is complete, the complete audio feature sequence, multiple candidate discrete codes generated during the inference process, frame state information, and local lip-sync templates are integrated to complete the spatiotemporal association binding between different types of data. Data integration strictly follows the model output format and business timing requirements to ensure that all types of information correspond one-to-one and flow in an orderly manner. The audio feature sequence adopts a unified processing standard, using 16kHz raw audio throughout, and sequentially performs WENet encoding, speech activity timestamp alignment, and silence feature replacement operations to finally form a frame-level feature sequence with a fixed form. This sequence serves as the basic input data for model inference and maintains a consistent format throughout the process.

[0069] During the formal inference process of the model, after each frame of audio features is calculated through the complete model chain, it outputs 5 sets of candidate discrete codes and their corresponding confidence weights. These 5 sets of candidate codes are the core basis for subsequent dynamic path selection and anomaly correction. All candidate data are generated sequentially with the audio frames to ensure temporal continuity. The frame status information is a comprehensive judgment result of single frames and local windows, specifically including whether the current frame is in the discrete code switching boundary region, the total number of code value switching within a 6-frame window centered on the current frame, the magnitude of the code value jump, and whether there are single-frame or double-frame abnormal spikes. The system will complete the status judgment based on preset thresholds. A code value jump magnitude of 16 is marked as a spike, and a switching count greater than or equal to 4 within a 6-frame window and a jump magnitude of 56 are marked as a high-risk event region. These judgment results are synchronously appended to the corresponding frame data for the execution of subsequent temporal control logic.

[0070] The local lip-sync templates correspond to the 512 standard lip-sync patterns obtained from the initial clustering. Each discrete code in the range of 0 to 511 uniquely corresponds to a set of exclusive facial region of interest templates. The templates are pre-indexed and can be quickly retrieved and used based on the discrete code. The system uses audio playback time as a unified timeline and performs data binding operations frame by frame, integrating audio features, 5 sets of candidate discrete codes, frame state information, and matched local lip-sync templates at the same time point into a complete set of data. All data within a single frame are interconnected, and the entire data segment is arranged according to the playback order, ensuring the temporal consistency of the three types of information: audio, code, and template. This lays a solid foundation for subsequent global frame sequence synchronization, feature fusion, and image generation.

[0071] Based on single-frame data with completed spatiotemporal correlation, the system performs unified frame-to-frame matching of the entire audio and lip-sync video segments according to the original playback order of the video, ensuring complete synchronization of audio and video timing. The matching process differentiates between frame states, balancing the realism of pronunciation with the stability of the image. The system uses industry-standard video playback frame rates of 25 frames per second (fps) and 30 fps as the global timing benchmark, strictly adhering to the playback rhythm of the original audio, and verifying the integrated dataset segment by segment from beginning to end. Combining the status identifiers attached to each frame, the system distinguishes between stable regions, bit-value switching regions, high-risk event regions, and video tail sections, and activates corresponding timing control logic for each region.

[0072] For normal steady-state frames, the system maintains the current data output state without additional intervention, ensuring a natural and smooth lip-syncing effect for regular pronunciation. For normal bit-value switching areas, the system retains reasonable bit-value change characteristics between frames, applying only basic constraints and avoiding excessive correction that could cause sluggish or abrupt lip movements. For frames marked as abnormal spikes, the system performs backfilling repair according to rules, while also enabling a boundary protection mechanism to avoid normal pronunciation switching locations and prevent erroneous corrections. For high-risk event areas with frequent switching and excessive jumps within a 6-frame window, the system extends the control range outward by 4 frames, enabling an enhanced stabilization strategy to suppress continuous jitter. Additional anti-shake logic is added to the end of the video to improve the lip-syncing jitter that is common at the end of the video.

[0073] The entire frame-sequence matching process does not alter the playback duration or syllable rhythm of the original audio, nor does it tamper with the lip-sync patterns corresponding to normal pronunciation. It only performs targeted corrections for abnormal code values ​​and image artifacts generated by the algorithm. After the entire segment is matched, the timing of the audio, lip movements, and video is perfectly aligned, and the audio and video remain synchronized. At this point, the data combination has completed the pre-processing and can directly proceed to the subsequent feature calculation, weighted fusion, and lip-sync image synthesis stages.

[0074] S106 relies on three-segment hierarchical supervised training, error code pairwise-attractor directional competition suppression, dynamic programming path selection, spike suppression, event-triggered stabilization, target code directional correction and temporal sequence path selection optimization and ROI local rendering algorithms to solve the mapping relationship between audio features, lip-sync discrete codes, inter-frame temporal states and lip-sync images. It combines candidate code confidence, cross-frame jump cost composed of emission cost, transition cost, and switching penalty, and template local features for weighted fusion to generate temporally stable and lip-sync accurate audio-driven lip-sync videos.

[0075] In one implementation, a combination of techniques including three-stage hierarchical training, error code-oriented competition suppression, dynamic programming path selection, spike suppression, event-triggered stabilization, target code-oriented correction, and ROI local rendering is used to deeply explore the intrinsic relationships between various techniques and audio features, discrete lip-sync codes, inter-frame temporal states, and lip-sync templates, conducting comprehensive feature mining and extraction. All feature extraction work is based on the output of the discrete code prediction model and frame state determination data. The extracted features will serve the entire workflow, including subsequent sequence optimization, logical operations, and image compositing.

[0076] The discrete code prediction model includes an audio encoder, a temporal smoothing module, and a vector quantizer. The audio encoder flattens the (20,256) features of each frame into 5120 dimensions, and then maps them into 16-dimensional continuous latent variables through a multilayer perceptron of 5120→512→256→128→16. The temporal smoothing module performs grouped one-dimensional convolution on the latent variable sequence to enhance short-term continuity. The vector quantizer contains a 512×16 codebook, selects the nearest code vector through Euclidean distance, and outputs a discrete lip-shape code index of 0~511 and the corresponding top-k candidates.

[0077] The original audio is sampled at 16kHz using WENet. After WENet encoding, speech activity timestamp alignment, and silence replacement, a frame-level audio feature of size (T, 20, 256) is generated. This feature is input into the audio encoder and then compressed through a multi-layer fully connected network to finally output a 16-dimensional continuous latent variable. The temporal smoothing module uses grouped one-dimensional convolution to optimize the temporal sequence of the latent variable, enhancing the continuity between frames. The vector quantizer has 512 built-in 16-dimensional codebooks. By calculating the Euclidean distance, it matches the latent variable with the codebook, and outputs multiple candidate discrete codes and corresponding weight values ​​for each frame according to confidence levels.

[0078] Candidate confidence features are extracted from the candidate weight data output by the model. These features directly reflect the credibility of each set of discrete lip-shape codes and serve as the basis for subsequent path selection and weight adjustment. Inter-frame transition features are extracted by combining the variation amplitude and switching frequency of discrete codes in adjacent frames. Simultaneously, abnormal glitches in single frames, double frames, and consecutive frames are identified using preset thresholds, generating abnormal glitch features. Target code matching features are extracted by combining the sorting position of the target code and the error code, the weight difference, and the distribution state of adjacent frames to determine whether the correction triggering condition is met. Template matching features are extracted based on the one-to-one binding relationship between discrete codes in the range of 0 to 511 and 512 standard lip-shape templates, providing a template index basis for the rendering process.

[0079] The candidate confidence features, inter-frame jump features, abnormal glitch features, target code matching features, and template matching features extracted in the previous step are summarized and organized. Dimensional merging and correlation analysis are carried out to integrate the scattered single features into a unified multi-dimensional audio-visual linkage feature set, realizing the transformation from independent features to a comprehensive state description. Various features are interconnected and work synergistically in actual operation. Candidate confidence features directly determine the calculation result of the internal cost of the dynamic programming algorithm; the higher the confidence level, the lower the corresponding computational cost. Inter-frame jump features and abnormal glitch features are the core judgment criteria for activating spike suppression and event-triggered stabilization functions. Target code matching features, combined with candidate confidence features, can comprehensively determine whether to activate target code directional correction. Template matching features directly connect to the template retrieval logic of the ROI local rendering stage.

[0080] During feature integration, single-frame confidence data and adjacent frame transition data are combined according to the temporal order of audio playback to comprehensively evaluate the overall stability of the local frame sequence. Simultaneously, the target code matching results are combined with candidate weight data to verify whether the correction conditions are met. Redundant information is eliminated during integration, and the linkage logic between different features is strengthened. The resulting audio-lip linkage feature set fully covers four dimensions: audio performance, lip shape state, temporal changes, and template information. It comprehensively describes the overall operational state of the current frame and surrounding sequences, providing complete data support for subsequent rule calculations.

[0081] Based on a multi-dimensional audio-mouth linkage feature set, and relying on a complete set of optimization techniques, an audio-mouth mapping solution rule, multi-feature weighted fusion logic, and a global frame order matching mechanism are constructed to generate a standardized computation batch containing audio feature vectors, lip-sync discrete code labels, frame state parameters, candidate cost parameters, and template attributes. The audio-mouth mapping solution uses a dynamic programming path selection algorithm as its core, and the entire computational cost is divided into three parts: transmission cost, transition cost, and handover penalty. The transmission cost is calculated based on candidate confidence features using a negative logarithmic operation; the higher the candidate code confidence value, the lower the corresponding transmission cost. The transition cost is calculated by combining inter-frame jump features and codebook embedding spatial distance, while also adding a handover penalty term between different code values ​​to constrain abnormal inter-frame jumps.

[0082] Multi-feature weighted fusion assigns corresponding weights to four types of features—candidate confidence, inter-frame transition, target matching, and template—based on business performance requirements, prioritizing lip-sync accuracy while also considering video smoothness. Global frame order matching strictly follows the original audio playback order, maintaining the frame sequence arrangement throughout to ensure consistent audio-visual timing.

[0083] Standardized computation batches are generated according to the above rules. Each batch uniformly contains five types of data: audio feature vectors, lip-shape discrete code labels, frame state parameters, candidate cost parameters, and template attributes. The audio features maintain the standard format of (T, 20, 256), and the lip-shape discrete code values ​​are fixed between 0 and 511. Frame state parameters are categorized into different types such as normal frames, switching frames, spike frames, and high-risk event areas. Candidate cost parameters are specific values ​​obtained through dynamic programming. Template attributes correspond to unique numbers for lip-shape templates. This unified batch format is adaptable to end-to-end algorithm computation, ensuring stable and orderly batch data processing.

[0084] The standardized computation batches are synchronized with the continuous frames of raw audio and video data through a spatiotemporal correlation mechanism. This process sequentially completes feature extraction, multi-cost weighted fusion, anomaly correction, and lip-sync synthesis, resulting in a stable and accurate audio-driven lip-sync video output. The spatiotemporal matching uses the original audio and video playback timeline as a benchmark, binding each standardized computation batch with the audio signal and video frame at the same time point to ensure complete temporal synchronization between the algorithm's computational data and the original media data.

[0085] Data processing is performed sequentially according to a fixed process. First, feature calculation is completed based on the sound-mouth linkage feature set and preset mapping rules to further optimize the output result of the single-frame lip-shape discrete code. Then, multi-cost weighted fusion is carried out by combining all cost parameters and feature data to solve the globally optimal lip-shape code combination for the entire sequence.

[0086] Sequence-level dynamic programming is performed on the top-k candidates frame by frame. The emission cost consists of the negative logarithm of the candidate weights and the deviation of the candidate code vector from the soft-fused target vector. The transition cost consists of the distance between adjacent candidate codes in the codebook embedding space and the code value switching penalty. The system solves for the globally optimal lip-code path by minimizing the cumulative cost, a method different from traditional local threshold judgment logic based on pixel offset. Specifically, frame cost = emission cost + transition cost: emission cost is -log(weight) + target vector deviation (codebook embedding distance); transition cost is the embedding distance between adjacent candidate codes + switching penalty. Default parameters are as follows. =0.35、 =0.55、 =0.12、 =5.

[0087] Subsequently, a layered anomaly correction operation is performed. When a single-frame or double-frame spike with a code value jump amplitude of 16 is detected, stable code values ​​from both sides are retrieved to complete the backfilling and repair. At the same time, a boundary protection mechanism is activated to avoid normal pronunciation switching areas and prevent erroneous correction. Within a window with a radius of 6 frames centered on the current frame, if the number of code value switching is greater than or equal to 4 and the jump amplitude reaches 56, it is determined to be a high-risk area. The control range is then expanded outward by 4 frames, and an enhanced stabilization strategy is activated to suppress continuous jitter. When performing target code directional correction, each of the seven preset conditions is strictly verified. Only when all conditions are met is the sequence squeezed out by the erroneous code replaced with the standard target code. No unfounded forced rewriting is performed.

[0088] Taking error code source=472 and target code target=45 as an example, the conditions and thresholds for triggering error correction are as follows: 1. The current frame output is That is, the top 1 value of the current frame is 472. 2. Those appearing in the top-k candidate list of the current frame, i.e., 45, are valid candidate codes for the current frame. 3. Its ranking in the top-k is less than 3. 4. The confidence weight is not less than 0.10. 5. and The weight difference is no greater than 0.35. 6. Taking the current frame t as the center, within the neighborhood [t±2], the top-k elements containing... The frame rate is no less than 2 frames. 7. Current The consecutive occurrence length is no more than 12 frames. When all 7 conditions above are met, a correction operation is performed: the output code indices[t] of the current frame is corrected to... That is, indices[t] = 45.

[0089] After anomaly correction, the corresponding ROI local lip-sync template is retrieved based on the template attributes. Local image fusion and audio-visual synthesis are then performed, preserving the lip-sync pattern and audio playback rhythm corresponding to normal pronunciation throughout the process. Once all processing is complete, a finished lip-sync video is obtained with highly matched lip movements and pronunciation, no noticeable jitter between frames, and smooth overall visuals.

[0090] like Figure 2 As shown, an audio-linked lip-sync video generation system based on a discrete code prediction model includes: The data acquisition module 201 is used to acquire associated data from audio, facial lip shape video, and the initial lip shape clustering code table, providing a complete raw data source for subsequent processing; The rule configuration module 202 is used to combine the requirements of lip expression accuracy, temporal continuity, training sample validity and rendering efficiency, and complete the three-stage sample confidence level division based on the top 1 of the teacher model, the discrimination margin and the transition mask. It configures differentiated soft and hard label supervision strategies, and sets error code orientation to attract and suppress rules, multi-condition triggering mechanism, temporal jump cost threshold and ROI rendering scheduling rules. The supervised dataset construction module 203 is used to perform hierarchical processing of the original sample labels, optimize the code value competition relationship, construct a supervised dataset for model adaptation, and extract frame-level audio semantic features, lip-shape discrete code sequences, candidate code confidence and sample quality evaluation information that have been encoded by WENet, completed silence replacement and have dimensions of (T,20,256). The strategy and parameter configuration module 204 is used to combine the complexity of pronunciation details, video segment length, inter-frame jitter and hardware computing resources to determine the discrete code prediction and timing optimization strategy. Based on the code value competition law, inter-frame state, abnormal glitch characteristics and rendering load, it sets the label supervision type, error code correction coefficient, window stability constraint and template caching, asynchronous prefetching and pipeline rendering related scheduling parameters. The spatiotemporal association matching module 205 is used to adaptively learn the association logic of audio features with lip-sync discrete codes, inter-frame code value switching, and ROI template matching based on the AudioEncoder+temporal smoothing+512×16 vector quantizer architecture. It completes the spatiotemporal association of audio feature sequences, 5 sets of top-k candidate discrete codes in the inference stage, inter-frame state information and local lip-sync templates to achieve the synchronization of the entire audio and video frame sequence. The lip-sync video generation module 206 is used to solve the mapping relationship between audio features, discrete lip-sync codes, inter-frame temporal states and lip-sync images by relying on three-segment hierarchical supervised training, error code directional competition suppression, dynamic programming path selection, spike suppression, event-triggered stabilization, target code directional correction and ROI local rendering technology. It combines candidate code confidence, cross-frame jump cost composed of emission cost, transition cost and switching penalty with template local features to complete weighted fusion, and finally generate temporally stable and accurate audio-driven lip-sync video.

[0091] A computing device includes a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute any audio-linked lip-sync video generation method based on a discrete code prediction model.

[0092] The methods and / or embodiments in this application can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by a processing unit, it performs the functions defined in the methods of this application.

[0093] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0094] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application.

Claims

1. A method for audio-visual lip-synching video generation based on discrete code prediction model, characterized in that, include: Acquire the associated data of audio, facial lip shape video, and initial lip shape clustering code table; Based on the teacher model top1, the discrimination margin and the transition mask, a three-stage sample confidence level division is implemented. A differentiated hard / soft label supervision strategy is provided, and error code orientation to attract and suppress rules, multi-condition triggering mechanism, temporal jump cost threshold and ROI rendering scheduling rules are set. Among them, top1 is the confidence score of the best candidate code output by the teacher model, which is used to measure the certainty of single frame lip shape prediction. The original sample labels are graded and the code value competition relationship is optimized. A supervised dataset is constructed to adapt to the training of the discrete code prediction model. The frame-level audio semantic features, lip-shape discrete code sequences, candidate code confidence and sample quality evaluation information of WENet encoded output, shape (T,20,256) with silence replacement are extracted. By combining the complexity of pronunciation details, video segment length, inter-frame jitter, and hardware computing resources, discrete code prediction and timing optimization strategies are determined. Based on the code value competition rules, inter-frame state changes, abnormal glitch characteristics, and rendering load, label supervision type, error code correction coefficient, window stability constraints, template caching, asynchronous prefetching, and pipeline rendering scheduling parameters are set. The discrete code prediction model, consisting of AudioEncoder + temporal smoothing + 512×16 vector quantizer, is adaptively trained according to the strategy. The model learns the mapping relationship between audio features and lip-sync discrete codes, the inter-frame code value switching logic and ROI template matching rules, and completes the spatiotemporal association between audio feature sequences, 5 sets of top-k candidate discrete codes retained during the inference stage, inter-frame state information and local lip-sync templates. Based on three-stage hierarchical supervised training, error code pairwise-attractor directional competition suppression, dynamic programming path selection, spike suppression, event-triggered stabilization, target code directional correction and temporal sequence path selection optimization and ROI local rendering algorithms, the mapping relationship between audio features, lip-sync discrete codes, inter-frame temporal states and lip-sync images is calculated. Combined with candidate code confidence, cross-frame jump cost composed of emission cost, transition cost, and switching penalty, and template local features for weighted fusion, a temporally stable and lip-sync accurate audio-driven lip-sync video is generated.

2. The audio-lingual lip-shaped video generation method based on a discrete code prediction model according to claim 1, characterized in that, Based on the teacher model top-1, discrimination margin, and transition mask, a three-stage sample confidence level classification is implemented, accompanied by a differentiated hard / soft label supervision strategy. Error code-oriented attraction and suppression rules, multi-condition triggering mechanisms, temporal transition cost thresholds, and ROI rendering scheduling rules are defined, including: The overall technical requirements are integrated by multi-dimensional constraint analysis techniques, including lip expression accuracy, temporal continuity, training sample validity, and rendering efficiency. Combined with the top 1 candidate weights, the margin between top 1 and top 2, and the transition mask output by the teacher model, three-segment sample confidence grading criteria, error code directional competition constraint indicators, inter-frame jump control thresholds, and ROI rendering link scheduling benchmark data are generated. The three-stage sample classification rules, error code orientation and attraction suppression logic, timing control standards and ROI rendering operation requirements are integrated and processed to establish a precise matching relationship between hierarchical supervision, error code management, timing constraints, rendering scheduling and multi-dimensional constraints, and generate a full-link rule-customized execution strategy. Based on the rule-customized execution strategy, the core configuration requirements are extracted. The sample confidence features consisting of top1, margin, and transition mask are used as the input dimension. The competition state between the target code and the error code is used as the core parameter. The temporal performance of the frame sequence is used as the judgment criterion. This enables the scientific setting of three sample levels, multi-condition triggering rules for error codes, temporal jump thresholds, and ROI rendering logic. Based on the requirements of lip-sync accuracy, image timing stability, and engineering operation efficiency, quantitative calibration and optimization adjustments were made to various rule parameters. The judgment boundaries and differentiated supervision methods for high / medium / low-quality samples, the triggering conditions and weight scaling parameters for error code suppression, the threshold ranges for various timing transitions, and the operation mechanism of ROI template caching / asynchronous prefetching / pipeline rendering were clarified, forming a complete end-to-end operation rule system.

3. The audio-linked lip-sync video generation method based on a discrete code prediction model according to claim 1, characterized in that, The original sample labels are graded and the code value competition relationship is optimized. A supervised dataset suitable for training the discrete code prediction model is constructed. Frame-level audio semantic features, lip-sync discrete code sequences, candidate code confidence, and sample quality assessment information are extracted from the WENet encoded output, the shape (T, 20, 256), and the silence replacement is completed. Based on the requirements of sample confidence difference, code value competition state, label adaptability and model training stability, the original lip shape sample labels are reconstructed and optimized for code value competition by combining the top 1, discrimination margin and transition mask output by the teacher model. Relying on the three-stage supervision rules and error code orientation de-attraction mechanism, a hierarchical label system, error code multi-condition control rules, sample quality rating standards and WENet audio feature extraction specification information are generated. Based on the architectural requirements of three-stage hierarchical supervision and error code orientation removal, the training data processing task is designed in a hierarchical and decoupled manner. The module division is divided into sample confidence judgment, label type adaptation, code value competition correction and multi-class feature extraction. The execution logic and data flow relationship of each module are clarified, and task execution information of label hierarchical, code value optimization and audio feature extraction are formed. Combining training sample utilization, lip-code prediction accuracy, error code suppression effect, and dataset standardization requirements, and based on sample scene characteristics, high / medium / low confidence classification criteria, soft and hard label usage rules, and weight allocation schemes are set according to a three-segment sample quantization threshold. At the same time, the error code triggering conditions, weight scaling range, and (T,20,256) frame-level audio semantic features, lip-code discrete codes, candidate codes extraction dimensions, judgment thresholds, and data output format information are clarified. The sample grading strategy, code value optimization scheme, and audio feature extraction rules are integrated into a unified whole. The training dataset construction control information includes a three-stage sample confidence grading rule, an error code orientation de-attracting execution process, WENet frame-level semantic features, and multi-dimensional feature extraction standards.

4. The audio-linked lip-sync video generation method based on a discrete code prediction model according to claim 1, characterized in that, By combining the complexity of pronunciation details, video segment length, inter-frame jitter, and hardware computing resources, a discrete code prediction and timing optimization strategy is determined. Based on the code value competition rules, inter-frame state changes, abnormal glitches, and rendering load, parameters such as label supervision type, error code correction coefficient, window stability constraints, template caching, asynchronous prefetching, and pipeline rendering scheduling are set, including: The complexity of pronunciation details, video segment length, inter-frame jitter status, and hardware computing power are organized and sorted out. Combined with the discrete code model architecture and timing control logic to match the model operation specifications, a pre-processing flow for discrete prediction and timing optimization is constructed. A multi-dimensional analysis framework is built based on audio pronunciation features, lip-code competition rules, inter-frame status and abnormal glitch features. Video segment attributes and rendering load data are integrated to establish a parameter linkage verification, abnormal status identification and hardware load dynamic adaptation mechanism. With the goal of improving the accuracy of lip-sync discrete code prediction and suppressing inter-frame jitter and abnormal glitches, this paper integrates information on pronunciation complexity, segment type, code value competition, timing anomalies and rendering load to build a system that links operation strategies with various configuration parameters. By identifying scene features, verifying code value running status, and evaluating hardware computing power load, suitable running conditions are identified, parameter conflict issues are avoided, and a general strategy solution for discrete code prediction and timing optimization that can be directly deployed is output. The determined optimization strategy is imported into the discrete code prediction and timing control module. Combined with inter-frame anomaly features including error code competition rules, single-frame / continuous spikes, tail jitter, and rendering operation pressure, the configuration of relevant parameters such as label supervision type, error code correction coefficient, timing window stability constraint, template caching, asynchronous prefetching, and pipeline rendering is completed in sequence to form an operation management and control scheme.

5. The audio-linked lip-sync video generation method based on a discrete code prediction model according to claim 4, characterized in that, Based on three-stage hierarchical supervised training, error code pairwise-attractor directional competition suppression, dynamic programming path selection, spike suppression, event-triggered stabilization, target code directional correction, temporal sequence path selection optimization, and ROI local rendering algorithms, this algorithm calculates the mapping relationship between audio features, discrete lip-sync codes, inter-frame temporal states, and lip-sync images. It then combines candidate code confidence, cross-frame transition cost (composed of emission cost, transition cost, and switching penalty), and template local features for weighted fusion to generate temporally stable and accurately lip-sync audio-driven lip-sync videos, including: We conduct pattern mining and feature extraction on the correlation characteristics of three-segment hierarchical supervised training, error code pairwise-attractor directional competition suppression, dynamic programming path selection, spike suppression, event-triggered stabilization, target code directional correction, and ROI local rendering technology with audio features, lip-sync discrete codes, inter-frame temporal states, and lip-sync templates, generating candidate confidence features, inter-frame transition features, abnormal glitch features, target code matching features, and template matching features. Dimensional integration and correlation analysis are performed on candidate confidence features, inter-frame jump features, abnormal glitch features, target code matching features, and template matching features to generate a multi-dimensional audio-visual linkage feature set; Based on a multi-dimensional audio-mouth linkage feature set, and relying on a complete set of optimization techniques, we construct audio-mouth mapping solution rules, multi-feature weighted fusion logic and global frame order matching mechanism to generate standardized operation batches containing audio feature vectors, lip shape discrete code labels, frame state parameters, candidate cost parameters and template attributes. The standardized computation batch and the continuous frame audio and video raw data are synchronized and matched through a spatiotemporal correlation mechanism. The entire process of feature calculation, multi-cost weighted fusion, anomaly correction, and lip-shape image synthesis is completed sequentially, and finally, audio-driven lip-shape video with stable timing and accurate lip shape is output.

6. An audio-linked lip-sync video generation system based on a discrete code prediction model, characterized in that, The system is used to execute executable instructions to perform the audio-linked lip-sync video generation method based on a discrete code prediction model as described in any one of claims 1 to 5.

7. An electronic device, characterized in that, include: First processor; And a memory for storing executable instructions of the first processor; wherein the first processor is configured to execute the audio-linked lip-sync video generation method based on a discrete code prediction model according to any one of claims 1 to 5 by executing the executable instructions.

8. A computing device, the device comprising a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein, When the computer program instructions are executed by the processor, the device is triggered to execute the audio-linked lip-sync video generation method based on the discrete code prediction model as described in any one of claims 1 to 5.

Images