A mouth shape retrieval model optimization method, system and device based on time sequence context modeling

Abstract

The application discloses a mouth shape retrieval model optimization method, system and device based on time sequence context modeling, which is applied to the technical field of data processing. Multi-frame audio feature coding is completed through a time sequence window, one-dimensional convolution and weighted fusion, so that audio representation with time sequence consistency is generated. A retrieval preference model is constructed in combination with historical retrieval data, time sequence smoothing constraints and state transition rules are formed. A retrieval sorting model is built relying on a multi-objective optimization function, candidate screening is completed by comprehensively matching multi-dimensional matching items and superimposing various types of penalty constraints. Meanwhile, boundary adaptive weight regulation and time sequence stable reasoning strategies are added, and dynamic optimization is performed for pronunciation boundary, high-risk and low-confidence scenarios. The whole scheme runs in a frame-by-frame streaming mode, upgrades traditional single-frame optimal retrieval to time sequence context constraint optimal retrieval, effectively solves the mouth shape jitter problem, and improves the overall time sequence stability and output naturalness.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, system and device for optimizing a lip-reading retrieval model based on temporal context modeling. Background Technology

[0002] Current technologies generally employ a single-frame audio independent retrieval approach, using the features of the current single-frame audio as the sole retrieval criterion to match the corresponding lip-sync entry in a preset material library or lip-sync codebook, directly completing the lip-sync output. Some existing solutions add post-processing steps such as posterior smoothing, optical flow correction, and simple filtering after the single-frame retrieval result is generated. A few offline solutions also use global optimization of the entire audio segment to achieve lip-sync matching. The above-mentioned implementation methods are the mainstream technical means for real-time lip-sync generation of digital humans, but they have several inherent technical defects in practical applications.

[0003] Single-frame retrieval relies solely on instantaneous audio features for matching, with adjacent frames making independent optimal selections. Multiple lip-sync candidates with similarity are easily selected alternately, leading to frequent lip-sync switching between frames and causing visual issues such as lip tremors and screen flickering. The limited pronunciation information carried by a single audio frame cannot fully express features related to articulation, syllable rhythm, and pronunciation stages, resulting in insufficient matching between lip-sync and actual pronunciation rhythm. Remedial solutions using post-smoothing and filtering are passive "error-first, correction-later" approaches, only mitigating surface anomalies and failing to prevent the selection of erroneous candidates at the retrieval source, thus offering poor root cause correction. While some solutions incorporate historical lip-sync states as constraints, they lack integration with audio context features, allowing historical errors to propagate and amplify the problem. Offline global optimization relies on large amounts of future audio data, resulting in high latency and making it unsuitable for real-time digital human scenarios with strict latency requirements, such as live streaming, real-time interaction, and embedded terminals. Summary of the Invention

[0004] To solve the above-mentioned technical problems, the present invention provides the following technical solution: An optimization method for lip-reading retrieval models based on temporal context modeling includes: constructing a temporal context modeling framework for lip-reading retrieval, extending single-frame audio features into temporal context representations, and establishing a correlation model between historical states and the current retrieval; encoding contextual features based on temporal windows, using temporal convolution and weighted fusion to perform temporal modeling on multi-frame audio features, extracting articulation coordination rules and temporal dependency features, and forming an audio representation with temporal consistency; designing a retrieval preference model that integrates historical states, encoding historical retrieval results, continuous states, and stability features into temporal priors, and constructing temporal smoothing constraints and state transition rules; and building a temporal context-aware retrieval ranking model that integrates single-frame matching, contextual... Consistency, historical continuity, and pronunciation stage adaptability are modeled as a multi-objective optimization function, incorporating jump penalties, confidence penalties, and risk constraints to optimize candidate ranking. A temporal boundary adaptive control mechanism is introduced to dynamically modulate model weights at pronunciation boundaries and high-risk retrieval points, achieving historical constraint decay and contextual preference enhancement to suppress error propagation across time. A temporally stable inference strategy is adopted to perform conservative selection and temporal smoothing correction on fuzzy candidates and low-confidence results, ensuring continuous and stable retrieval output in the temporal dimension. Through frame-by-frame streaming temporal modeling and iterative inference of the retrieval model, lip-reading retrieval is transformed from single-frame optimality to temporal contextual constraint optimality, improving the temporal stability and naturalness of the retrieval model.

[0005] A lip-reading retrieval model optimization system based on temporal context modeling is provided. The system is used to execute executable instructions to perform the aforementioned lip-reading retrieval model optimization method based on temporal context modeling.

[0006] Its beneficial effects are as follows: It abandons the traditional single-frame independent retrieval mode, completing multi-frame audio feature encoding through temporal windows, convolution, and weighted fusion to construct audio temporal representations; it relies on a FIFO circular buffer to parse historical lip-sync data, building a retrieval preference model and forming temporal constraint rules; it builds a retrieval ranking model based on a multi-objective optimization function, integrating multi-dimensional evaluation indicators and multiple penalty terms to complete candidate scoring and ranking; it also adds a temporal boundary adaptive adjustment mechanism, combining phoneme alignment and VAD detection to delineate boundaries and high-risk frames, dynamically adjusting model weights and configuring strategies such as gradual recovery and index rollback; finally, it uses temporally stable inference to conservatively select and smoothly correct fuzzy and low-confidence candidates. The entire architecture operates frame-by-frame streaming, upgrading single-frame optimal retrieval to temporally context-constrained optimal retrieval, and collaboratively improving the quality of lip-sync output across the entire chain.

[0007] This application fully leverages audio context and historical lip-sync states to address the issues of lip jitter and teeth flickering that often occur in traditional single-frame retrieval. It significantly improves the temporal coherence of lip-sync sequences, distinguishes between strict causal and short look-ahead dual-audio window modes, and caters to both low-latency real-time scenarios and high-definition offline scenarios, adapting to various terminal devices. Furthermore, it implements differentiated weight control and risk handling for syllable boundaries, pause boundaries, and high-risk frames, ensuring natural pronunciation transitions while preventing the propagation of abnormal states across frames. This constrains abnormal candidates from the retrieval source, supplemented by multi-level smoothing and fallback mechanisms, eliminating the need for additional post-processing corrections, reducing overall computational overhead, and balancing matching accuracy and visual effects. Attached Figure Description

[0008] Figure 1 A flowchart illustrating an optimization method for a lip-sync retrieval model based on temporal context modeling, provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a lip-reading retrieval model optimization system based on temporal context modeling, provided in an embodiment of the present invention. Detailed Implementation

[0009] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention. Figure 1 This paper describes a method for optimizing a lip-reading retrieval model based on temporal context modeling, according to an exemplary embodiment of this application.

[0010] In this application embodiment, a lip-sync retrieval model optimization method based on temporal context modeling is provided, such as... Figure 1 As shown: S101. Construct a temporal context modeling framework for lip-sync retrieval, extend single-frame audio features into temporal context representations, and establish a correlation model between historical states and the current retrieval.

[0011] In one implementation, this framework builds an overall operational architecture around real-time lip-sync driven services, divided into two main functional links: audio processing and historical state processing. These two links operate independently and in parallel, completing data interaction within each frame processing cycle and uniformly delivering basic data to backend business modules such as candidate retrieval, joint scoring, and risk control. The audio processing link corresponds to the system's audio window feature extraction module, undertaking the entire process of audio framing, local window construction, and multi-frame feature fusion. This link is equipped with a 4-frame capacity FIFO circular buffer, which only stores the extracted 80-dimensional spectral features and does not store the original audio stream file. Data is automatically updated according to frame-by-frame rules. When a new feature enters the end of the buffer, the oldest historical feature is automatically removed, eliminating the need for repeated computation of historical data and ensuring low overall system latency, adapting to real-time driven scenarios. The core function of this link is to perform temporal extension on the original single-frame audio, converting it into audio features with contextual information, and generating results that serve as the core judgment basis for lip-sync retrieval.

[0012] The historical state processing link corresponds to the system's historical video state reading module, equipped with a 5-frame first-in-first-out circular buffer. The buffer only stores the final output data for each round of the system, excluding intermediate candidate results. Specifically, it stores three types of data: a global index of the codebook, lip blendShape parameters, and tooth status identifiers. This link is responsible for reading the generated historical lip shape states, extracting temporal constraint information, and providing a continuous basis for lip shape retrieval in the current frame.

[0013] Two links follow a streaming-serial logic to synchronously process data frame by frame. Within each computation cycle, the audio processing link outputs a temporal audio representation, while the historical state processing link synchronously retrieves historical data from the buffer. Both types of data are simultaneously distributed outwards. The entire architecture does not perform offline global computations, perfectly matching the latency requirements of real-time output from the digital human. The audio processing link is used to convert single-frame audio features into temporal context representations. The original audio sampling rate is set to 16000Hz, the single frame duration is 25ms, the frame shift is 10ms, and Hamming windows are used for framing. After framing, Mel-spectral features are extracted frame by frame, with the feature dimension of a single frame fixed at 80 dimensions. Subsequently, global numerical scaling is performed to constrain the feature values ​​to the range of 0 to 1. The entire process does not use MFCC features.

[0014] The system supports two audio window working modes, which can be flexibly switched according to the usage scenario. The first is the strict causal mode, where the audio window consists of the current frame superimposed with the previous four historical audio frames, for a total of 5 frames and a duration of approximately 160ms. This mode does not call future audio and is the mainstream choice for real-time scenarios. The second is the near real-time short look-ahead mode, which adds 1 to 2 short future audio frames to the current frame and 4 historical frames, resulting in a total of 6 to 7 frames and a duration of approximately 200ms to 230ms. This mode can optimize the pronunciation transition effect but will slightly increase the running latency. All audio windows slide and update frame by frame according to the frame shift interval, while relying on a 4-frame first-in-first-out circular buffer to dynamically replace historical audio frames.

[0015] The multi-frame Mel-spectral feature fusion within a window offers three fusion methods, which can be flexibly selected based on device computing power and performance requirements. The first method is a combination of concatenation and one-dimensional convolution fusion, where all features within the window are concatenated in chronological order, and then the fusion result is obtained through convolution. The second method is mean pooling fusion, which calculates the average value of each dimension of features within the window to complete the integration. The third method is weighted summation fusion, where the weight of the current frame is set to 0.5, and historical frames are weighted according to an exponential decay rule. All three fusion methods ultimately output 8-dimensional features, fully preserving temporal information such as syllable rhythm and coarticulation, maintaining consistency in feature dimensions with the original single frame, and ensuring universality of upstream and downstream module interfaces.

[0016] A correlation model between historical states and current retrieval is built based on the historical state processing link. All data in the cache and the global mouth shape codebook system are strictly unified, and cross-codebook calls are not performed throughout the process. The tooth state adopts a binary classification identification rule, where 0 represents that the teeth are not visible and 1 represents that the teeth are visible. At the same time, the state is constrained by the mouth opening degree. When the mouth opening degree value is less than 0.15, it is forcibly determined that the teeth are not visible. When the mouth opening degree value is greater than 0.40, the teeth are allowed to be visible. When it is between the two, the tooth state of the previous frame is maintained to avoid frequent screen flickering.

[0017] For historical data within a 5-frame circular buffer, two methods are used to extract valid states. The first is a primary index voting method, which counts the frequency of occurrence of each codebook index in the buffer and selects the index with the most occurrences as the primary index for the current stage. The second is a continuous stability filtering method, which requires multiple conditions to be met simultaneously: the codebook index must appear consecutively for at least 3 frames, the lip-sync distance between frames must be less than 0.20, there must be no tooth flipping phenomenon, and the overall confidence level must be greater than 0.10. After all conditions are met, the data segment is determined to be a continuous and stable representative mode.

[0018] The project employs three methods to calculate inter-frame differences, generating historical continuity scores to constrain the current lip shape retrieval results. The first method is index distance, which compares the codebook index of the current candidate with that of historical frames to determine the overall lip shape switching amplitude; this is often used for quickly screening index jump issues. The second method is absolute mouth opening difference, comparing the numerical difference in lip opening between consecutive frames; this method has simple calculation logic and low computational overhead, making it a frequently used method. The third method is L2 Euclidean distance, which calculates the overall morphological difference by integrating all lip blendShape parameters, resulting in the highest calculation accuracy.

[0019] The continuity score is uniformly calculated based on the distance result. The smaller the difference distance between frames, the higher the continuity score, indicating a smoother transition between lip movements in consecutive frames. This continuity index is directly integrated into the joint retrieval scoring process, serving as a core constraint in candidate selection. The model ultimately outputs three types of data: historical continuity index, statistically derived master index, and selected stable representative patterns, which are pushed to the joint retrieval scoring module in real time. This association model changes the traditional single-frame independent retrieval mode, relying on multi-frame historical data to form temporal constraints, effectively suppressing problems such as frequent lip movement switching, abrupt changes in shape, and image jitter between frames, ensuring the visual coherence of the entire lip movement sequence.

[0020] S102, based on the temporal window, completes contextual feature encoding, and uses temporal convolution and weighted fusion to perform temporal modeling of multi-frame audio features, extracts pronunciation coordination rules and temporal dependency features, and forms an audio representation with temporal consistency.

[0021] In one implementation, based on the practical application scenario of digital phrasing retrieval, audio framing, local window construction, and multi-frame feature fusion are used as fundamental technical supports. Three core processing stages are identified: local audio window construction, first-in-first-out (FIFO) cache update, and multi-frame feature fusion, thus establishing a complete audio context encoding execution system. This system, as an independent functional unit, connects upstream to the front-end raw audio input and downstream to business modules such as candidate retrieval and joint retrieval scoring. The three processing stages within the system are sequentially connected and run serially, with clear division of labor and collaborative cooperation. Local audio window construction is responsible for defining the audio temporal processing range and completing the planning and construction of the audio window. FIFO cache update is responsible for dynamically maintaining historical audio data to ensure real-time data availability. Multi-frame feature fusion is responsible for extracting features from the audio data within the window to generate standardized temporal representations. The entire architecture is natively adapted to real-time streaming processing modes and can stably and continuously receive complete audio streams.

[0022] Audio framing follows a unified standard, with an original audio sampling rate of 16000Hz, a single frame duration of 25ms, and a frame shift of 10ms. Hamming windows are used for framing. After framing, 80-dimensional Mel-frequency features are extracted frame by frame, but MFCC features are not used in the operational process. The system distinguishes between two standard audio window modes, which can be selected according to on-site operational requirements. The strict causal mode consists of the current frame combined with four previous historical audio frames, totaling five frames and approximately 160ms in duration. This mode does not call future audio data and is suitable for high real-time scenarios. The near real-time short look-ahead mode adds one to two short future audio frames to the current frame and four historical frames, resulting in a total window of six to seven frames and a total duration of 200ms to 230ms. All audio windows slide and update frame by frame according to the frame shift interval.

[0023] The system is equipped with a fixed-length, first-in-first-out (FIFO) circular buffer with a capacity of 4 frames. It stores only the extracted 80-dimensional spectral features and does not retain the original audio data. The buffer follows a fixed update rule: as new audio features enter the end of the buffer, the oldest historical features are removed, maintaining the buffer's capacity. Historical audio frames are dynamically maintained using this buffer, and combined with a sliding window, the entire process avoids recalculating historical data, ensuring low-latency operation.

[0024] The multi-frame Mel-spectral features within a window support three fusion methods. The first is a temporal concatenation combined with one-dimensional convolution fusion, where all features within the window are concatenated chronologically and then integrated through convolution. The second is mean pooling fusion, which calculates the average value of each dimension of the features within the window to achieve data fusion. The third is weighted summation fusion, where the current frame has a weight of 0.5, and historical frames are weighted according to an exponential decay rule. All three fusion methods ultimately output 80-dimensional features, maintaining consistency with the single-frame feature dimensions and ensuring compatibility with upstream and downstream module interfaces.

[0025] The temporal audio representation generated through the entire process fully carries contextual information such as co-pronunciation and syllable timing, and is continuously output to the downstream candidate recall and joint retrieval scoring modules, providing standard audio basis for lip-sync matching and temporal constraint judgment, and ensuring the continuity of subsequent lip-sync output from the audio processing stage.

[0026] Following the processing specifications of complete temporal information, unified feature dimensions, and adaptation to real-time streaming inference, the three key pieces of information—frame-by-frame 80-dimensional Mel-spectrum features, audio window frame sequences, and fusion algorithm control parameters—are jointly matched to complete the alignment of the entire audio encoding process. The work is carried out in accordance with three core requirements: First, ensuring complete temporal information by fully capturing the features and temporal relationships of all frames within a local audio window, without losing contextual content such as pronunciation rhythm and co-location. Second, ensuring unified feature dimensions by maintaining fixed feature specifications throughout the process to avoid module integration anomalies caused by dimensional changes. Third, adapting to real-time streaming inference by designing all information processing and rule invocation according to a continuous frame-by-frame operating logic to meet the low-latency business requirements of digital humans.

[0027] Based on the above processing specifications, the core interactive information of the three types of synchronous access coding systems is determined. Each type of information has its own function and works together to complete the process alignment. Frame-by-frame 80-dimensional Mel-spectrum features are extracted, which are the basic input units for audio coding. The original audio is framed according to industry standards, with a sampling rate set to 16000Hz, using a 25ms frame length, a 10ms frame shift, and a Hamming window for framing processing. After framing, Mel-spectrum features are extracted frame by frame, with a uniform feature dimension of 80. Only Mel-spectrum features are used throughout the entire audio coding process; MFCC features are not used, thus unifying the basic data format from the source.

[0028] The audio window frame sequence, serving as a carrier of temporal information, is used to define the range of audio frames covered by a single operation. The system generates corresponding frame sequences in two operating modes: strict causality and short look-ahead. In strict causality mode, the frame sequence consists of the current frame superimposed with the previous four historical audio frames, covering a total of five audio frames. In short look-ahead mode, the frame sequence adds one to two future audio frames to the historical and current frames, corresponding to a frame range of six to seven frames. The frame sequence slides and updates frame by frame following the audio stream, always matching the local audio window division rules.

[0029] The fusion algorithm control parameters govern the overall operational logic of the multi-frame feature fusion process, specifically including algorithm selection, weight configuration, and computation rules. The system supports three fusion methods: one-dimensional convolution, mean pooling, and weighted summation. The corresponding algorithm selection parameters can be flexibly switched based on device computing power and performance requirements. In the weighted summation mode, the basic weights of the current frame are preset, and historical frames are assigned exponentially decaying weights. All computation rule parameters are pre-configured to ensure standardized execution of the fusion process.

[0030] Three types of core interactive information are synchronously integrated into the audio context encoding execution system, achieving end-to-end alignment. Frame-by-frame 80-dimensional Mel-spectrum features are continuously input into the system as raw computation data; the audio window frame sequence defines the selection range for each round of data, guiding the FIFO circular buffer to retrieve corresponding historical features; and the fusion algorithm control parameters are sent to the feature fusion stage, determining the current fusion logic and weighting standards. These three types of information cooperate and take effect synchronously, unifying the format and dimensions of all input data and standardizing the order and rules of each processing step. From audio framing and buffer retrieval to window assembly and feature fusion, the entire process is executed according to the aligned standards, eliminating execution differences between different frames and operating modes, ensuring the stable operation of the audio encoding chain.

[0031] After information matching and process alignment, the temporal representation format output by the audio window feature extraction module is fixed and follows unified rules, allowing seamless integration with downstream modules such as candidate recall and joint retrieval scoring. Standardized data flow effectively reduces data adaptation costs between modules and aligns with the real-time streaming architecture, providing standardized audio data support for subsequent lip-sync retrieval and temporal constraint calculations, thus ensuring the overall system's operational stability from the encoding stage.

[0032] Based on the execution requirements of accurate co-pronunciation representation, strong temporal consistency, and low latency, two window modes are distinguished: strict causality and short look-ahead. The number of window frames and feature normalization rules are set, and three fusion methods—one-dimensional convolution, mean pooling, and weighted summation—are used to complete the temporal modeling of multi-frame audio features. The system divides audio windows into two main types: strict causality real-time mode and quasi-real-time short look-ahead mode, which can be flexibly selected according to the on-site latency indicators. Specifically, the strict causality real-time mode is for high real-time business scenarios. The window selects 4 historical audio frames backward, superimposed on the current frame, for a total of 5 frames, corresponding to a duration of approximately 160ms. This mode does not call future audio content, fully meeting the causal constraints of real-time data streams, and is the mainstream choice for online deployment. The quasi-real-time short look-ahead mode is suitable for scenarios with higher latency tolerance. In addition to the 4 historical audio frames and the current frame, 1 to 2 future audio frames are added, resulting in a total of 6 to 7 frames in the window, corresponding to a duration of approximately 200ms to 230ms. By supplementing the pronunciation context with short-future audio, the representation of syllable transitions can be optimized, slightly improving the naturalness of lip-sync. Both window modes update frame by frame with a 10ms frame shift interval, and use a 4-frame first-in-first-out circular buffer to dynamically replace historical audio frames. There is no need to repeatedly calculate historical data throughout the process, ensuring stable latency.

[0033] All single-frame Mel spectrum features undergo uniform global numerical scaling. After normalization, the 80-dimensional Mel spectrum features are uniformly constrained to the range of 0 to 1. This processing method can smooth out the differences in feature values ​​across different time periods and pronunciation states, ensuring that the feature scale of all audio frames remains consistent and avoiding matching deviations caused by inconsistent numerical ranges. The entire normalization process is embedded in the audio framing and feature extraction stages, executed synchronously frame by frame, without adding extra processing time. Furthermore, this process does not use MFCC features throughout, uniformly employing Mel spectrum as the basic feature.

[0034] The system provides three standard feature fusion methods. All fusion schemes output a fixed 80-dimensional feature dimension, consistent with single-frame features, ensuring universality of upstream and downstream module interfaces and allowing flexible selection based on device computing power and performance requirements. The first method is one-dimensional convolutional fusion. It first concatenates all audio features within the window in chronological order, then integrates multi-frame information through a single-layer one-dimensional convolution operation. This method has strong capabilities for extracting temporally correlated features and can fully explore the patterns of co-pronunciation and temporal dependencies. The second method is mean pooling fusion. It calculates the average value of features in each dimension within the window. The computational logic is simple, and the computational power consumption is low, making it suitable for embedded and low-computing-power deployment environments. The third method is weighted summation fusion. Weights are set according to predetermined rules. The base weight for the current frame feature is 0.5, and historical frame features use exponentially decaying weights based on their time proximity. This method emphasizes the audio information of the current frame while also considering historical temporal content, achieving a balance between real-time performance and representational effect.

[0035] The normalized Mel-spectrum features frame by frame are input into the execution system. Historical audio frames are dynamically updated using a fixed-length FIFO circular buffer to construct local audio windows. Feature calculations are performed according to the selected fusion method, extracting articulation coordination patterns and temporal dependency features, and outputting a fixed-dimensional audio representation with temporal consistency. The frame-by-frame Mel-spectrum features, after global numerical scaling, are then integrated into the audio context encoding execution system. Prior to this, the original audio has undergone framing, feature extraction, and normalization operations according to the standard audio processing workflow. The values ​​of all single-frame Mel-spectrum features are uniformly constrained to the range of 0 to 1, with a feature dimension of 80, and MFCC features are not used throughout the process. The normalized features serve as the standard input data for this stage, being fed into the execution system frame by frame in the form of a continuous audio stream, providing basic data support for subsequent buffer loading and window construction.

[0036] The system is configured with a fixed-capacity, first-in-first-out (FIFO) circular cache. The standard cache storage capacity is 4 frames, used only to store normalized 80-dimensional Mel-spectrum features, and not the original audio files. The cache follows a fixed update operation rule: when a new frame of audio features enters the system, it is directly stored at the end of the cache. Simultaneously, the cache automatically removes the oldest historical audio features, always maintaining the internal data volume at a fixed capacity of 4 frames. This update method eliminates the need for repeated audio framing and feature extraction operations, significantly reducing redundant computation and ensuring low latency across the entire link from a data scheduling perspective, meeting the latency requirements of real-time driven scenarios. Cache data is updated frame-by-frame synchronously with the audio stream, maintaining the real-time validity of the data.

[0037] Within each audio processing cycle, four frames of historical audio features stored in the cache are retrieved and combined with the latest frame features to form a local audio window. This configuration corresponds to the basic frame structure of the system's two main operating modes. In strict causal mode, this window is the complete computation window, with a total of five frames and an overall duration of approximately 160ms. In near real-time short look-ahead mode, one to two future audio frames can be added to expand the window structure to six to seven frames. The window slides and updates frame by frame as the audio frame shifts, always changing synchronously with the audio stream, ensuring that each set of window data corresponds to the current local pronunciation segment.

[0038] After the local audio window is constructed, the previously selected feature fusion method is invoked for computation. The system supports three compliant schemes: one-dimensional convolutional fusion, mean pooling fusion, and weighted summation fusion, which can be flexibly selected according to the device's computing power and effect requirements. When using one-dimensional convolutional fusion, all features within the window are first sequentially concatenated according to time order, and then multi-frame information is integrated through a single-layer one-dimensional convolution. When using mean pooling fusion, the average value of each dimension of features within the window is calculated to achieve information extraction. When using weighted summation fusion, the current frame is set with a base weight of 0.5, and each historical frame is assigned an exponentially decaying weight according to its time proximity. Through the above fusion operations, the system fully explores the co-pronunciation correlations and syllable temporal change patterns between frames, extracting the inherent temporal dependency features of the audio.

[0039] After fusion calculation, a unified 80-dimensional audio representation is output, with the output feature dimensions completely consistent with the input single-frame Mel spectrum features. This unified feature specification ensures that the audio window feature extraction module has a universal interface with the downstream candidate recall module and joint retrieval scoring module, eliminating the need for additional format adaptation. This output representation fully carries key information such as local context and coarticulation, and as standard audio data, it continuously flows to downstream business modules, participating in audio similarity matching, temporal consistency evaluation, and other tasks, providing support for the continuity and naturalness of lip-sync sequences from the audio processing stage.

[0040] By combining the output window features of the system's overall streaming operation logic, contextual basis is provided for subsequent lip-shape candidate retrieval and joint scoring, generating audio encoding results adapted to digital lip-shape stability-driven retrieval. The module uniformly outputs an 80-dimensional temporal audio representation, with feature dimensions consistent with single-frame Mel-spectrum features, and the interfaces between upstream and downstream modules are universal. This representation integrates the effective information of all audio frames within a local audio window, fully carrying three core contents: local audio context, coarticulation rules, and syllable rhythm. This data replaces traditional single-frame audio features, compensating for the shortcomings of one-sided single-frame information, and serves as the foundational data for stable lip-shape retrieval.

[0041] The temporal audio representation is directly input into the candidate recall module as the core input for similarity matching. This step uniformly uses cosine similarity to calculate the matching between the audio representation and the audio features built into the lip-sync codebook. The number of candidate entries ranges from 5 to 10, with 8 fixed in standard real-time scenarios and 5 fixed in low-computing-power embedded scenarios. A similarity filtering threshold is also set, retaining only candidate entries with a similarity greater than or equal to 0.5; if the number of valid entries is insufficient, all compliant results are retained. This method relies on temporal features to filter candidates, reducing the probability of abnormal lip-sync switching at the source.

[0042] The temporal audio representation is synchronously fed into the joint retrieval and scoring module, participating in the calculation of two core scores. The first is used to calculate the current audio matching score, measuring the degree of fit between a single frame of audio and the candidate lip movements; the second is used to calculate the audio window consistency score, judging the matching effect between the candidate and the entire audio context. Both scores are uniformly participated in the multi-objective optimization function operation, combined with preset weights to complete the comprehensive score calculation, and the evaluation is carried out by comprehensively considering local pronunciation features, so that the scoring results are more in line with the actual pronunciation rules.

[0043] The audio encoding results generated in this step are used throughout the entire lip-reading retrieval chain, completely replacing traditional single-frame audio features in the entire process. Traditional single-frame retrieval relies solely on instantaneous audio data to filter lip movements, which is prone to repeated candidate switching, causing problems such as lip-reading jitter and screen flickering. This solution uses temporal audio representations that include contextual information, incorporating local pronunciation constraints into the retrieval judgment scope, avoiding the shortcomings of single-frame retrieval from the data source, and ultimately ensuring that the digital lip-reading output is coherent and natural.

[0044] S103, design a retrieval preference model that integrates historical states, encodes historical retrieval results, continuous states and stability features as temporal priors, and constructs temporal smoothing constraints and state transition rules.

[0045] In one implementation, the system employs a 5-frame capacity FIFO circular cache. This cache is used solely to store the final output data of the digital human frame by frame, without recording intermediate candidate data generated during the retrieval process, ensuring the authenticity and validity of historical state data. The cache uniformly stores three types of standard data: a global codebook index, 20-dimensional lip blendShape parameters, and tooth status identifiers. The tooth status uses a binary definition: a value of 0 represents invisible teeth, and a value of 1 represents visible teeth. All data in the cache fully matches the standard 512-class lip shape codebook index system. Cross-codebook data retrieval is prohibited during system operation to ensure the consistency of the index, lip shape parameters, and status identifiers. The cache is updated frame by frame according to the FIFO rule. When new final output data is written to the cache, the oldest historical data is automatically removed, maintaining a fixed capacity of 5 frames to adapt to the overall streaming operation mode of the system.

[0046] For all codebook indices within a 5-frame buffer, a primary index voting rule is executed. The frequency of each codebook index within the 5-frame range is counted sequentially, and the index with the highest frequency is selected and defined as the primary index for the current stage. This rule has low computational overhead and fast response speed, quickly summarizing recent mainstream lip-shape states. The extracted results reflect short-term lip-shape change trends and serve as a fundamental method for historical state extraction.

[0047] For historical data within a 5-frame buffer, a continuous stability filtering rule is applied. A pattern can only be considered a stable representative pattern if it meets several stringent conditions: First, the number of consecutive occurrences of the corresponding codebook index must be no less than 3 frames; second, the lip-shape distance between frames must be less than 0.20; third, there must be no tooth state flipping within the data interval; and fourth, there must be no significant lip-shape jumps throughout the entire data. Historical data that meets all of these conditions is extracted as a stable representative pattern. This rule focuses on filtering reliable lip-shape states over a long period with low fluctuations and is often used in retrieval scenarios requiring high stability.

[0048] The main index, stable representative pattern, accompanying mouth shape parameters, and tooth state information obtained through two methods—main index voting and continuous stable filtering—are integrated and uniformly encoded to generate temporal prior features. These features fully encapsulate the variation patterns and stability characteristics of mouth shapes across multiple frames, centrally summarizing temporal-related information. The encoded temporal prior features directly serve as the basic input data for the retrieval preference model, facilitating data flow between modules.

[0049] The main structure of the retrieval preference model is built based on temporal prior features. This model is an independent functional unit specifically designed to handle various historical state-related data. Upstream, the model receives temporal prior features as input, and downstream, it directly connects to the joint retrieval scoring module, synchronously transmitting the temporal constraint information corresponding to historical states to the retrieval scoring stage, thus achieving linkage between historical states and the current retrieval logic. The entire structure adapts to the system's frame-by-frame streaming mode, with data continuously flowing in a fixed order.

[0050] The model employs three standard methods to calculate historical continuity scores, which can be flexibly selected based on device computing power and accuracy requirements. The first method is index distance calculation, comparing the current candidate codebook index with the historically output codebook index to determine the transition amplitude of lip shape states. This method has low computational overhead and is suitable for quickly completing large-scale state screening. The second method is absolute mouth opening difference calculation, comparing the lip opening values ​​of the current frame and historical frames, using the numerical difference to measure the smoothness of lip shape transitions; this is the mainstream calculation method used daily by the system. The third method is L2 lip shape distance calculation, integrating all lip blendShape parameters to calculate the overall morphological difference. This method has the highest calculation accuracy and is suitable for scenarios with high requirements for output detail stability. The distance results obtained from the three methods are uniformly used in the continuity score conversion; the smaller the distance value, the higher the corresponding historical continuity score, representing a smoother transition between lip shapes.

[0051] Fixed risk thresholds are configured according to technical standards, corresponding to three types of abnormal scenarios: index jumps, tooth flips, and mouth deviations. The calculation results are used to distinguish between normal and abnormal states. For index jump scenarios, if the codebook index jumps consecutively two or more times, it is judged as an index jump anomaly. For tooth flip scenarios, tooth status uses a binary classification label: 0 represents invisible teeth, and 1 represents visible teeth. A change in tooth label between adjacent frames is judged as a tooth flip anomaly. For mouth deviation scenarios, a mouth opening change threshold of 0.35 is set. If the difference in mouth opening between frames exceeds this value, it is judged as a mouth deviation anomaly. Simultaneously, a mouth shape distance of 0.20 is uniformly used as the basic stability judgment benchmark. When the overall mouth shape distance is greater than this value, it is also included in the abnormal range. Historical continuity scores, various distance calculation results, and corresponding thresholds are compared one by one to complete frame-level state classification, distinguishing between normal and various abnormal states. The state judgment results will serve as the core basis for subsequent penalty calculations and strategy adjustments.

[0052] The retrieval preference model ontology, temporal prior features, historical continuity scores, and multiple risk verification parameters are integrated and bound together, clarifying the source, function, and supporting standards for each type of content. The retrieval preference model serves as the core computational carrier, uniformly accepting all inputs and rule configurations. Temporal prior features are derived from a 5-frame FIFO circular buffer, generated through two rule parsing methods: main index voting and continuous stable filtering. These features include temporal information such as historical codebook indexes, lip blendShape parameters, and tooth states, serving as the model's fundamental input data.

[0053] Historical continuity score is calculated using three methods: index distance, absolute difference in mouth opening, and L2 mouth shape distance. This score quantifies the smoothness of the connection between the current candidate and historical mouth shapes. A complete set of risk verification parameters and thresholds are simultaneously bound to three scenarios: index jump, tooth flip, and mouth opening offset. Index jump is judged based on two consecutive jumps; tooth status uses a binary classification: 0 represents invisible teeth, and 1 represents visible teeth; a change in the classification between adjacent frames indicates tooth flip; mouth opening offset uses a difference in mouth opening between frames of 0.35 as the critical threshold; and a mouth shape distance threshold of 0.20 is set as a general evaluation criterion for overall mouth shape stability. After all elements are bound, a unified system is formed within the model, with data, algorithms, and thresholds corresponding one-to-one. During computation, the corresponding parameters can be directly called without additional external configuration.

[0054] Based on the system's streaming operation characteristics, a fixed, serial data flow sequence is defined, with the output of each step serving as the input for the next, ensuring a seamless and orderly process. The first step involves importing prior temporal features. The parsed historical temporal features are input into the model. These features comprehensively record the mainstream states, stable patterns, and morphological change patterns of recent lip shapes, serving as a prerequisite for continuous calculation. The second step involves calculating historical continuity scores. The model calls upon three built-in distance calculation rules, combined with the imported prior temporal features, to compare the differences between the current candidate lip shape and historical lip shapes. Following a unified conversion rule, the smaller the inter-frame distance value, the higher the calculated historical continuity score, indicating a smoother lip shape transition. The three calculation methods can be flexibly selected based on computing power and accuracy requirements. Indexed distance calculation is fast, absolute mouth opening difference is a common mainstream algorithm, and L2 lip shape distance calculation has the highest accuracy. The third step involves performing frame-level state discrimination. The calculated historical continuity scores and various distance results are verified against pre-defined risk thresholds. Combining multiple rules such as index jump, tooth flip, mouth opening offset, and overall mouth shape distance, the current frame is divided into a normal state or the corresponding abnormal state, and the abnormal type is marked. The entire process is executed frame by frame, perfectly matching the overall streaming architecture of the system, and the next frame is processed immediately after the processing of a single frame is completed.

[0055] The model integrates feature parsing, distance calculation, and risk assessment rules into a single retrieval preference model, achieving deep fusion of data, algorithms, and validation rules. Feature parsing rules reuse the main index voting and continuous stable filtering logic to ensure input features conform to historical state definitions. Distance calculation rules fully retain the execution logic of the three calculation methods, differentiating between different usage scenarios. Risk assessment rules are applied in layers: first, the overall mouth shape stability is determined using a mouth shape distance of 0.20, then three sub-categories of anomalies—index jumps, tooth flips, and mouth opening deviations—are verified separately. When multiple anomalies occur simultaneously, the model judges them sequentially according to their risk impact, with each rule complementing the others without conflict. All calculations and judgments can be completed independently within the model, without calling external modules or temporarily adding parameters, ensuring the model's operational logic is self-consistent. The entire integrated set of rules works stably for both regular pronunciation frames and boundary frames, adapting to all retrieval needs across various scenarios.

[0056] By integrating various features and verification rules based on a retrieval preference model, a temporal smoothing constraint and state transition rule are ultimately constructed. The retrieval preference model, with its parameter fusion completed, serves as a unified carrier, integrating and aggregating multiple types of data, algorithms, and decision rules. The model internally incorporates pre-output temporal prior features and historical continuity results calculated using three methods, while also loading a complete set of risk verification rules and corresponding state judgment conditions. The temporal prior features are taken from a 5-frame FIFO circular buffer and parsed using two methods: main index voting and continuous stable filtering. They include complete historical information such as the codebook global index, lip blendShape parameters, and tooth states. Historical continuity results are divided into three categories: index distance, absolute difference in mouth opening, and L2 Euclidean distance, and a continuity score is generated according to a fixed conversion logic. The risk verification rules are accompanied by clearly defined judgment thresholds, including a threshold for the number of consecutive jumps in the codebook index, a tooth state flip judgment standard, an inter-frame mouth opening change threshold, and a general mouth shape distance threshold.

[0057] Based on historical continuity scores within the normal range, a temporal smoothing constraint effective across the entire frame is established. The core objective of this constraint is to limit drastic changes in lip shape and tooth status between adjacent frames, ensuring temporal continuity and visual naturalness throughout the entire lip sequence. The normal range is defined by two core thresholds: a lip shape distance of 0.20 and a mouth opening variation of 0.35. When the lip shape distance between frames is less than 0.20 and the mouth opening variation between frames is less than 0.35, it is considered a normal continuous state. In this state, the temporal smoothing constraint is activated, setting upper limits on changes in the overall lip shape, mouth opening amplitude, and lip expansion and rounding movements of adjacent frames, while also constraining the switching behavior of tooth status. A binary classification standard is used for tooth status: 0 represents invisible, and 1 represents visible. In non-syllable, non-pause ordinary frame ranges, arbitrary switching of tooth status is strictly restricted. This is linked to the mouth opening rule constraint: when the mouth opening value is less than 0.15, the teeth are forcibly kept invisible; when the mouth opening value is greater than 0.40, the teeth are allowed to be visible; the intermediate value range continues the tooth status of the previous frame, eliminating irregular flickering. The entire set of smoothing constraints covers all regular audio frames, using historical continuous states as a benchmark to mitigate random fluctuations caused by single-frame retrieval.

[0058] By combining various anomaly detection thresholds, different state intervals are defined, and standardized state transition rules are established to clarify the switching conditions, switching ranges, and behavioral restrictions between normal states and various abnormal states. First, the anomaly detection criteria are defined: two or more consecutive jumps in the codebook index, changes in tooth identification in adjacent frames, a difference in mouth opening between frames exceeding 0.35, or an overall mouth shape distance greater than 0.20; any one of these conditions constitutes an abnormal state. For the transition from normal to abnormal states, the system sets strong restrictions: significant jumps in mouth shape index, lip shape, and tooth state are prohibited in frames that are not syllable boundaries or pause boundaries; only at legal boundary locations such as phoneme switching and speech pauses are the switching range appropriately relaxed, allowing reasonable changes in mouth shape according to pronunciation rules. For the rollback from abnormal to normal states, the rules require a smooth, step-by-step transition, disallowing instantaneous restoration of the form. Priority is given to referencing the main index within a 5-frame buffer and continuous stable representative patterns to gradually correct mouth shape parameters and tooth state, returning to a stable state in a gradual manner to avoid secondary abrupt changes during the repair process. At the same time, differentiated migration requirements are set for different types of anomalies. For index jump anomalies, historical high-frequency indexes are locked first. For mouth offset anomalies, the mouth opening range is gradually narrowed. For tooth flip anomalies, the exposed state is corrected step by step to ensure that each type of anomaly has corresponding switching control logic.

[0059] The established temporal smoothing constraints and state transition rules are embedded throughout the two core stages of candidate scoring and candidate ranking, permeating the entire lip-reading retrieval process. In the candidate scoring stage, these two types of rules serve as crucial constraints in the multi-objective optimization function. For candidate entries that violate smoothing requirements or pose anomaly transition risks, corresponding penalty terms are applied to lower the overall score and reduce the probability of selection. In the candidate ranking stage, the rule execution results serve as an auxiliary ranking criterion. When overall scores are similar, candidate entries that meet temporal smoothing requirements and comply with state transition regulations are prioritized. This entire set of rules controls the process from the retrieval source, proactively intercepting candidate content that is prone to causing lip-reading jitter, morphological abrupt changes, and frequent tooth rotation, eliminating the need for secondary corrections after output and fundamentally improving the stability and naturalness of digital lip-reading output.

[0060] S104. Build a temporal context-aware retrieval and ranking model, unifying single-frame matching, contextual consistency, historical continuity, and pronunciation stage adaptability into a multi-objective optimization function, and adding jump penalties, confidence penalties, and risk constraints to complete candidate ranking optimization.

[0061] In one implementation, a temporal context-aware computational system is built by combining Top-K lip-sync candidates retrieved from the codebook, audio window fusion features, historical continuity features, pronunciation stage labels, syllable boundary markers, as well as cosine similarity calculation rules and score normalization rules. The computational system has five types of fixed input data, each with clearly defined generation rules, screening criteria, and classification basis. The first type is the Top-K lip-sync candidates retrieved from the codebook. The system differentiates the number of candidates based on the deployment scenario: 8 items are fixed for standard real-time scenarios, 5 items for low-computing-power embedded scenarios, and 10 items for offline optimization scenarios. All candidate items need to undergo similarity screening, retaining only valid content with a similarity greater than or equal to 0.5; items with insufficient matching are directly discarded.

[0062] The second category is audio window fusion features. These features are derived from 80-dimensional Mel-spectrum features, processed through three fusion methods: one-dimensional convolution, mean pooling, and weighted summation. The output dimension remains consistent, fully preserving audio context and coarticulation information. The third category is historical continuity features. This data is generated based on codebook indices, lip blendShape parameters, and tooth states within a 5-frame FIFO circular buffer, used to characterize the smoothness of lip movements between consecutive frames. The fourth category is articulation stage labels. These are uniformly divided into three categories—start, core, and end—according to the articulation stage of the current audio. The fifth category is syllable boundary identifiers. Dedicated identifiers distinguish syllable boundaries, pause boundaries, and regular audio frames, providing a basis for subsequent weight adjustments and state switching.

[0063] The computational system incorporates two fundamental calculation rules, serving as a unified standard for the calculation and comparison of all-dimensional indicators. The first rule is the cosine similarity calculation rule. This rule calculates the degree of matching between the audio window fusion features and the built-in audio features in the codebook. The result ranges from 0 to 1, with values ​​closer to 1 indicating better matching. This rule is the core basis for candidate recall and audio matching. The second rule is the score normalization rule. All similarity, continuity, and fit calculation results within the system are uniformly mapped to a fixed numerical range of 0 to 1. After normalization, indicators of different types and calculation logics can be directly compared horizontally, creating conditions for multi-objective joint computation.

[0064] By integrating five types of input data with two types of computational rules, a temporal context-aware computational system was formally established. This system serves as an independent data scheduling and computational carrier, simultaneously connecting upstream to the audio window feature extraction module and the historical video state reading module, continuously receiving streaming audio fusion features, historical continuity features, and other data. Downstream, it directly connects to the retrieval and ranking model, outputting well-organized data and standard computational logic. The entire system adopts a frame-by-frame streaming operation mode, with data input, computation, and transmission frame by frame, without performing offline batch computations. This aligns with the low-latency requirements of real-time digital human operation. The internal data flow order is fixed, computational rules are automatically invoked, and overall operation is stable and orderly.

[0065] A retrieval and ranking model is built upon this computational system. Single-frame audio matching, audio window consistency, historical video continuity, pronunciation stage adaptation, and boundary preferences are unified into a multi-objective optimization function, simultaneously introducing three types of constraints: lip abrupt change penalty, tooth state abrupt change penalty, and risk penalty. The retrieval and ranking model is directly built upon the pre-existing temporal context-aware computational system. This system provides the model with a complete set of standardized input data, including Top-K candidates from the codebook, 80-dimensional audio window fusion features, historical continuity features, pronunciation stage labels, and syllable boundary markers, along with two general calculation rules: cosine similarity and score normalization.

[0066] The retrieval and ranking model is an independent computation module. Upstream, it connects to a time-series context-aware computation system, while downstream, it handles subsequent steps such as boundary reset and risk control. The entire model operates in a frame-by-frame streaming manner, scoring and ranking each candidate item one by one. The core of the model is a multi-objective optimization function, which includes five evaluation dimensions. The calculation results of each dimension are uniformly normalized to the interval between 0 and 1, and are configured with fixed computational weights.

[0067] Cosine similarity is used to calculate the matching degree between the current frame's audio features and the candidate audio features in the codebook. The result ranges from 0 to 1, with higher values ​​indicating better fit. This dimension has a weight of 1.0, primarily measuring the basic matching accuracy between the current audio and the corresponding lip shape, and is the core basis for scoring. Average similarity is calculated for features across multiple frames within a local audio window to evaluate the adaptation effect of the candidate lip shape with the overall audio context and co-pronunciation; this dimension has a weight of 0.7. This dimension breaks through the limitations of a single frame, considering the overall syllable rhythm matching degree. The inter-frame difference between the current candidate and historical lip shapes is calculated using index distance, absolute difference in mouth opening, and L2 lip shape distance. This difference is converted into a continuity score according to the rule that smaller distances result in higher scores; this dimension has a weight of 0.8. This dimension is used to constrain the range of lip shape changes between consecutive frames, ensuring a smooth temporal transition.

[0068] The system combines the three categories of pronunciation stage labels (start, core, and end) from the codebook annotations to perform matching, determining whether the candidate lip shape matches the current vocalization action. A weight of 0.4 is set to ensure consistency between lip shape and pronunciation rules. Scenes are distinguished based on strong syllable boundaries and weak pause boundaries. Lip shape switching restrictions are appropriately relaxed at boundary positions, while constraints are tightened in ordinary audio ranges. This dimension has a weight of 0.5 to balance pronunciation naturalness and overall stability. The five dimensions are weighted and summed according to their respective weights to generate a basic comprehensive score for the candidate entries.

[0069] Three types of penalty constraints are superimposed on the base score. The results of the penalty items are also normalized to the range of 0 to 1, corresponding to fixed weights, to suppress various abnormal lip movements. The weight of the lip jump penalty is set to 1.2. A change in mouth opening of 0.25 between frames is used as the judgment threshold. When the difference in mouth opening exceeds this value, it is judged as a large lip jump and a full penalty is triggered to limit drastic changes in lip shape and avoid lip tremors.

[0070] The weight of the penalty for abrupt changes in tooth state is set to 1.5. Tooth state uses a binary classification: 0 represents invisible teeth, and 1 represents visible teeth. When a tooth state change occurs at a non-audio boundary, a full penalty of 1.0 is triggered; when a state change occurs at a syllable or pause boundary, a half penalty of 0.5 is applied, effectively preventing frequent tooth flickering. The weight of the risk penalty is set to 1.0. Two judgment conditions are set: when the difference between the highest and second-highest candidate scores is less than 0.08, it indicates low candidate discrimination and ambiguous selection; when the highest score of a single candidate is below 0.30, it indicates insufficient overall matching confidence. Meeting either condition triggers the risk penalty, suppressing the scores of abnormal candidates.

[0071] The optimization function, standard weight parameters, penalty thresholds, and risk assessment thresholds are linked and bound to complete the integrated configuration of model parameters and computational logic. The standard weight parameters balance the proportion of different evaluation dimensions and penalty items in the overall score. The model assigns fixed standard weights to the five evaluation dimensions and three types of penalty items. The weights for the five evaluation dimensions are: single-frame audio matching weight 1.0, audio window consistency weight 0.7, historical video continuity weight 0.8, pronunciation stage adaptation weight 0.4, and boundary preference weight 0.5. The weights for the three types of penalty items are: lip abrupt change penalty weight 1.2, tooth state abrupt change penalty weight 1.5, and risk penalty weight 1.0. This entire weight system remains fixed throughout the process, serving as the core basis for weighted calculations and ensuring consistency in scoring across different candidate items.

[0072] Regarding penalty thresholds, these thresholds define the critical range for abnormal changes in lip shape and tooth status, serving as the basis for determining when to trigger penalties. The threshold for lip jumps is 0.25; changes in mouth opening between frames exceeding this standard are considered significant lip changes, triggering a lip jump penalty. Tooth status is distinguished by 0 representing invisible and 1 representing visible. Judgment rules are set based on audio frame type; minor transitions are allowed in boundary frames other than ordinary audio frames, while state transitions in ordinary frames trigger abrupt tooth status penalties. These two types of thresholds clearly define the boundaries for penalty activation, preventing normal mouth shape changes from being incorrectly constrained.

[0073] Regarding risk assessment thresholds, these thresholds are used to identify high-risk retrieval scenarios, primarily setting critical standards for two scenarios: candidate score difference and overall matching confidence. The candidate score difference threshold is 0.08; when the difference between the highest and second-highest scores is less than this value, it is judged as low candidate discrimination and ambiguous selection. The overall confidence threshold is 0.30; when the highest overall candidate score is lower than this value, it is judged as poor overall matching effect and insufficient retrieval reliability. Meeting either of these conditions constitutes a high-risk scenario and triggers risk penalties.

[0074] The multi-objective optimization function, all weights, and two-level thresholds are comprehensively linked and bound together to determine a fixed order of data computation, with the entire process executed serially. First, based on the multi-objective optimization function, individual scores for the five evaluation dimensions are calculated, with all scores uniformly mapped to a numerical range of 0 to 1 to ensure horizontal comparison of results across different dimensions. Second, weighted calculations are performed on the scores of the five dimensions using corresponding standard weights to generate a basic comprehensive score without accumulated penalties. Third, penalty thresholds and risk assessment thresholds are retrieved, and the lip changes, tooth condition, and score corresponding to the current candidate are verified item by item to determine whether the corresponding penalty item needs to be activated. Fourth, the corresponding values ​​for triggered penalty items are calculated, and the basic comprehensive score is adjusted based on the penalty item weights to finally obtain the complete comprehensive score for each candidate.

[0075] Based on the configured ranking model, a comprehensive score is calculated for all candidate items and they are sorted in descending order, ultimately optimizing the candidate item ranking. The candidate score calculation and ranking belong to the joint retrieval scoring module, which receives the output results from the candidate recall module. Based on the retrieval ranking model with integrated parameters and rules, the comprehensive score calculation and priority ranking of all valid candidates are completed.

[0076] The pre-process instructions and input data specifications are the core intermediate links in the lip-shape retrieval chain. Upstream, it connects to the candidate recall module, and downstream, it sequentially connects to the boundary reset module, risk control module, and final lip-shape output module. The entire process adopts a frame-by-frame streaming mechanism, with each frame of data processed independently and immediately transferred to the next stage after processing, adapting to the 30fps real-time output requirements of digital humans. The input for this stage is the valid candidate entries filtered by the codebook after approximate nearest neighbor retrieval. Candidate generation uses cosine similarity to complete audio feature matching, retaining only entries with a similarity greater than or equal to 0.5. The number of candidates is determined based on the deployment scenario: 8 entries for standard real-time scenarios, 5 entries for low-computing-power embedded scenarios, and 10 entries for offline optimization scenarios. If the number of qualified candidates is insufficient, all valid content is retained. All candidate entries come from the standard 512-class lip-shape codebook. Each entry contains complete fields such as audio features, 20-dimensional lip blendShape parameters, tooth status identifiers, articulation stage labels, and syllable boundary identifiers. The entire process follows the unified indexing rules of the codebook, avoiding cross-codebook calls. The current retrieval ranking model has completed the integrated configuration of multi-objective optimization functions, weight system, all-class thresholds, and penalty rules, and can directly perform scoring calculations.

[0077] The model is based on a multi-objective optimization function, and its operation is divided into two main stages. The first stage calculates the scores of five evaluation dimensions and weights them to obtain a basic comprehensive score. The second stage verifies abnormal scenarios and adds three types of penalty items to correct the score. All individual calculation results and penalty values ​​are uniformly normalized to the range of 0 to 1, ensuring that results from different dimensions and rules can be compared and jointly calculated. Individual scores are calculated for each valid candidate in turn. The five dimensions correspond to different calculation methods and business functions, and weighted summation is performed with fixed weights.

[0078] For the current audio matching, cosine similarity is used to calculate the matching degree between the 80-dimensional Mel-spectral features of the current frame and the candidate built-in audio features. The score ranges from 0 to 1, with higher values ​​indicating better matching. This has a fixed weight of 1.0 and is the core benchmark for scoring. For audio window consistency, the average cosine similarity is calculated for all frames within a local audio window to evaluate the candidate's adaptability to the overall audio context and co-articulation. This has a fixed weight of 0.7. For historical video continuity, the inter-frame difference between the current candidate and historical lip shapes is calculated using three optional methods: index distance, absolute difference in mouth opening, and L2 lip shape distance. The score is converted according to the rule that the smaller the distance, the higher the score. This has a fixed weight of 0.8 to ensure smooth transitions between frames. For articulation stage adaptation, matching verification is performed based on the three types of articulation stage labels (start, core, and end) marked in the codebook to determine whether the lip shape and vocalization state match. This has a fixed weight of 0.4. For boundary preferences, binary identifiers of syllable boundaries and pause boundaries are read, and preference scores are assigned to the boundary positions. Ordinary frames do not receive additional scores and are assigned a fixed weight of 0.5. The scores of the five dimensions are multiplied by their corresponding weights and then summed to generate the candidate's basic comprehensive score.

[0079] Based on the basic comprehensive score, three types of abnormal scenarios are sequentially verified: lip abrupt changes, sudden changes in tooth state, and risks. Upon triggering a rule, the corresponding penalty score is deducted to correct the final score. Each of the three types of penalties has an independent weight and judgment threshold. For lip abrupt changes, the judgment threshold is 0.25 for the difference in mouth opening between frames. When the difference in mouth opening between adjacent frames exceeds this value, it is judged as a significant lip deformation, triggering the penalty item, which has a weight of 1.2. For sudden changes in tooth state, tooth state uses a binary classification label: 0 represents invisible teeth, and 1 represents visible teeth. When a state switch occurs in a normal frame that is not a syllable or a pause, a full penalty is applied; when a switch occurs within a syllable boundary or pause boundary, a half penalty is applied, which has a weight of 1.5. For risks, two judgment conditions are set: the difference between the highest and second-highest scores is less than 0.08, indicating low candidate discrimination; the highest score of a single candidate is less than 0.30, indicating insufficient overall matching confidence. Meeting either condition triggers the risk penalty, which has a weight of 1.0. By combining the penalty item weights with the normalized penalty values, the base comprehensive score is reduced to obtain the final comprehensive score for each candidate. The higher the score, the better the candidate's overall performance in audio matching, temporal coherence, pronunciation adaptation, and state stability.

[0080] After calculating the scores of all valid candidates, they are uniformly sorted in descending order of their comprehensive scores. The sorting process does not remove any valid entries; all candidates in this round are retained, only their order is adjusted. The sorting priority is entirely based on the comprehensive score; candidates with higher scores are the preferred selection targets for the current frame, while those with lower scores are considered as backups. This sorting logic is deeply integrated with the multi-objective optimization function and penalty rules, comprehensively reflecting the dual requirements of matching accuracy and temporal stability.

[0081] After sorting, the candidate queue, serving as the core basis, flows frame by frame to downstream modules. First, it is sent to the boundary reset and risk control module. This module, combining phoneme alignment, boundary types identified by VAD detection, and risk levels such as index jumps, mouth opening offsets, and score deviations, performs secondary screening and strategy adjustments on the ordered candidates. The processed results are then sent to the final lip-sync output module to finalize the lip-sync parameters, tooth state, and codebook index for the current frame. Each round of final output results is simultaneously written to a 5-frame historical video state FIFO circular buffer, providing data for the historical continuity calculation and state extraction of the next frame, forming a streaming computation closed loop. The entire scoring and sorting process is embedded in the end-to-end retrieval system, integrating single-frame matching, context adaptation, historical temporal constraints, and anomaly risk management. This avoids issues such as lip-sync jitter, tooth flickering, and morphological abrupt changes from the source of candidate selection, balancing audio matching accuracy with the overall naturalness of the lip-sync sequence.

[0082] S105 introduces a temporal boundary adaptive control mechanism to dynamically modulate the model weights at pronunciation boundaries and high-risk retrieval points, thereby achieving historical constraint decay and contextual preference enhancement and suppressing error propagation across time.

[0083] In one implementation, combining phoneme alignment results, VAD speech status, high-risk judgment threshold, and joint scoring original weights, and following the execution requirements of boundary recognition, risk identification, dynamic weight adjustment, gradual restoration, and risk blocking, the complete control process is determined by boundary classification judgment, high-risk point screening, weight adaptive correction, weight gradual restoration, and risk emergency control. This step is the core execution part of the temporal boundary adaptive control mechanism, connecting upstream to the joint retrieval and scoring module and downstream to the candidate sorting and final lip shape output module. The whole process adopts a frame-by-frame streaming operation mode, which will be explained step by step below with specific parameters.

[0084] This process includes four types of basic input data. The generation basis, purpose, and supporting standards for each type of data are as follows: The first type is phoneme alignment results, generated by the MontrealForcedAligner tool. This is used to accurately locate syllable segmentation positions and is the core basis for strong syllable boundary recognition. The second type is VAD speech state, output by speech activity detection technology. It combines two quantification standards to determine silence intervals: the short-term audio energy is less than 20% of the global average energy, and the silence duration is not less than 150ms, corresponding to approximately 4 to 5 frames. This distinguishes between audible intervals and pause intervals, supporting the determination of weak pause boundaries. The third type is high-risk judgment thresholds, which include a full set of quantification standards: the codebook index jumps continuously twice or more, the teeth state flips continuously twice or more, the difference in mouth opening between frames is greater than 0.35, the difference between the highest and second-highest candidate scores is less than 0.08, and the highest score of a single candidate is less than 0.30. Meeting any one of these criteria determines a high-risk point. This set of thresholds is the unified standard for anomaly recognition. The fourth category is the original weights of the joint scoring, which are the initial weight values ​​of the multi-objective optimization function and various penalty terms. All weight adjustments are based on this set of values ​​and will not be permanently changed after adjustment.

[0085] The entire process operates in a fixed serial order, with the output of the previous stage serving as the input for the next stage. Data flows unidirectionally and consists of five execution stages. Boundary delineation is completed by combining phoneme alignment results with VAD speech status. Based on phoneme switching rules, consonant and vowel switching and word / syllable intervals are identified to define strong syllable boundaries; weak pauses are defined based on VAD detection results, silence duration, and energy standards. Both types of boundaries are uniformly marked as boundary identifier 1, while other regular audio frames are marked as 0. This stage completes the full-frame boundary type differentiation and outputs boundary identifiers, which serve as prerequisites for subsequent weight adjustments.

[0086] The system invokes a predetermined high-risk threshold to verify each item in the codebook index, tooth status, mouth opening, and candidate score of the current frame. Tooth status uses a binary classification: 0 represents invisible teeth, and 1 represents visible teeth. The system compares the current frame against criteria such as the number of jumps, the number of state flips, the difference in mouth opening, the difference in score, and the minimum confidence level to identify high-risk points and simultaneously label the specific risk type, providing a basis for subsequent emergency response. Based on the original weights of the joint scoring, differential coefficient adjustments are performed for different boundary types. For strong syllable boundary scenarios, the historical continuity weight is adjusted to 0.2 times the original value, while the boundary preference weight is adjusted to 1.5 times the original value. For weak pause boundary scenarios, the historical continuity weight is adjusted to 0.1 times the original value, while the boundary preference weight is adjusted to 2.0 times the original value. For identified high-risk points, based on the above weight adjustments, the weights related to historical states are further weakened to prevent the continued propagation of historical errors and to strengthen the matching weight between the current audio and the pronunciation stage.

[0087] The weight adjustment state of the boundary frames is not permanent. Within three consecutive frames after the end of the syllable boundary and pause boundary, a linear gradual recovery operation is performed. The weights gradually recover in a smooth manner, eventually returning to their original values ​​completely. The entire process does not involve parameter jumps, preventing abrupt weight changes from causing abnormal lip-syncing.

[0088] For high-risk locations identified through screening, specific risk blocking operations are performed. The system activates a state freeze mechanism to lock the current abnormal lip-sync index and prevent the risk from spreading. At the same time, it retrieves historical stable indexes from the 5-frame circular buffer and performs an index rollback operation, replacing the abnormal candidates with verified stable states to block the propagation of errors across frames at the source.

[0089] This stage receives data such as scores, features, and original weights from the joint retrieval and scoring module upstream, and transmits boundary types, risk levels, adjusted weights, and emergency instructions downstream to the candidate ranking and final lip-sync output modules. It is adapted to the 30fps frame-by-frame streaming rules of digital humans, immediately proceeding to the next frame's computation after the entire process of a single frame is completed, without performing offline batch processing, thus meeting the low-latency requirements of real-time driving. This stage collaborates with front-end audio processing, historical state analysis, candidate retrieval, and other end-to-end modules to form a complete dynamic risk and boundary control system.

[0090] The MontrealForcedAligner tool was used to perform forced phoneme alignment, analyze the phoneme distribution and switching nodes in the audio, and identify strong syllable boundaries based on phoneme switching rules. The switching between consonants and vowels, word intervals, and the position of the end of a core vowel were all identified as strong syllable boundaries. Simultaneously, VAD speech detection technology was used to distinguish weak pause boundaries. The criteria for this were that the short-term audio energy was lower than the global average energy, and the silence duration reached 150ms, corresponding to 4 to 5 audio frames. Meeting these conditions was considered a weak pause boundary. Both strong syllable boundaries and weak pause boundaries were used as trigger scenarios for weight adjustment, and the results of both types of boundary identification were output simultaneously, serving as the basis for subsequent differentiated weighting.

[0091] Based on five predetermined thresholds—index jumps, tooth flips, mouth opening deviations, score differences, and confidence levels—high-risk locations are screened frame by frame. Tooth status uses a binary classification: 0 represents invisible teeth, and 1 represents visible teeth. The judgment criteria are as follows: two or more consecutive index jumps indicate frequent index switching risk; two or more consecutive tooth flips indicate abnormal tooth status risk; a mouth opening difference greater than 0.35 between frames indicates excessive mouth opening risk; a difference between the highest and second-highest candidate score less than 0.08 indicates low candidate discrimination; and a single candidate highest score below 0.30 indicates low overall matching confidence. If any of these conditions are met, the current frame is marked as a high-risk propagation location, and the screening results are simultaneously transferred to subsequent control stages.

[0092] Based on the identified boundary types and screened risk levels, historical continuity weight attenuation coefficients and boundary preference gain coefficients are configured for different scenarios. For scenarios with strong syllable boundaries, the historical continuity weight is adjusted to 0.2 times the original value, and the boundary preference weight is adjusted to 1.5 times the original value. For scenarios with weak pause boundaries, the historical continuity weight is adjusted to 0.1 times the original value, and the boundary preference weight is adjusted to 2.0 times the original value. For identified high-risk points, based on the corresponding boundary weight configuration, the historical continuity weight is further weakened, and the matching weight between the current audio and pronunciation stage is strengthened to prevent the continuous propagation of historical error states. All coefficients are configured in advance, with different adjustment standards for different scenarios to ensure that differentiated weight adjustments can be directly executed. The system uniformly sets the weight repetition rule, and after the boundary frame ends, a linear gradual recovery is performed within 3 consecutive frames. The weights will not jump instantaneously, but will gradually return to the original weights frame by frame in a smooth manner from the adjusted values. This rule is preset in advance to avoid abnormal mouth shape and teeth state caused by sudden weight changes at the parameter level, ensuring the stability of streaming operation.

[0093] By combining boundary type, risk level, and original weight parameters, differentiated weight modulation is applied to ordinary frames, syllable boundaries, and pause boundaries. For high-risk points, state freezing and index rollback strategies are simultaneously implemented. When the current frame is determined to be an ordinary audio frame, no weight modification is made throughout the process; the original weights of the joint scoring are directly used for calculation. The original weights include weights for five evaluation dimensions and three types of penalty items: single-frame audio matching weight 1.0, audio window consistency weight 0.7, historical video continuity weight 0.8, pronunciation stage adaptation weight 0.4, boundary preference weight 0.5, lip jump penalty weight 1.2, tooth state change penalty weight 1.5, and risk penalty weight 1.0. This processing method ensures that regular segments within syllables and outside of speech pauses strictly adhere to the original constraint logic, continuously relying on the historical video continuity weight to control the amplitude of lip shape changes, avoiding irregular shape switching, and ensuring the basic stability of the entire lip shape sequence.

[0094] For two types of boundary frames—syllable boundaries and pause boundaries—weight adjustments are made using fixed coefficients. The core idea is to weaken historical state constraints and strengthen boundary adaptation capabilities. For strong syllable boundaries, the original weight of historical continuity is adjusted to 0.2 times its original value, while the original weight of boundary preference is adjusted to 1.5 times its original value. By reducing the weight of historical continuity, the forced constraint of past lip shape states on the current frame is reduced. Combined with the increased boundary preference weight, this allows the lip shape to follow the syllable switch and complete reasonable shape changes. For weak pause boundaries, the original weight of historical continuity is adjusted to 0.1 times its original value, while the original weight of boundary preference is adjusted to 2.0 times its original value. Compared to syllable boundaries, pause scenarios further weaken the historical constraints, allowing for greater freedom in lip shape switching and matching the lip shape change patterns before and after speech silence. Only the weights of historical continuity and boundary preference are adjusted for both types of boundaries; the original weights of other evaluation dimensions and penalty items remain unchanged to ensure that core indicators such as audio matching and pronunciation adaptation are not affected. The weight adjustment only takes effect in the current boundary frame, and subsequent linear gradual recovery is performed according to the established rules.

[0095] For high-risk locations identified through screening, in addition to implementing corresponding boundary weight modulation, two supporting emergency strategies are simultaneously activated to double-block the propagation of abnormal states. The first is a state freeze operation, which locks the currently identified abnormal lip shape index, prohibiting it from participating in the candidate selection and output of subsequent frames, preventing the continued spread of problems such as lip shape jumps, tooth flips, and low confidence. The second is an index rollback operation, which retrieves historical stable indices and the main index from the system's 5-frame FIFO circular buffer as alternative options, switching the current operation to a verified reliable lip shape state. High-risk determination follows the full set of threshold standards: two or more consecutive jumps in the codebook index, two or more consecutive flips in the tooth state, a difference in mouth opening between frames greater than 0.35, a difference in candidate scores less than 0.08, or a single highest score less than 0.30. Meeting any one of these criteria triggers the entire set of emergency strategies. Regardless of whether the current frame is a boundary frame, high-risk locations are given priority for emergency control, followed by comprehensive handling in conjunction with weight adjustments.

[0096] Integrating boundary types, risk assessment results, dynamic weights, gradual change rules, and emergency control strategies, the system achieves adaptive adjustment of temporal boundaries, attenuating historical constraints and enhancing contextual preferences, effectively suppressing error propagation across time. The integrated control system incorporates five categories of pre-output content, all of which form the core basis for system operation. The first category is audio boundary classification results, categorized into three types: strong syllable boundaries, weak pause boundaries, and ordinary audio frames. The second category is high-risk level assessment results, identifying point types based on predetermined thresholds, including risks such as continuous jumps in codebook indexes, continuous flips in tooth states, excessive inter-frame mouth opening, small candidate score differences, and low overall matching confidence, and marking the corresponding risk type. The third category is dynamically adjusted weight values, including historical continuity weights and boundary preference weights after attenuation and gain processing, while the weights of other evaluation dimensions and penalty items remain unchanged. The fourth category is a linear gradual weight recovery rule, performing smooth recovery for three consecutive frames after the boundary is defined. The fifth category is risk emergency control strategies, including state freezing and index rollback. After the five categories of content are integrated, a unified control system with unified rules and interconnected data will be formed.

[0097] The core objective of the system is to achieve historical constraint decay and contextual preference enhancement, which is achieved through differentiated weight adjustments. In scenarios with strong syllable boundaries, the historical continuity weight is adjusted to 0.2 times its original value, and the boundary preference weight is adjusted to 1.5 times its original value, weakening historical state constraints and increasing the reference ratio of the current audio context. In scenarios with weak pause boundaries, the historical continuity weight is adjusted to 0.1 times its original value, and the boundary preference weight is adjusted to 2.0 times its original value, further relaxing lip-sync restrictions. In ordinary audio frame scenarios, all weights maintain their initial standard values, preserving the strong constraint capability of historical states. The weight adjustment results are directly integrated into the multi-objective optimization function of joint retrieval scoring, completing the control at the computational level. The weights only take effect temporarily in the current boundary frame, and the system is configured with a fixed linear gradual recovery mechanism. Within three consecutive frames after the syllable boundary and pause boundary, the weights smoothly transition to their original values ​​frame by frame according to a linear law, without any parameter jumps. After the weights are fully recovered, the system returns to its normal operating mode until the next boundary frame triggers the adjustment logic. This rule can prevent abnormalities in mouth shape and tooth condition caused by sudden changes in weights.

[0098] For all frames identified as high-risk, a combination of weight adjustment and emergency control measures are applied simultaneously to prevent errors from propagating across frames. The high-risk assessment thresholds are as follows: ≥2 consecutive jumps in the codebook index, ≥2 consecutive flips in tooth state, inter-frame mouth opening difference > 0.35, difference between the highest and second-highest candidate score < 0.08, and a single highest score < 0.30. Meeting any one of these criteria qualifies a frame as high-risk. For high-risk frames, a weight adjustment strategy is first implemented at the corresponding boundary, while two emergency operations are activated simultaneously. The first is state freezing, locking the abnormal codebook index to prevent the abnormal state from continuing. The second is index rollback, retrieving historical stable indices and the main index from the 5-frame FIFO circular buffer to replace the current abnormal candidate. This dual control approach curbs mouth jitter and tooth flickering issues at their source.

[0099] S106 employs a time-stable inference strategy, performing conservative selection and time-smoothing correction on fuzzy candidates and low-confidence results to ensure that the retrieval output is continuous and stable in the time dimension.

[0100] In one implementation, this time-series stable inference strategy operates after joint retrieval scoring and candidate ranking, representing the final control stage before lip shape output. It connects upstream to the retrieval and ranking module and downstream directly to the final lip shape output module, fully integrated into the system's frame-by-frame streaming architecture. This strategy primarily handles two types of target objects: fuzzy candidate sets and low-confidence retrieval results. Fuzzy candidates are defined as multiple lip shape entries where the difference between the highest and second-highest scores is less than 0.08. These entries have similar matching effects, and directly selecting the highest score can easily cause inter-frame lip shape jumps. Low-confidence retrieval results are defined as the set of entries with the highest candidate score below 0.30, representing insufficient overall retrieval reliability. The standard input for this stage includes the entire set of ranked lip shape candidates, the comprehensive score of each entry, historical lip shape parameters, historical codebook index, and tooth status data. Tooth status uses a binary classification identifier: 0 represents invisible teeth, and 1 represents visible teeth. All data is taken from the system's standard business links, uniformly adapted to the 512-class standard lip shape codebook index system, and cross-codebook calls are strictly prohibited throughout the process. Meanwhile, this step reuses the entire set of risk assessment criteria for lip jumps, teeth flips, and mouth opening deviations. The threshold for mouth opening deviation assessment is 0.35, and the threshold for lip jump assessment is 0.25.

[0101] The ranked candidate entries are reviewed one by one according to predetermined thresholds and classified into different processing types. The first type is judged based on the score difference; if the difference between the highest and second-highest scores is less than 0.08, it is classified as a fuzzy candidate. The second type is judged based on overall confidence; if the highest score of a single candidate is below 0.30, it is classified as a low-confidence result. After classification, three processing paths are distinguished: normal high-confidence candidates directly enter the output stage; fuzzy candidates and low-confidence results uniformly initiate a time-stability inference process. The judgment process simultaneously refers to the continuous state of lip movements in the most recent 5 frames, combining the lip movement change pattern for comprehensive judgment, avoiding judgment bias caused by relying solely on scores.

[0102] For fuzzy candidates with similar scores, the approach of directly selecting the highest score is abandoned, and a conservative selection logic prioritizing temporal continuity is adopted. During screening, entries with the smallest differences from historical stable indices and historical lip shape parameters are selected first. Three methods can be used to calculate inter-frame differences: index distance, absolute difference in mouth opening, and L2 lip shape distance. The method can be flexibly selected based on computing power and accuracy, with the core objective being to ensure that there are no abrupt changes in lip shape and tooth status between adjacent frames. Simultaneously, differential constraints are applied based on audio frame type: for ordinary frames within syllables and corresponding to speech pauses, the range of lip shape variation is further tightened; for legal switching positions such as syllable boundaries and pause boundaries, the selection restrictions are appropriately relaxed, taking into account the naturalness of pronunciation. This method effectively reduces the problem of frequent lip shape switching at the candidate selection level.

[0103] To address low-confidence retrieval results, a three-level gradient temporal smoothing correction scheme is implemented. All operations adhere to a lip transition threshold of 0.25 and tooth state switching rules. The first level is inter-frame linear interpolation correction, which merges the current candidate lip shape parameters with the final output parameters of the previous frame to smooth the transition of lip shape and reduce the abruptness caused by single-frame anomalies. The second level is historical median filtering correction, which extracts the median state from the lip shape parameters and tooth states of the most recent 5 frames as the basis for correction, filtering out random fluctuations in single frames. The third level is a fallback: when the confidence level is severely insufficient or there are no effective candidates, the stable output result of the previous frame is directly used to prevent lip shape distortion and screen flickering.

[0104] After completing conservative selection and temporal smoothing correction, a final temporal verification is conducted. The verification includes three aspects: inter-frame lip shape jump amplitude, tooth state continuity, and overall sequence smoothness. Each aspect is checked against predetermined risk thresholds such as 0.25 and 0.35. If all verifications meet the standards, the final lip shape output data is generated. If the verification does not meet the temporal stability requirements, a multi-level conservative backoff mechanism is automatically triggered, retrieving historical stable patterns from a 5-frame FIFO circular buffer as output content for fallback processing.

[0105] S107 transforms lip-sync retrieval from single-frame optimality to temporal context-constrained optimality through frame-by-frame streaming temporal modeling and iterative reasoning of the retrieval model, thereby improving the temporal stability and naturalness of the retrieval model.

[0106] In one implementation, the system relies on frame-by-frame audio streams, dual-mode audio window features, codebook candidate data, and historical lip-sync states to differentiate operating scenarios, classifying them into three categories: regular frames, syllable / pause boundary frames, and high-risk frames. The classification results serve as the core basis for subsequent differentiated inference and parameter adjustment. The entire process adheres to streaming operation specifications, temporal constraint rules, and sequence optimization requirements, adapting to the low-latency characteristics of real-time digital human operation.

[0107] This stage has four types of standard inputs. All data formats, storage rules, and field definitions are consistent with the 512-class standard mouth shape code ontology system, and cross-codebook calls are prohibited. For continuous frame-by-frame audio streams, the original audio sampling rate is 16000Hz, the single frame duration is 25ms, and the frame shift is 10ms. Hamming windows are used for framing, serving as the basic data source for audio feature extraction. The dual-mode audio window features include two types: strict causal mode and short look-ahead mode. The strict causal mode window consists of the current frame plus four previous historical audio frames, for a total of 5 frames and a duration of approximately 160ms. The short look-ahead mode adds 1 to 2 future audio frames to the original window, for a total of 6 to 7 frames and a duration of approximately 200ms to 230ms. The two modes can be flexibly switched according to the on-site latency requirements. The 80-dimensional Mel-spectrum features within the window are uniformly normalized to the 0-1 interval, supporting three fusion methods: one-dimensional convolution, mean pooling, and weighted summation.

[0108] The codebook recall candidate item set is generated by the candidate recall module through cosine similarity retrieval, retaining only valid items with a similarity ≥ 0.5. Eight candidates are selected for standard real-time scenarios, five for low-computing-power embedded scenarios, and ten for offline optimization scenarios. Historical mouth shape state data is taken from a 5-frame FIFO circular cache. The cache only stores the final system output data, does not retain intermediate candidates, and stores a fixed set of three types of data: codebook index, 20-dimensional lip blendShape parameters, and tooth state. Tooth state uses a binary classification identifier: 0 represents invisible teeth, and 1 represents visible teeth. The linkage rule is that when mouth opening < 0.15, it is forcibly set to 0; when mouth opening > 0.40, it is allowed to be set to the range of 0.15 to 0.40, maintaining the state of the previous frame.

[0109] The system adheres to three fundamental specifications throughout the process, serving as general guidelines for scene differentiation and subsequent processing. Data is processed sequentially frame by frame according to the audio order, with the next frame only proceeding after all calculations for each frame are completed, avoiding offline batch processing to ensure real-time low latency. Temporal constraint rules calculate inter-frame differences based on index distance, absolute difference in mouth opening, and L2 lip shape distance, combined with predetermined thresholds to limit the range of changes in lip shape and tooth status, ensuring sequence continuity. Comprehensive processing is performed by incorporating audio context, phoneme rhythm, and boundary features, balancing audio matching accuracy with visual naturalness of lip shape.

[0110] For regular frames, the criteria are that the current frame does not belong to a syllable boundary or pause boundary, and all risk indicators are within the normal range. For example, the audio is in a continuous pronunciation phase within a word or phrase, with no phoneme switching, and VAD detection continuously identifies it as an audible state. The codebook index has less than 2 consecutive jumps, the teeth state has no flipping, and the difference in mouth opening between frames is less than 0.35; the difference between the highest and second-highest candidate scores is ≥0.08, and the highest score of a single candidate is ≥0.30. Standard timing constraints are used throughout this frame, and candidate selection and lip shape output are completed according to conventional logic.

[0111] Syllable / pause boundary frames are divided into two categories: strong syllable boundaries and weak pause boundaries. These are determined jointly by phoneme alignment tools and VAD detection, and both types of boundaries are uniformly marked with the identifier 1. An example of a strong syllable boundary is shown below. Forced phoneme alignment is performed using MontrealForcedAligner, identifying the transition between consonants and vowels, word intervals, and the end of core vowels, thus classifying them as strong syllable boundaries. An example of a weak pause boundary is shown below. When VAD detects a switch from a spoken state to a silent state, the audio energy is below 20% of the global average energy for a short period, and the silence duration is ≥150, corresponding to 4 to 5 audio frames, it is classified as a weak pause boundary. This type of frame allows for reasonable lip-syncing based on pronunciation rules, and subsequent processes will dynamically adjust the weights.

[0112] A frame is classified as high-risk if it meets any of the following threshold conditions: ≥2 consecutive jumps in the codebook index; ≥2 consecutive flips in tooth state; inter-frame mouth opening difference > 0.35; difference between the highest and second-highest candidate score < 0.08; and highest single candidate score < 0.30. For example, frames with frequent switching of mouth shape indexes, repeated changes in tooth state, excessive mouth opening, or similar candidate scores with low overall matching confidence are all classified as high-risk frames and require the activation of risk emergency control strategies.

[0113] Scene differentiation results generate unique scene tags, which are synchronously transmitted with the data stream to downstream modules such as temporally stable inference, boundary control, and final lip-sync output. Regular frames retain the original weights of the joint scoring, strictly enforcing temporal constraints and tightening the switching range of lip-sync and tooth states. Syllable / pause boundary frames have their weights adjusted according to rules: historical continuity weights are reduced to 0.2 or 0.1 times the original value, while boundary preference weights are increased to 1.5 or 2.0 times the original value. After the boundary ends, linear gradual recovery is performed for three consecutive frames. High-risk frames simultaneously activate state freezing and index rollback strategies, retrieving historical stable indexes from the five-frame cache as a fallback to prevent abnormal cross-frame propagation.

[0114] The system processes scenario differentiation results based on strict causal / short look-ahead window configuration, joint comprehensive score, boundary markers, and predetermined risk thresholds, setting a frame-by-frame inference mechanism and temporal optimization granularity. Based on three operating scenarios—regular frames, syllable / pause boundary frames, and high-risk frames—the system combines audio window configuration, candidate comprehensive score, and risk judgment thresholds to complete the frame-by-frame inference rules and temporal optimization granularity configuration, forming a unified execution standard. The system provides two audio window configurations, which can be flexibly switched according to business latency requirements, with fixed standards for cache, frame count, and duration. For the strict causal mode, only historical frames and the current frame are combined to construct the audio window; future audio data is not used. The total number of frames in the window is 5, corresponding to a duration of approximately 160ms. Combined with a 4-frame capacity FIFO circular cache for dynamic updating of audio features, this mode has no additional latency and is suitable for high real-time deployment scenarios such as live streaming, real-time interaction, and embedded systems.

[0115] For the short look-ahead mode, 1 to 2 future audio frames are added to the historical and current frames, resulting in a total window of 6 to 7 frames and a duration of approximately 200ms to 230ms. This leverages future audio to optimize syllable transitions, making it suitable for scenarios with high latency tolerance, such as recording and offline production. The 80-dimensional Mel-spectral features in both modes are uniformly normalized to the 0-1 range, ensuring consistent feature specifications.

[0116] The overall score is calculated using a multi-objective optimization function, with all scores uniformly normalized to the 0-1 range. It comprises a weighted calculation of five evaluation dimensions and three types of penalties, serving as the core basis for candidate selection. Syllable pause boundary markers use binary identifiers: marker 1 represents a strong syllable boundary or a weak pause boundary, while marker 0 represents a regular audio frame. A full set of thresholds is used for high-risk judgment: ≥2 consecutive jumps in the codebook index, ≥2 consecutive flips in tooth state, a difference in mouth opening between frames > 0.35, a difference between the highest and second-highest candidate score < 0.08, and a single highest score < 0.30. Meeting any one of these criteria classifies a frame as high-risk.

[0117] By combining window mode, boundary markers, and risk thresholds, differentiated inference logic is set for three types of scenarios. For regular frames, the boundary marker is 0, and all risk indicators are within the standard range. The inference process adopts the standard mode, fully utilizing the original weights: single-frame audio matching 1.0, audio window consistency 0.7, historical video continuity 0.8, pronunciation stage adaptation 0.4, boundary preference 0.5, lip jump penalty 1.2, tooth state change penalty 1.5, and risk penalty 1.0. The comprehensive score is used as the main screening criterion, strictly tightening the switching range of mouth shape and tooth state. The tooth state follows the rule that 0 represents invisible and 1 represents visible, and the mouth opening difference is controlled within 0.35 to avoid irregular jumps. For boundary frames, the boundary marker is 1, including strong syllable boundaries and weak pause boundaries. In the strong syllable boundary scenario, the historical continuity weight is adjusted to 0.2 times the original value, and the boundary preference weight is adjusted to 1.5 times the original value. In scenarios with weak pauses, the weight of historical continuity is adjusted to 0.1 times the original value, and the weight of boundary preference is adjusted to 1.5 times the original value. Temporal constraints are appropriately relaxed, allowing lip movements to change reasonably with syllable switching and speech pauses. After the boundary ends, a linear transition is performed for three consecutive frames, and the weights gradually return to their original values. For high-risk frames, if any high-risk threshold condition is met, the inference process tightens the judgment criteria across the board. In addition to referencing the comprehensive score, state freezing and index rollback strategies are simultaneously enabled, retrieving historical stable indices and lip movement parameters from a 5-frame FIFO circular buffer. Candidates with the smallest difference from historical data are prioritized, avoiding entries with similar scores or low confidence, thus preventing the propagation of abnormal states across frames.

[0118] Different optimization focuses are set according to scene type to balance audio matching accuracy and lip-sync timing stability. Regular frames maintain a normal optimization step size, evenly optimizing various indicators such as single-frame audio matching, audio window consistency, historical video continuity, and pronunciation stage adaptation, ensuring audio matching accuracy while maintaining overall lip-sync sequence stability. Boundary frames adjust the optimization focus, emphasizing pronunciation stage adaptation and boundary preference dimensions. The optimization weight of historical continuity is reduced to adapt to the pronunciation rhythm of syllable switching and speech pauses, improving the naturalness of lip-sync transitions. High-risk frames focus on temporal continuity as the core optimization direction, emphasizing the optimization of historical video continuity while weakening the weight of single-frame audio matching. Prioritize correcting abnormal issues such as mouth opening offset, index jumps, and tooth flipping, making sequence stability the primary goal.

[0119] The system processes scene differentiation results, audio features, scoring results, and dynamic weights, incorporating execution parameters for lip shape parameter continuity verification, lip and teeth state transition identification, and stable index rollback. Two audio window configuration methods are provided, allowing flexible switching based on business needs for latency and effect. Both modes are equipped with a fixed number of frames and operational logic. The first is a strict causal mode, which uses only historical and current audio frames to construct a local audio window, without calling any future audio data. This mode has a total window of 5 frames, corresponding to a duration of approximately 160ms. Data processing strictly follows causal logic, without generating additional latency, and is primarily used in high real-time scenarios such as embedded devices, live streaming, and real-time interaction. The second is a short look-ahead mode, which adds 1 to 2 future audio frames to the historical and current frames, resulting in a total window of 6 to 7 frames, corresponding to a duration of approximately 200ms to 230ms. By supplementing collaborative pronunciation information with short future audio, it optimizes syllable transitions and improves the overall naturalness of lip movements, making it suitable for scenarios with high latency tolerance, such as recording and offline production. Both window modes are equipped with a 4-frame capacity FIFO circular buffer to dynamically update audio features, ensuring low latency in data flow. The window configuration result will serve as the basic input condition for the inference mechanism.

[0120] When configuring rules, three types of core data are uniformly retrieved as the basis for judgment. All data are taken from upstream joint retrieval scoring, boundary identification, and risk screening modules. The first type is the full joint comprehensive score. All scores are normalized and range from 0 to 1. It is calculated by weighting five evaluation dimensions and three types of penalty items and is the core basis for candidate selection. The second type is the syllable pause boundary marker. A unified identifier is used to distinguish frame types: a marker of 1 represents a strong syllable boundary or a weak pause boundary, and a marker of 0 represents a normal frame. The third type is various risk judgment thresholds, including: ≥2 consecutive jumps in the codebook index, ≥2 consecutive flips in tooth state, a difference in mouth opening between frames > 0.35, a difference in candidate scores < 0.08, and a single highest score < 0.30. Meeting any one of these criteria results in a high-risk frame.

[0121] By combining window mode, boundary markers, and risk thresholds, a dedicated frame-by-frame inference process is set up for three types of operating scenarios, with clear rule distinctions and independent execution logic.

[0122] For the standard frame inference rules, the current frame boundary flag is 0, and all risk indicators are within the normal range. The inference process adopts a standard mode, using the original weights of the joint scoring throughout, and taking the candidate comprehensive score as the primary screening criterion. Entries with higher scores are selected first, while strictly enforcing temporal constraints, tightening the switching range of mouth shape and tooth state, prohibiting irregular jumps, and ensuring a smooth and continuous mouth shape sequence during pronunciation.

[0123] For the boundary frame inference rules, the current frame boundary flag is 1, encompassing both strong syllable boundaries and weak pause boundaries. The inference process appropriately relaxes temporal constraints, adjusting weights according to predetermined coefficients. For strong syllable boundaries, the historical continuity weight is adjusted to 0.2 times the original value, and the boundary preference weight is adjusted to 1.5 times the original value. For weak pause boundaries, the historical continuity weight is adjusted to 0.1 times the original value, and the boundary preference weight is adjusted to 2.0 times the original value. Based on the comprehensive score, lip movements are allowed to follow syllable transitions, and speech pauses are allowed to complete reasonable morphological changes, conforming to natural pronunciation rules.

[0124] For high-risk frame inference rules, if a frame meets any one of the high-risk judgment conditions, the judgment criteria are tightened across the entire inference process to strengthen risk control. In addition to referring to the comprehensive score, two emergency strategies, state freezing and index rollback, are simultaneously activated. Candidates with the smallest differences in historical stable indices and lip-shape parameters within the 5-frame circular buffer are prioritized. Entries with similar scores and low confidence are actively avoided to prevent the propagation of abnormal states across frames from the inference stage.

[0125] The temporal optimization granularity corresponds to the optimization focus of different scenarios, working in conjunction with inference rules to balance retrieval matching accuracy and lip-sync output stability. For the regular frame optimization granularity, the system's default normal optimization step size is maintained, with the optimization focus balancing audio matching accuracy and temporal continuity. Simultaneously, balanced optimizations are performed on dimensions such as single-frame audio matching, audio window consistency, and historical video continuity, maximizing the fit between audio and lip movements while ensuring lip-sync stability.

[0126] The optimization granularity for boundary frames was adjusted, with a focus on pronunciation adaptation. The optimization weight of historical temporal constraints was appropriately reduced, and the optimization was strengthened for pronunciation stage matching and boundary preference dimensions to make lip movements adapt to the rhythm of syllable switching and speech pauses, thereby improving the naturalness of pronunciation transitions.

[0127] For high-risk frames, the optimization granularity focuses on temporal continuity, significantly allocating optimization resources to the continuity dimension of historical video while reducing the weight of single-frame audio matching. Priority is given to correcting anomalies such as abrupt lip movements, flickering teeth, and excessive mouth opening, with sequence stability as the primary objective.

[0128] This system integrates scene differentiation results, feature data, weight configurations, and constraint parameters. Leveraging frame-by-frame streaming temporal modeling and iterative inference, it transforms independent single-frame optimal retrieval into temporally context-constrained optimal retrieval, outputting temporally stable and naturally performing lip-sync driven results. Before inference begins, all data and rules are summarized as a unified input for the overall computation. The integrated content includes three types of scene differentiation results, dual-mode audio window features, a complete set of model weights, and boundary and risk constraint parameters. Scene differentiation results are divided into three categories: regular frames, syllable / pause boundary frames, and high-risk frames. All frame labels are updated synchronously frame-by-frame with the audio stream. Audio features are uniformly normalized to 80-dimensional Mel-spectrum features in the 0-1 range, generating contextual representations based on corresponding audio window modes. Model weights include five evaluation dimensions, three types of penalty terms, and boundary scene-specific weight attenuation coefficients and gain coefficients. Constraint parameters cover a complete set of risk assessment thresholds, including lip transitions, tooth flips, mouth opening deviations, candidate score differences, and overall confidence, while also including execution rules such as linear weight transitions, index rollback, and state freezing. All data and parameters are interconnected, providing a complete basis for comprehensive global judgment.

[0129] The entire reasoning process no longer relies solely on the current single-frame audio for judgment, but instead employs a multi-dimensional global judgment logic. It comprehensively considers local audio context information, historical lip shape states, boundary control rules, and risk constraints to jointly evaluate candidates. Traditional methods only retrieve the optimal lip shape based on the audio features of the current frame, with frames operating independently, easily leading to frequent switching and abrupt changes in morphology. This solution relies on temporal context modeling, combining window features composed of preceding and following audio frames, along with historical codebook indexes within a 5-frame FIFO circular buffer, 20-dimensional lip blendShape parameters, and tooth state data for joint judgment. Tooth state uses a binary classification rule where 0 represents invisible and 1 represents visible, while also incorporating mouth opening values ​​for linkage constraints. This approach upgrades the independent optimal retrieval of a single frame to a globally optimal retrieval under temporal context constraints, logically avoiding the shortcomings of single-frame retrieval.

[0130] The process follows a predetermined sequence of rules to complete two core operations: candidate selection and lip shape correction. First, target candidates are selected based on a comprehensive score, scene label, and weight configuration. Regular frames retain their original weights, strictly adhering to temporal constraints, prioritizing entries with higher scores and smaller differences from historical states. Syllable / pause boundary frames have their weights adjusted according to coefficients: historical continuity weights are lowered, while boundary preference weights are increased, allowing for reasonable lip shape switching based on pronunciation patterns. After the boundary ends, a linear gradient is applied for three consecutive frames, gradually restoring the weights to their original values. High-risk frames simultaneously employ state freezing and index rollback strategies, retrieving stable historical indexes from the cache for selection. Second, state correction is performed based on the selection results. If low confidence or morphological abnormalities exist, inter-frame linear interpolation and historical median filtering are used sequentially to smooth lip shape parameters. If no valid candidates are found, the stable output of the previous frame is directly used. Throughout the correction process, thresholds such as a mouth opening difference of 0.25 and an inter-frame lip shape distance of 0.20 are used to ensure that the lip and teeth states conform to physical morphological rules.

[0131] After filtering and correction, the final driving data for the standard digital human is output, including lip shape key point coordinates, 20-dimensional lip blendShape parameters, and tooth status indicators. All output results for the current frame are written in real-time to a 5-frame historical state FIFO circular buffer. The buffer is automatically updated following a first-in-first-out (FIFO) rule, and the retained data serves as the historical state basis for the next frame's inference. The audio stream progresses frame by frame, and the inference process repeats continuously, forming a complete streaming computation loop to ensure uninterrupted real-time system operation.

[0132] like Figure 2 As shown, a lip-sync retrieval model optimization system based on temporal context modeling includes: The temporal framework construction module 201 is used to build a temporal context modeling framework for lip-sync retrieval, extend single-frame audio to temporal context representation, and establish the association between historical states and the current retrieval. The audio encoding module 202, relying on the temporal window, uses temporal convolution and weighted fusion to complete the temporal modeling of multi-frame audio, extracts articulation coordination and temporal dependency features, and generates temporally consistent audio representations. The historical preference modeling module 203 encodes historical retrieval, continuous state and stability features to generate temporal priors, and constructs temporal smoothing constraints and state transition rules. The retrieval and ranking module 204 builds a time-series context-aware ranking model, integrates multiple matching indicators with a multi-objective optimization function, and superimposes various penalties and constraints to achieve candidate item ranking optimization. The boundary control module 205 introduces a temporal boundary adaptive mechanism to dynamically modulate the model weights at the pronunciation boundary and high-risk points, thereby weakening historical constraints and blocking the temporal propagation of errors. The stable inference module 206 performs conservative selection and temporal smoothing correction for fuzzy and low-confidence candidates to ensure continuous and stable lip-reading retrieval output. The streaming inference module 207 adopts frame-by-frame streaming temporal modeling and retrieval inference, upgrading the single-frame optimal retrieval to context-constrained optimal retrieval, thereby improving the model's temporal stability and output naturalness.

[0133] A computing device includes a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute any lip-sync retrieval model optimization method based on temporal context modeling.

[0134] The methods and / or embodiments in this application can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by a processing unit, it performs the functions defined in the methods of this application.

[0135] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0136] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application.

Claims

1. A method for optimizing a lip-sync retrieval model based on temporal context modeling, characterized in that, include: A temporal context modeling framework for lip-sync retrieval is constructed, which extends single-frame audio features into temporal context representations and establishes a correlation model between historical states and the current retrieval. Contextual feature encoding is completed based on temporal windows. Temporal convolution and weighted fusion are used to perform temporal modeling of multi-frame audio features, extract pronunciation coordination rules and temporal dependency features, and form an audio representation with temporal consistency. We design a retrieval preference model that integrates historical states, encodes historical retrieval results, continuous states and stability features as temporal priors, and constructs temporal smoothing constraints and state transition rules. A temporal context-aware retrieval and ranking model is constructed, which unifies single-frame matching, contextual consistency, historical continuity, and pronunciation stage adaptability into a multi-objective optimization function. Jump penalties, confidence penalties, and risk constraints are added to complete the candidate ranking optimization. An adaptive temporal boundary control mechanism is introduced to dynamically modulate the model weights at pronunciation boundaries and high-risk retrieval points, thereby achieving historical constraint decay and contextual preference enhancement and suppressing error propagation across time. A time-stable inference strategy is adopted to perform conservative selection and time-smoothing correction on fuzzy candidates and low-confidence results, ensuring that the retrieval output is continuous and stable in the time dimension. By using frame-by-frame streaming temporal modeling and iterative reasoning of the retrieval model, lip-sync retrieval is transformed from single-frame optimization to temporal context-constrained optimization, thereby improving the temporal stability and naturalness of the retrieval model.

2. The lip-sync retrieval model optimization method based on temporal context modeling according to claim 1, characterized in that, Contextual feature encoding is performed based on a temporal window. Temporal convolution and weighted fusion are used to perform temporal modeling of multi-frame audio features, extracting articulation coordination patterns and temporal dependency features to form an audio representation with temporal consistency, including: Combining the application scenarios of digital phrasing retrieval with the technical characteristics of audio framing, local window construction, and multi-frame feature fusion, this paper introduces three core processing steps: local audio window construction, FIFO cache update, and multi-frame feature fusion, to build an audio context encoding execution system. Following the processing specifications of complete temporal information, unified feature dimensions, and adaptation to real-time streaming inference, the three key pieces of information—frame-by-frame 80-dimensional Mel spectrum features, audio window frame sequence, and fusion algorithm control parameters—are jointly matched to complete the alignment of the entire audio encoding process. Based on the execution requirements of accurate co-pronunciation representation, strong temporal consistency, and low-latency operation, two window modes, strict causality and short look-ahead, are distinguished. The number of window frames and feature normalization rules are set, and three fusion methods, one-dimensional convolution, mean pooling, and weighted summation, are selected to complete the temporal modeling of multi-frame audio features. The normalized Mel spectrum features are input into the execution system, and historical audio frames are dynamically updated based on a fixed-length FIFO circular buffer to form a local audio window. Feature calculation is completed according to the selected fusion method, and the articulation coordination rules and temporal dependency features are extracted to output an audio representation with fixed dimensions and temporal consistency. By combining the characteristics of the system's overall streaming operation logic output window, contextual basis is provided for subsequent lip-shape candidate recall and joint scoring, generating audio encoding results adapted to digital lip-shape stable driving.

3. The lip-sync retrieval model optimization method based on temporal context modeling according to claim 1, characterized in that, Design a retrieval preference model that integrates historical states, encoding historical retrieval results, continuous states, and stability features as temporal priors, and constructing temporal smoothing constraints and state transition rules, including: By combining the historical codebook index, lip shape parameters, and tooth state data stored in a 5-frame FIFO circular buffer, as well as two types of state extraction rules—main index voting and continuous stable filtering—the historical video state is parsed and encoded to generate temporal prior features. Based on temporal prior features, a retrieval preference model integrating historical states is designed. Historical continuity scores are calculated based on absolute difference in mouth opening, index distance, and L2 mouth shape distance. The state judgment conditions are defined by combining risk thresholds for index jumps, tooth flips, and mouth opening deviations. The retrieval preference model, time-series prior features, continuous scores, and multiple risk verification parameters are linked and bound together to complete the fusion of the model's internal logic and decision conditions. By integrating various features and verification rules based on the retrieval preference model, a time-series smoothing constraint and state transition rule are finally constructed.

4. The lip-sync retrieval model optimization method based on temporal context modeling according to claim 1, characterized in that, A temporal context-aware retrieval and ranking model is constructed, unifying single-frame matching, contextual consistency, historical continuity, and pronunciation stage adaptability into a multi-objective optimization function. Jump penalties, confidence penalties, and risk constraints are added to optimize candidate ranking, including: By combining Top-K lip-sync candidates retrieved from the codebook, audio window fusion features, historical continuity features, pronunciation stage labels, syllable boundary markers, as well as cosine similarity calculation rules and score normalization rules, a temporal context-aware computational system is built. Based on this computing system, a retrieval and ranking model is built, which unifies single-frame audio matching, audio window consistency, historical video continuity, pronunciation stage adaptation, and boundary preference into a multi-objective optimization function, and simultaneously introduces three types of constraint terms: lip jump penalty, tooth state change penalty, and risk penalty. The optimization function, standard weight parameters, penalty thresholds, and risk assessment thresholds are linked and bound together to complete the fusion configuration of model parameters and computational logic; Based on the configured ranking model, a comprehensive score is calculated for all candidate items and they are sorted in descending order to finally complete the candidate item ranking optimization.

5. The lip-sync retrieval model optimization method based on temporal context modeling according to claim 1, characterized in that, An adaptive temporal boundary control mechanism is introduced to dynamically modulate model weights at pronunciation boundaries and high-risk retrieval points, achieving historical constraint decay and contextual preference enhancement, and suppressing error propagation across time, including: Combining phoneme alignment results, VAD speech status, high-risk judgment threshold, and joint scoring original weights, and following the execution requirements of boundary recognition, risk identification, dynamic weight adjustment, gradual restoration, and risk blocking, a complete control process is determined by boundary classification judgment, high-risk point screening, weight adaptive correction, weight gradual restoration, and risk emergency control. Relying on MontrealForcedAligner phoneme forced alignment and VAD speech detection to distinguish between strong syllable boundaries and weak pause boundaries, and combining index jumps, teeth flips, mouth opening offsets, score differences, and confidence thresholds to identify high-risk propagation points, the system configures historical continuity weight decay coefficients and boundary preference gain coefficients for different scenarios, and sets linear gradual recovery rules for the three frames after the boundary. Combining boundary type, risk level and original weight parameters, differentiate weight modulation is performed for ordinary frames, syllable boundaries and pause boundaries, and state freeze and index rollback strategies are simultaneously enabled for high-risk points; By integrating boundary types, risk assessment results, dynamic weights, gradual rules, and emergency control strategies, adaptive adjustment of temporal boundaries is achieved, historical constraint decay and contextual preference enhancement are realized, and error propagation across time series is effectively suppressed.

6. The lip-sync retrieval model optimization method based on temporal context modeling according to claim 5, characterized in that, By employing frame-by-frame streaming temporal modeling and iterative inference of the retrieval model, lip-sync retrieval is transformed from single-frame optimization to temporal context-constrained optimization, thereby improving the temporal stability and naturalness of the retrieval model, including: Combining frame-by-frame audio stream, dual-mode audio window features, codebook candidate data, and historical lip-sync status, and following streaming operation specifications, timing constraint rules, and sequence optimization requirements, three types of operation scenarios are divided: regular frames, syllable / pause boundary frames, and high-risk frames. The scene differentiation results are processed based on strict causal / short look-ahead window configuration, joint comprehensive score, boundary markers, and predetermined risk thresholds, and a frame-by-frame inference mechanism and temporal optimization granularity are set. The scene differentiation results, audio features, scoring results, and dynamic weights are processed, and execution parameters such as mouth shape parameter continuity verification, lip and teeth state transition identification, and stable index rollback are added. By integrating scene differentiation results, feature data, weight configuration and constraint parameters, and relying on frame-by-frame streaming temporal modeling and retrieval iterative reasoning, the independent optimal retrieval of a single frame is transformed into the optimal retrieval constrained by temporal context, outputting temporally stable and naturally performing lip-sync driven results.

7. A lip-reading retrieval model optimization system based on temporal context modeling, characterized in that, The system is used to execute executable instructions to perform the lip-sync retrieval model optimization method based on temporal context modeling as described in any one of claims 1 to 6.

8. An electronic device, characterized in that, include: First processor; And a memory for storing executable instructions of the first processor; wherein the first processor is configured to execute the lip-sync retrieval model optimization method based on temporal context modeling as described in any one of claims 1 to 6 by executing the executable instructions.

9. A computing device, the device comprising a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein, When the computer program instructions are executed by the processor, the device is triggered to execute the lip-sync retrieval model optimization method based on temporal context modeling as described in any one of claims 1 to 6.

Images