A Phoneme Time-Axis Driven Method for Automatic Lip Synthesis in High-Definition Video

Abstract

This invention discloses a phoneme timeline-driven method for automatic lip-sync synthesis in high-definition video, relating to the field of electronic digital data processing technology. The method includes: Step 1: Acquiring input text or speech, generating a phoneme sequence and corresponding start and end time points using a fixed phoneme set and deterministic pronunciation rules, establishing a phoneme timeline; outputting the phoneme timeline, frame index mapping table, and target frame set; Step 2: Based on a fixed and finite lip-sync dictionary and articulation rules, simultaneously establishing three complementary derivation chains to obtain a lip-sync keyframe sequence that satisfies phase continuity and boundary traceability; Step 3: Extracting three-dimensional facial feature points and head posture trajectories from the original video, and outputting a video lip-sync synthesis result with the same resolution as the original video and synchronized with the phoneme timeline based on the posture-consistent lip-sync sequence. This invention significantly improves the accuracy and realism of lip-sync video generation.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing technology, specifically to a method for automatic lip-sync synthesis in high-definition video driven by a phoneme time axis. Background Technology

[0002] In the current fields of video generation and speech synthesis, lip-syncing technology for realistic audio-visual synchronization has become a crucial supporting element in digital humans, virtual anchors, human-computer interaction, and film and television post-production. Traditional methods generally rely on deep learning models to extract features from the input speech, then map these features to lip movements or lip-syncing rendering parameters through learning. The rendering module then generates or modifies facial image sequences based on the prediction results, thereby achieving face-driven audio-visual synchronization. However, despite the widespread application of speech-driven video lip-syncing technology, several technical bottlenecks remain, particularly in high-resolution video, accurate audio-visual synchronization, smooth lip-syncing transitions, and interpretable modeling.

[0003] Existing lip-syncing techniques can be broadly categorized into data-driven end-to-end models and parameter-controlled image manipulation methods. The former is typically represented by speech-to-video synthesis frameworks based on LSTM, Transformer, VAE, or diffusion models. These methods usually use speech features such as Mel spectrum, MFCC, or wav2vec as input, combined with historical frame lip-syncing or images as conditions, to generate a frame or segment of facial image output. Their advantage lies in strong overall modeling capabilities and the absence of explicit rule-based relationships between speech and lip-syncing. However, this type of method suffers from the following problems:

[0004] First, the mapping from speech to video is a complex transformation process from high dimension to high dimension. Although end-to-end models can learn the correlation between features through a large amount of training data, they lack the inherent constraints of language structure and pronunciation principles. This can lead to the phenomenon of "sound wrong, shape right" or "shape right, sound wrong" when mouth movements are driven by speech. That is, the mouth shape generated by the model is visually reasonable but inconsistent with the actual speech or misaligned with the speech temporal sequence.

[0005] Second, existing models lack high-precision synchronization mechanisms in the temporal dimension. Most methods perform frame-by-frame prediction at a fixed frame rate rather than being controlled by external speech structure, which easily leads to cumulative delays or synchronization drift in fast-paced speech, pauses, or non-linear speech segments. Especially in high-definition long-sequence video generation tasks, since phoneme boundaries are not explicitly modeled, the boundaries of lip movements cannot be accurately positioned, resulting in abrupt, discontinuous, or blurred transitions between adjacent phonemes. Summary of the Invention

[0006] To address the aforementioned technical challenges, a novel phoneme-timeline-driven method for automatic lip-sync synthesis in high-definition video is presented. This method accurately models the correspondence between phonemes and lip movements by constructing a phoneme timeline, a frame index mapping table, and three complementary derivation chains. Furthermore, it achieves phase continuity and traceable calibration of articulation points at the keyframe level. By combining 3D facial feature points and head posture trajectories, it achieves posture-consistent lip-sync synthesis. This method offers advantages such as interpretable structure, high time synchronization accuracy, strong visual naturalness, and high resolution adaptability, significantly improving the accuracy and realism of lip-sync video generation. It is suitable for various application scenarios, including digital humans, voice-driven animation, and virtual broadcasting.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] A method for automatic lip-sync synthesis in high-definition video driven by a phoneme time axis, the method comprising:

[0009] Step 1: Obtain the input text or speech, generate a phoneme sequence and corresponding start and end time points using a fixed set of phonemes and deterministic pronunciation rules, and establish a phoneme timeline; using the timestamp of the original video as the subordinate clock, construct a frame index mapping table that corresponds one-to-one between the target frames of the original video and the phoneme timeline; output the phoneme timeline, the frame index mapping table, and the target frame set.

[0010] Step 2: Under the constraints of the phoneme time axis and frame index mapping table, phoneme slicing and phase calibration are performed sequentially. Based on a fixed and limited lip shape dictionary and articulation rules, three complementary derivation chains are established. The conflict resolution and key frame determination are completed by using the principle of non-feedback single forward adjudication and shortest interval preservation, so as to obtain a lip shape key frame sequence that satisfies phase continuity and boundary traceability.

[0011] Step 3: Extract 3D facial feature points and head pose trajectory from the original video, register the lip-sync keyframe sequence with the head pose trajectory according to the frame index mapping table to obtain a pose-consistent lip-sync sequence; output a video lip-sync synthesis result with the same resolution as the original video and synchronized with the phoneme time axis based on the pose-consistent lip-sync sequence.

[0012] Furthermore, in step 2, the process of sequentially performing phoneme slicing and phase calibration includes: dividing the duration of each phoneme into three types of phoneme slices: initial phase, transition phase, and steady-state phase; according to the frame index mapping table, uniquely assigning each target frame to its corresponding phoneme slice and recording the phase type label to form a phase-labeled frame column.

[0013] Furthermore, in step 2, based on a fixed and limited dictionary of mouth shapes and rules of articulation, three complementary derivation chains are established: a main chain, a first side chain, and a second side chain. The main chain is a one-to-one mapping from phonemes to mouth shape categories, used to generate a basic mouth shape sequence without transitions. The first side chain is used for phonological connection, sequentially connecting the initial phase, transition phase, and steady-state phase belonging to the same syllable to generate segment labels for monotonic opening and closing trajectories and lip shape change trajectories. The second side chain corresponds to the articulation point touchpoints, used to mark discrete event sites of lip closure, tongue tip contact, and soft palate opening for phonemes that require closure, friction, or plosives, based on the deterministic touchpoint rules of the lips, alveoli, hard palate, soft palate, and glottis. The mouth shape categories of the main chain, the segment labels of the first side chain, and the event sites of the second side chain are merged on the same timeline according to the phoneme time axis to obtain a three-way chain merge table.

[0014] Furthermore, in step 2, the phoneme timeline is first read. The phoneme timeline consists of phoneme records arranged in chronological order. Each phoneme record contains a phoneme identifier and start and end times. The frame index mapping table is then read. The frame index mapping table provides the correspondence between the target frame and the phoneme timeline. Each record contains the target frame number, the corresponding phoneme identifier, and the frame timestamp. The lip shape dictionary and articulation point contact rules are then loaded. The lip shape dictionary is a fixed and finite mapping set, providing a one-to-one mapping relationship from phoneme categories to lip shape categories. The articulation point contact rules are a fixed set of rules, providing the contact conditions and corresponding event types for the lips, gingiva, hard palate, soft palate, and glottis.

[0015] Furthermore, in step 2, the process of generating the main chain includes: traversing each phoneme record in the order of the phoneme timeline, searching for the mouth shape category corresponding to the phoneme category in the mouth shape dictionary, and generating basic mouth shape entries; adding time boundary information to each basic mouth shape entry, with the time boundary consistent with the start and end time points of the phoneme record; writing all basic mouth shape entries into the main chain in chronological order to obtain a basic mouth shape sequence without transitions; any entry in the main chain only covers the time range of its own phoneme record and does not cross the time boundaries of adjacent phonemes.

[0016] Furthermore, in step 2, the generation process of the first side chain includes: dividing the phoneme time axis into syllables, the time range of which is determined by the minimum start time point and the maximum end time point of the continuous phoneme records constituting the syllable; within each syllable, the corresponding target frame numbers are concatenated according to the order of the start phase, transition phase, and steady-state phase to generate a segment label for a monotonic opening and closing trajectory; the segment label records the segment type, the start target frame number, and the end target frame number; within each syllable, a segment label for a lip shape change trajectory is generated according to the deterministic order of lip shape from contraction to relaxation or from relaxation to contraction. The segment label records the lip shape change type, the starting target frame number, and the ending target frame number; the segment labels of the monotonic opening and closing trajectory and the segment labels of the lip shape change trajectory are written into the first side chain in chronological order; no segment label may exceed the time range of its corresponding syllable; the generation process of the second side chain includes: checking the phoneme time axis line by line according to the articulation point contact rules to determine the set of phonemes that need to close, rub, or plosive; for phonemes that belong to bilip closure, marking the discrete event points of lip closure within the time range of the phoneme recording; the time position of the event point is preferentially taken from the starting time point and the ending time point of the phoneme recording. One or two; for phonemes involving tongue tip contact, mark discrete event sites of tongue tip contact within the time range of phoneme recording; the time position of the event site is preferably taken from the target frame number closest to the middle of the phoneme; for phonemes involving soft palate opening, mark discrete event sites of soft palate opening within the time range of phoneme recording; the time position of the event site is preferably taken from the target frame number near the start time point of phoneme recording; write the above discrete event sites into the second side chain in chronological order; bind each event site to a target frame number, and if the event time point is not equal to any target frame timestamp, assign it to the target frame number closest in time.

[0017] Further, in step 2, the lip-sync keyframe sequence is obtained through the following process: The overlapping regions of records at the temporal boundaries of adjacent phonemes are searched in the tri-chain merging table, and stitching segments are generated according to a fixed stitching strategy; the stitching segments exist only within the common boundaries of adjacent phonemes and do not cross the common boundaries; keyframes are marked at the phase boundaries, stitching boundaries, and event points based on the segment labels of the first sidechain and the discrete event points of the second sidechain; keyframes have discrete values ​​in the target frame number. For short-duration segments below the shortest duration standard appearing in the tri-chain merging table, they are merged into segments that are temporally adjacent and of the same category; the merging operation does not change the positions of the marked keyframes; all keyframes are combined in chronological order to form the lip-sync keyframe sequence.

[0018] Further, step 3 specifically includes: in each target frame of the original video, 3D facial feature points are calibrated according to a fixed facial geometry template; the head pose trajectory is obtained in the time sequence of the target frames, the head pose trajectory including the head rotation and head translation of the target frame; according to the frame index mapping table, each keyframe in the lip shape keyframe sequence is assigned to the corresponding target frame number; the head rotation and head translation of the target frame are read from the head pose trajectory and attached to the corresponding lip shape keyframe; between adjacent lip shape keyframes, spherical linear interpolation is used for head rotation, and linear interpolation is used for head translation to obtain a pose-consistent lip shape sequence; a mouth region mesh is generated on the target frame using 3D facial feature points as anchor points; based on... Based on the lip shape configuration of the frame in the pose-consistent lip shape sequence, perform analytical geometric deformation on the mesh and write it into the mouth region; when the original video does not have the required lip shape combination, select a fixed oral cavity geometry template and lip and teeth texture patch that match the current pose, and embed them into the mouth region after pose transformation; set a transition band of fixed width at the boundary of the mouth region and perform boundary feathering; use the facial reference region of the same frame as a reference to perform color transfer so that the mouth region and the reference region are consistent in brightness and chromaticity; use the phoneme time axis as the only master clock, write the replaced and processed mouth region back to the corresponding target frame, keeping the frame timestamp and resolution unchanged; output a video lip shape synthesis result with the same resolution as the original video and synchronized with the phoneme time axis.

[0019] Furthermore, when the time position of the lip-sync keyframe is not equal to the timestamp of any target frame, it is assigned to the target frame number with the smallest time difference; if there are two equidistant target frames, the one with the smaller time is selected; the pose-consistent interpolation is performed between adjacent lip-sync keyframes without crossing the common boundary of adjacent phonemes.

[0020] Compared with existing technologies, the advantages of this invention are: it can achieve high-precision alignment of audio and video in the temporal dimension and maintain consistency between lip shape and head posture in the spatial dimension. By using the phoneme time axis as the sole master clock, this invention avoids the problem of misalignment between lip movements and speech rhythm in traditional data-driven methods, significantly improving the temporal accuracy of lip synthesis results. Based on this, each phoneme is divided into three phases—initial, transitional, and steady-state—through phoneme slicing and phase calibration. Three complementary derivation chains are established by combining a fixed lip shape dictionary and articulation point contact rules, realizing structured modeling of the articulation process and effectively solving the problem of lacking physical explanation of speech configuration in existing methods. This invention also uses a keyframe adjudication mechanism to calibrate discrete keyframes at seam boundaries, event points, and phase transitions, constructing a traceable and interpolable lip shape keyframe sequence. Furthermore, by combining 3D facial feature point extraction and head posture estimation, this invention synchronizes the lip shape sequence with head rotation and translation parameters in each frame, ensuring that the generated lip shape maintains posture consistency during dynamic processes and avoiding visual jumps and configuration drift. Furthermore, this invention utilizes high-precision geometric mesh deformation and texture fusion technology to achieve a natural transition between the mouth area and the original video in terms of boundaries, color, and resolution, ensuring that the generated video possesses high definition and realism. Overall, this invention balances language structure-driven performance, temporal accuracy, visual continuity, and speech matching accuracy, making it suitable for applications such as digital human broadcasting, speech-to-video conversion, virtual character-driven applications, and other scenarios requiring high lip-sync accuracy and image quality. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a phoneme time-axis driven method for automatic lip-sync synthesis in high-definition video, as proposed in this invention.

[0022] Figure 2 This is a schematic diagram of the phoneme time axis and frame index mapping table in an embodiment of the present invention;

[0023] Figure 3 This is a schematic diagram illustrating the working principle of the three complementary derivation chains in this embodiment of the invention;

[0024] Figure 4 This is a graph showing the phoneme phase slice experiment in an embodiment of the present invention. Detailed Implementation

[0025] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0026] Reference Figure 1 As shown, a method for automatic lip-sync synthesis in high-definition video driven by a phoneme time axis is disclosed, the method comprising:

[0027] Step 1: Obtain the input text or speech, generate a phoneme sequence and corresponding start and end time points using a fixed set of phonemes and deterministic pronunciation rules, and establish a phoneme timeline; using the timestamp of the original video as the subordinate clock, construct a frame index mapping table that corresponds one-to-one between the target frames of the original video and the phoneme timeline; output the phoneme timeline, the frame index mapping table, and the target frame set.

[0028] Step 1 first establishes a unified data entry point for input text or speech. When the input type is text, a fixed set of phonemes and deterministic pronunciation rules are called through a preset word-to-phoneme mapping module to generate a phoneme sequence and corresponding start and end time points. When the input type is speech, an end-to-end forced alignment model is first used to segment the continuous speech stream into segments. Then, the same set of fixed phoneme sets and deterministic pronunciation rules are used to perform acoustic analysis on each segment to obtain phoneme identifiers and synchronously output start and end time points. Finally, all phoneme records are written in chronological order on a linear time axis to establish a phoneme time axis. Since the phoneme time axis is the sole master clock source for all subsequent timing operations, time accuracy and stability must be guaranteed during the generation process. Therefore, this embodiment adopts a double-buffered writing method: the main buffer is used to record the start and end time points of the current phoneme in real time, and the secondary buffer is used to temporarily store the gap information between the end time of the previous phoneme and the start time of the current phoneme. If the gap length is less than half the length of a single frame, it is directly merged into the current phoneme, thereby avoiding the appearance of unmapped zero-length gaps in the phoneme time axis. After the phoneme timeline is established, the system initiates the coupling process with the original video, sets the original video decoder to frame-by-frame fetch mode and reads the native timestamp of each frame. This timestamp is then regarded as a reference node with the original video timestamp as the subordinate clock.

[0029] Based on this, the system extracts target frames from the original video frame by frame. The target frame selection strategy can be set to full-frame acquisition or acquisition at a fixed downsampling rate as needed, as long as a target frame with a time difference of no more than half a frame can be found at every time point within the coverage of the phoneme timeline. Subsequently, the system constructs a frame index mapping table that corresponds one-to-one with the target frames of the original video to the phoneme timeline. Specifically, after parsing the timestamp of the target frame, a binary search is used to locate the phoneme record to which the timestamp belongs in the phoneme timeline. If the timestamp of the target frame falls between the start and end times of a certain phoneme record, the mapping relationship between the target frame number and the phoneme identifier is recorded. If the timestamp of the target frame falls exactly at the boundary of two adjacent phoneme records, the target frame number is mapped to the phoneme identifier with the smaller time difference based on the comparison result with the start times of the two phonemes. At the same time, the target frame timestamp field is retained in the mapping table entries for subsequent phase operation calls. To improve processing throughput in long-duration high-definition video scenarios, the mapping table construction process employs a partitioned parallel mechanism. First, the target frame set is divided into equal-length segments according to the original video decoding order. Each segment is handled by an independent thread, which reads the target frame timestamp, queries the phoneme timeline, and generates mapping entries. Then, the results from each segment are merged in global time order. Once all mapping entries are written, a complete frame index mapping table is obtained. Thus, this embodiment outputs the phoneme timeline, frame index mapping table, and target frame set through the above process. This lays a unified and precise timing foundation for subsequent phoneme slicing and phase calibration under the constraints of the phoneme timeline, ensuring that subsequent steps maintain frame-level synchronization accuracy when processing large-scale high-definition content and avoiding the impact of accumulated errors on lip-sync quality.

[0030] Step 2: Under the constraints of the phoneme time axis and frame index mapping table, phoneme slicing and phase calibration are performed sequentially. Based on a fixed and limited lip shape dictionary and articulation rules, three complementary derivation chains are established. The conflict resolution and key frame determination are completed by using the principle of non-feedback single forward adjudication and shortest interval preservation, so as to obtain a lip shape key frame sequence that satisfies phase continuity and boundary traceability.

[0031] Step 2 uses the phoneme timeline and frame index mapping table as strict timing constraints. First, it enters the phoneme slicing stage. The system reads the phoneme records arranged continuously on the phoneme timeline and divides the duration of each phoneme into three types of phoneme slices in memory: start phase, transition phase, and steady-state phase. The phase type label is written to the slice by looking up the table. Then, the frame index mapping table is called to match the target frames within the time range of the phoneme record one by one. The target frame number is uniquely assigned to its phoneme slice and the phase type is recorded synchronously, thereby completing the phase calibration and generating a phase-labeled frame column. In this process, all write operations follow a dual buffer strategy of real-time writing and delayed verification to ensure phase continuity. Next, the system loads a fixed and finite lip shape dictionary and articulation rules, and initializes three complementary derivation chains. The main chain, based on a one-to-one mapping from phonemes to lip shape categories, sequentially writes basic lip shape entries without transitions and adds time boundary information as it traverses the phoneme timeline. This ensures that any entry in the main chain only covers the time range of its own phoneme record and does not cross the boundaries of adjacent phonemes. The first side chain, within the syllable time range output by the syllable segmentation module, sequentially connects the initial phase, transition phase, and steady-state phase of the same syllable to generate monotonic opening and closing trajectory segment labels. Within the same syllable, it determines the lip shape based on whether it is contracted or relaxed. The system generates lip shape change trajectory segment labels in the order of relaxation to contraction. Both types of segment labels are strictly written into the first sidechain and must not exceed the time range of their respective syllables. The second sidechain, based on the rules of articulation, determines the set of phonemes that need to close, fricative, or plosive while traversing the phoneme timeline, and writes events such as bilip closure, tongue tip contact, and soft palate opening into discrete event sites. The target frame number selection strategy for event sites prioritizes the start time point, end time point of the phoneme record, or the target frame number near the middle of the phoneme. If the event time point cannot be strictly aligned with any target frame timestamp, it is assigned to the target frame with the smallest time difference to ensure event traceability. After the independent generation of the three chains is completed, the system performs a three-way chain merging operation under a unified temporal coordinate system, merging the mouth shape category of the main chain, the segment labels of the first sidechain, and the event sites of the second sidechain into a three-way chain merge table according to the phoneme timeline. The merge table uses a time index plus category segment double-key index mode in its storage layout to facilitate subsequent retrieval. Next, the system enters the single forward adjudication phase without feedback. When traversing the ternary chain merging table, if an overlapping area of ​​records is found at the time boundary of adjacent phonemes, the conflict detection module is triggered. The module first generates a stitching segment according to a fixed stitching strategy and ensures that the stitching segment exists only within the common boundary of adjacent phonemes and does not cross the common boundary. Then, according to the shortest interval preservation principle, the stitching segment and the phase segment are compared by weight. The segment with shorter duration is selected and retained. The longer segment is truncated at the boundary without affecting the rest. The whole process does not require backtracking or secondary adjustment of the adjudicated result, ensuring that the computational complexity increases linearly.The merged table, after adjudication, is sent to the keyframe calibration module. This module combines the segment labels of the first sidechain with the discrete event points of the second sidechain to calibrate keyframes at each phase boundary, seam boundary, and event point. The position of the keyframe is a discrete value in the target frame number and must be consistent with its bound timestamp. For short segments in the merged table that are below the shortest duration standard, the system immediately calls the shortest interval merging subroutine to merge them into segments that are temporally adjacent and of the same category. During the merging process, the already calibrated keyframe positions are not moved, thus avoiding the short segments from disrupting the continuity of lip movements. All keyframes are automatically combined in memory in chronological order to form the lip-sync keyframe sequence. The system also performs an integrity check on the keyframe sequence to ensure that the sequence covers the entire phoneme timeline and that there are no duplicate or missing timestamps. Thus far, step 2, through a continuous process including phoneme slicing, phase calibration, construction of three complementary derivation chains, feedback-free single forward adjudication, conflict resolution based on the shortest interval preservation principle, and keyframe localization, outputs a lip-sync keyframe sequence that satisfies phase continuity and boundary traceability, laying an accurate and stable keyframe foundation for subsequent pose-consistent interpolation and high-definition lip-sync synthesis.

[0032] Step 3: Extract 3D facial feature points and head pose trajectory from the original video, register the lip-sync keyframe sequence with the head pose trajectory according to the frame index mapping table to obtain a pose-consistent lip-sync sequence; output a video lip-sync synthesis result with the same resolution as the original video and synchronized with the phoneme time axis based on the pose-consistent lip-sync sequence.

[0033] In the implementation of this invention, step 3 uses the lip shape keyframe sequence output in step 2 as the only lip shape reference. First, the target frame is extracted frame by frame in the original video decoding thread according to the time order of the frame index mapping table. In each target frame, a fixed face geometry template is called to perform multi-scale feature fitting. The three-dimensional face feature point set is obtained by using the predefined contour, facial features and perioral control points on the template through minimum residual iteration calculation. At the same time, the head pose parameters are extracted in real time during the projection matrix inverse calculation process. The head pose parameters are composed of rotation vector and translation vector and correspond one-to-one with the timestamp of the target frame, thus forming a complete head pose trajectory. Next, the system calls the frame index mapping table to precisely bind each keyframe in the lip-sync keyframe sequence to the original video timeline according to the target frame number. It synchronously reads the rotation vector and translation vector of the corresponding target frame and writes them into the keyframe record, thereby generating the keyframe queue with the initial posture. To ensure the temporal continuity between lip-sync movement and head movement, the system uses spherical linear interpolation for the rotation vector and linear interpolation for the translation vector in the adjacent interval of the keyframes. The interpolation results are written to the newly generated posture-consistent lip-sync sequence in real time. This sequence is of equal length to the phoneme timeline on the timeline, and the interpolated values ​​at the keyframes are completely consistent with the original parameters.

[0034] Subsequently, the system, driven by a pose-consistent lip-shape sequence, constructs a mouth region mesh on each target frame using 3D facial feature points as anchors. Mesh vertices correspond one-to-one with template control points. The system then searches for the analytical geometric deformation parameters corresponding to the lip shape category in the lip shape mapping module. Based on these parameters, it performs vertex-by-vertex displacement and scaling operations on the mesh vertex coordinates. After deformation in the local coordinate system, the mesh region is mapped back to the target frame image coordinate system via a head pose rotation matrix and translation vector. When a native mouth texture matching the current lip shape category exists in the original video frame, the system prioritizes using the native texture to avoid unnecessary pixel replacement. Otherwise, it selects the template model and its lip and teeth texture patch that best match the current head pose from a fixed oral cavity geometry template library. After pose transformation and resolution consistency processing, these are embedded into the mouth region. To reduce replacement... The system automatically generates a fixed-width transition band at the boundary of the mouth area, and achieves boundary feathering through Gaussian weighted fusion. At the same time, it performs color transfer using the brightness and chromaticity distribution of the facial reference area in the same frame as the target, so that the mouth area and facial skin are consistent in brightness and chromaticity channels. After the above operations are completed in the image buffer, the target frame is immediately written back, keeping the timestamp and resolution of the target frame unchanged, and the result frame is sent to the output buffer. When all target frames have been processed, the output buffer writes the final frame stream in the original video time order, forming a video lip-sync result with the same resolution as the original video and synchronized with the phoneme time axis. The lip shape of this result at any moment is directly driven by the phoneme time axis and is completely consistent with the head posture in space, thus ensuring the stability and consistency of lip movement logic, visual coherence and high-definition rendering accuracy in long-duration complex corpus scenarios.

[0035] Furthermore, within the framework of this invention, the core motivation for phoneme slicing and phase calibration lies in accurately describing the evolutionary law of morphological changes in the articulatory organs using the smallest observable temporal unit. Therefore, the system actively divides the duration of each phoneme into three types of phoneme slices: the initial phase, the transition phase, and the steady-state phase. The initial phase corresponds to the instantaneous convergence segment where the articulatory organs initiate a transition from the previous target configuration to the specific configuration of the current phoneme. At this time, the lips, tongue tip, or soft palate are in the initial stage of rapid closing, lifting, or opening movements. The transition phase describes the dynamic segment where the articulatory organs traverse between two stable configurations, where the rate of change in the vocal tract morphology is the highest and it mixes the synergistic features of the preceding and following phonemes. The steady-state phase covers the segment where the articulatory organs have reached the current phoneme configuration and remain relatively still in that configuration. The airflow pattern and formant position tend to be stable, and the acoustic characteristics exhibit quasi-constant output. The system calculates the total duration for each phoneme by reading the phoneme timeline, then divides the memory into three boundary segments using a proportional strategy. It then retrieves all target frames within these three segments based on the frame index mapping table, writes the target frame number to the corresponding phoneme slice, and records the phase type label in an additional field. To avoid cross-phoneme confusion, target frame assignment follows a unique ownership principle; once a frame is identified as being in a specific phoneme slice, it will not appear in other phoneme slices, thus forming a temporally non-overlapping sequence of phase-labeled frames. Specifically, the bilabial plosive / p / has a visually typical strong onset characteristic: the lips close rapidly and muscle tension rises quickly during the onset phase; the transition phase is characterized by a sudden opening of the lips accompanied by a slight lip rebound; and the steady-state phase is almost negligible because the lips immediately transition to the next phoneme configuration after the airflow is released. Therefore, when processing / p / , the onset phase occupies most of the phoneme duration and corresponds to the largest number of target frames. Taking the nasal / m / as an example, its initial phase involves the subtle movement of the lips from half-closed to fully closed. The transition phase is the process of the soft palate lowering to open the nasal passage. The steady-state phase is characterized by the lips remaining closed while the cheeks slightly bulge. Therefore, the number of steady-state phase frames in the phase-annotated frame series is significantly greater than that of the initial and transition phases. For the vowel / a / , the initial phase is characterized by an increased speed of jaw descent and lip widening. The transition phase is characterized by the continued expansion of the oral cavity volume and its rapid approach to the resonance position. The steady-state phase is characterized by the oral cavity maintaining a constant longitudinal opening and the tongue surface being fixed in the low region. When selecting frames, the system detects the two local extremes of the lowest point of the tongue back and the largest point of the labial opening, and adds adjacent target frames to the steady-state slice accordingly, so that the phase-annotated frame series stably covers the entire vowel segment. Through the above-mentioned refinement process, the three types of phoneme slices are not only accurately separated in time, but also the phased patterns of vocal organ deformation are captured at the visual dynamics level. This provides a high-resolution temporal reference for the mapping, conflict resolution, and keyframe localization of the three subsequent complementary derivation chains, ensuring that the final synthesized lip movements can maintain physical consistency and visual coherence at both the camera and speech scales.

[0036] Furthermore, when the system enters the stage of constructing three complementary derivation chains, the primary prerequisite is that a fixed and finite lip shape dictionary and articulation rules have been loaded, and the phoneme timeline and phase annotation frames are in a state of random access in memory. The generation logic of the main chain is based on a one-to-one mapping from phonemes to lip shape categories. It focuses on the central configuration presented by each phoneme in a steady-state phase. Therefore, when traversing the phoneme timeline, it directly retrieves the lip shape category paired with the current phoneme identifier from the lip shape dictionary, and then writes the lip shape category along with the start and end times of the phoneme record into the main chain entry. The resulting basic lip shape sequence without transitions is arranged by phoneme in time sequence and will not overlap. Taking the syllable "ma" as an example, the phonemes / m / and / a / are mapped to the closed-lip and open-lip mouth shapes, respectively. Since the steady-state duration of / m / is extremely short while that of / a / is relatively long, the main chain shows that the duration of the closed-lip mouth shape is extremely narrow while the open-lip mouth shape occupies a significantly longer time range, visually indicating the necessity of the subsequent transition.

[0037] The first sidechain handles the phonological connection function, taking syllable segmentation results and phase-labeled frames as input. Within the same syllable, the system sequentially concatenates the target frame numbers of the initial phase, transition phase, and steady-state phase, writing a segment label and indicating the segment type for each phase continuum. If multiple opening-closing cycles occur within the same syllable, for example, the diphthong "ai" undergoes a significant contraction-extension process from / a / to / i / , the first sidechain generates two monotonous opening-closing trajectory labels: the first labels the rapid descent of the mandible until the oral cavity reaches its maximum volume, and the second labels the elevation of the tongue tip and the lip contraction until the closure stabilizes. Taking the bilabial plosive sequence "ba" as another example, the transition phase of / b / overlaps with the initial phase of / a / , and in the first sidechain, this is considered a short-duration but large-amplitude opening-closing segment label; the subsequent steady-state phase of / a / is labeled as a monotonous opening-closing segment label with slow lip relaxation. Each segment label records the starting target frame number, the ending target frame number, and their order within the syllable, thus providing sparse control points for subsequent curve fitting.

[0038] The second sidechain focuses on discrete articulation point events. It scans the phoneme timeline line by line according to articulation rules to determine whether bilip closure, tongue tip contact, or soft palate opening is triggered. For example, / p / satisfies both bilip closure and plosive conditions, so the system marks a bilip closure event point at the start of / p / and a plosive release event point near its end. For the fricative / s / , which corresponds to alveolar fricatives, the system inserts a tongue tip contact event point in the middle of the phoneme. The nasal / n / requires soft palate opening, so the system places the soft palate opening event point near the target frame number at the start of / n / . In the second sidechain, each event point is bound to a precise target frame number and includes an event type identifier; these discrete points will subsequently be directly converted into keyframes.

[0039] Once the three chains are written, the system initiates the tri-chain merging process. This process uses the phoneme timeline as a unified coordinate, aligning the main chain entries, the first side chain segment labels, and the second side chain event sites to the same timeline, and then sorting them according to timestamps to create a tri-chain merging table. The time index of the merging table points to the target frame number rather than absolute time, allowing for direct retrieval in the video frame domain later. For the "ma" example, the merging table will show overlapping records of three types at the common boundary between the end of / m / and the beginning of / a / : closed lip shape category entries, soft palate opening event sites, and lip opening / closing segment labels. In subsequent decision-making without feedback, the system determines the keyframe and adjudicates the stitched segment based on the shortest interval preservation principle. In the diphthong "ai" example, the merging table will show the continuous transition from open to closed lips, interspersed with multiple tongue tip contact event sites. These sites will be preferentially retained during the adjudication phase to ensure the traceability of tongue tip movements. Through this merge table, the system can integrate steady-state spatial information, dynamic temporal information, and articulation point contact information in the same data structure. This provides a full-dimensional context for subsequent keyframe localization and avoids the retrieval overhead caused by random cross-linking of multi-source linked lists. This fundamentally improves the computational efficiency and temporal consistency of high-definition lip-syncing automatic synthesis on long speech sequences.

[0040] Furthermore, before the system processing flow enters the main chain generation stage, the runtime environment first loads the phoneme timeline and frame index mapping table into the cache synchronously through memory mapping. Each phoneme record in the phoneme timeline stores its phoneme identifier, start time, and end time in the form of a structure, while each record in the frame index mapping table provides three fields: target frame number, corresponding phoneme identifier, and frame timestamp. This parallel design of two tables allows both time-sequential retrieval and random positioning to be completed within constant time complexity. Subsequently, the system calls the persistent resource manager to load the lip shape dictionary and articulation point rules. The lip shape dictionary is a fixed and finite set of mappings that categorizes all appearing phoneme categories into several distinguishable lip shape categories. For example, it maps / p / , / b / , and / m / to "closed lips," / f / to "lower lip teeth," and / t / , / d / , / n / , and / l / to "pre-alveolar," and so on. / 、 / The symbol / is mapped to "hard palate abrasion", / k / , / g / , and / h / are mapped to "soft palate opening", / a / is mapped to "maximum opening", / i / is mapped to "flattened lip", / u / is mapped to "rounded lip contraction", and / e / is mapped to "central opening". The mapping is "rounded lips half-open," so even with hundreds of phonemes, only about a dozen mouth shape categories need to be maintained, thus controlling the upper limit of the rendering template. The articulation point touch rules also exist in the form of a fixed set of rules. It defines Boolean conditions and event type mappings for five major articulation points: bilips, alveoli, hard palate, soft palate, and glottis. For example, the rule table specifies that "if the phoneme belongs to / p / , / b / , / m / , / ..." "If the phoneme belongs to / t / , / d / , / n / , / l / , then the tongue tip alveolar contact event is triggered." "If the phoneme belongs to / k / , / g / , / " / triggers the soft palate elevation and nasal cavity closure event." "If the phoneme belongs to / " " / then triggers the glottal closure event", and applies to / s / , / z / , and / / 、 / / fricative sounds are uniformly marked with gingival friction events, for / x / , / / etc., mark hard palate friction events. After loading, the system hashes the two tables into hash tables for constant-time lookups.

[0041] The main chain generation process begins with the first phoneme record on the phoneme timeline. The system reads the phoneme identifier and performs a lookup in the lip shape dictionary to determine the lip shape category of the phoneme. Then, basic lip shape entries are created, combining this category with the start and end times of the phoneme record and writing it into a pre-allocated sequential array. For phonemes that are temporally adjacent but have distinct articulation points, such as the sequence " / b / 0-100ms, / a / 100-300ms, / t / 300-350ms" on the timeline, the main chain will continuously write three basic lip shape entries: "lip closed 0-100ms," "maximum open 100-300ms," and "pre-alveolar 300-350ms." Each entry strictly covers the time range of its respective phoneme and does not cross the boundaries of adjacent phonemes. Since the main chain completely ignores transition phases, the deformation information from lip closed to open or from open to pre-alveolar in the above sequence will not appear. This dynamic will be supplemented in the first and second side chains. For example, the vowel strings " / i / 0-120ms, / u / 120-240ms" yield "flat lip contraction 0-120ms" and "rounded lip contraction 120-240ms" after dictionary lookup. The main chain maintains the hard boundary between the two without providing a connection path, thus providing a clear time switching point for subsequent feedbackless decisions. The main chain uses sequential write caching combined with a lockstep counter during writes to ensure that the time order of entries remains intact even in a multi-threaded environment. Simultaneously, an atomic bit marker is appended to the end of each entry to indicate that the entry has not yet undergone conflict detection between the first and second sidechains. This marker is set to zero during the subsequent ternary chain merging phase to indicate that the entry has been safely integrated into the global time order. Through this design, the main chain not only quickly provides a steady-state spatial reference for each phoneme in the entire corpus, but also uses a fixed and finite lip shape dictionary to compress high-dimensional pronunciation classification into a finite lip shape template, keeping the storage and computation scale of high-definition lip shape rendering within a predictable range. At the same time, the instantiation of the articulation point touch rule ensures that the subsequent discrete event points can be accurately aligned with the main chain's time boundary, reserving reliable anchor points for keyframe determination and high-precision transition interpolation.

[0042] Furthermore, in actual operation, the generation of the first sidechain is organized around the higher-level phonological unit of syllables. Its fundamental purpose is to abstract the opening and closing rhythms and lip-shape gradation patterns scattered among continuous phonemes into two clear segment sequences, thereby providing constraints for subsequent interpolation trajectory calculation. The system first calls an existing pinyin segmenter or IPA segmentation algorithm to divide the phoneme time axis into syllables. The division results are written to the buffer in chronological order. The time range of each syllable is directly determined by the minimum start time point and the maximum end time point of the internal continuous phoneme records, forming non-overlapping closed intervals in time sequence. Subsequently, the system reads the phase-labeled frame series within the same syllable. According to the natural order of the start phase, transition phase, and steady-state phase, these target frame numbers are sequentially concatenated to generate monotonic opening and closing trajectory segment labels. The segment type field "opening and closing trajectory" as well as the start target frame number and end target frame number are written into the label structure. For example, the syllable "ba" contains the transition of / p / and the steady state of / a / . The system identifies a monotonically increasing opening and closing trajectory from the closed lips to the full opening of the oral cavity. Its starting frame usually appears in the frame before the end of / p / , and the ending frame falls in the middle of the steady state of / a / . Then, based on the deterministic order of lip shape from contraction to relaxation or from relaxation to contraction, the system retrieves the extreme points of lip shape curvature within the same syllable. It performs a one-time global minimum and maximum value search on the curvature sequence using interpolation, and generates lip shape change trajectory segment labels with the target frame number near the extreme point as the boundary. The label structure contains the segment type field "lip shape change" as well as the starting target frame number and the ending target frame number. Taking the diphthong "ai" as an example, the initial descent of the mandible and the widening of the lips produce a lip shape change segment that slowly contracts to relaxes, while the subsequent elevation of the tongue tip and the re-contraction of the oral fissure produce a segment that relaxes to contract. Both segments are labeled in the first side chain and maintain the chronological order. When inserting segment tags, the system also performs boundary validity checks to ensure that the start and end target frame numbers of any tag do not exceed the time range of its corresponding syllable; if an out-of-bounds error is detected, it is automatically truncated to the syllable boundary. After the first sidechain is generated, each segment tag has a defined time period and a clear functional classification, providing a sparse control segment for continuous curve fitting for subsequent feedback-free adjudication.

[0043] Relative to the continuous trajectory abstraction of the first side chain, the second side chain focuses on capturing the discrete event of the contact points of the articulatory positions. The system traverses the phoneme time axis item by item according to the rules of the contact points of the articulatory positions. First, it looks up in the rule table whether the current phoneme meets the conditions such as bilabial closure, tongue tip contact, or velum opening. If it meets the conditions, it enters the process of marking the event sites. For phonemes with bilabial closure, such as / p / , / b / , / m / , the system will first generate the event site of lip closure at the starting time point of the phoneme record, and generate the explosion or release site near the ending time point; if the duration of the phoneme is short and there are less than two frames between the start and the end, only the starting site is retained. For phonemes that require the tongue tip to contact the alveolar ridge, such as / t / , / d / , / n / , / l / , the system calculates the midpoint of the phoneme time range and rounds it to the nearest target frame number, and marks this target frame number as the event site of tongue tip contact; the midpoint position is also applicable in the case of fricative sounds / s / , / z / , because the tongue tip has reached the position closest to the alveolar ridge during the fricative process. For nasal sounds with velum opening, such as / m / , / n / , / / , the system tends to select the target frame number closest to the reference frame near the starting point of the phoneme record to mark the event site of velum opening, so as to ensure that the opening of the nasal passage can occur before the start of acoustic nasalization. If the event time point is not equal to the time stamp of any target frame, the target frame number with the smallest time difference is found through binary search and this number is used to avoid the displacement error caused by subsequent interpolation. All event sites are sorted by time stamp and written into the second side chain. The linked list node contains three fields: event type, target frame number, and original time stamp. Taking the phrase "关" / g u a n / as an example, / g / triggers an event site of velum elevation and closure, / u / triggers an event site of lip constriction, / n / triggers a double event site of tongue tip-alveolar contact and velum opening. These discrete nodes do not overlap with each other in the time line, which not only meets the requirement of traceability of the contact points, but also provides a clear anchor point for key frame positioning.

[0044] When the first side chain and the second side chain each complete writing and pass the boundary check, the system can merge them with the main chain on the comprehensive time coordinate. The time series segments or discrete points carried by each of the three are uniformly mapped to the dimension of the target frame number, realizing the multi-channel information fusion of the steady-state configuration, dynamic transition, and contact events. The whole process maintains a one-way scan without backtracking, ensuring that the computational complexity is linearly related to the number of phonemes, and high-precision mouth key frames can still be output in real time in the case of high-definition long segments.

[0045] Furthermore, during runtime, once the three-way chain merging table is written and indexed in ascending order by target frame number, the system immediately starts a collision detection scanner. The scanner uses phoneme time boundaries as the segment index of a sliding window. Whenever the window crosses the common time interval of adjacent phonemes, it checks whether there is an overlap of lip shape category entries, opening / closing segment labels, or touch event points within that interval. If an overlap is detected, the stitching generator is triggered. The stitching generator follows a fixed stitching strategy: first, it calculates the duration of each entry within the overlapping segment, retains the entry with the shortest duration that is semantically compatible with adjacent segments, and truncates the remaining entries at the boundary to form a stitching segment with a limited length. The stitching segment is labeled as a "transition" category and exists only within the common boundary. After generation, it is immediately written back to the merging table without spreading to either side. The system then reads the monotonic opening and closing trajectory segment labels and lip shape change trajectory segment labels provided by the first sidechain, and combines them with the discrete event points given by the second sidechain to perform a single forward traversal of the merge table: when the cursor encounters the starting phase boundary, seam segment boundary, or event point, a keyframe is marked at the target frame number. To avoid excessive keyframe density leading to interpolation redundancy, the system performs distance threshold filtering on consecutive marked points; if the time interval between two adjacent points is less than half a frame, only the first point is retained. Next, the segment optimizer checks the duration of all segments in the merge table, merging short segments below the minimum duration standard into the longest segment that is temporally nearest and has the same lip shape category. The merging operation strictly maintains the target frame number of the marked keyframe unchanged; if the categories on both sides of a short segment are different, the preceding segment is merged first to ensure semantic continuity. Once the merging is complete, the system collects all keyframe numbers again in chronological order. The generated lip-sync keyframe sequence includes both the steady-state spatial anchor points of each phoneme and captures the rapid deformation inflection points caused by seam fragments and articulation events. The time intervals within the sequence are variable but not less than half a frame, ensuring that spherical linear interpolation and analytical geometric deformation do not introduce visible jitter or redundant calculations due to overly sparse or dense sampling. Through this process, which focuses on seam fragment control, event location protection, and minimum interval merging optimization, the system compresses multi-source temporal information into a series of discrete, traceable keyframes that cover the entire process, providing sufficient yet concise frame-level support for subsequent pose-consistent interpolation.

[0046] Furthermore, in the implementation architecture of this invention, the core objective of step 3 is to seamlessly integrate the lip-shape keyframe sequence containing only static spatial information with the actual head motion trajectory in the original video, so that the generated lip shape strictly obeys the phoneme time axis in the time dimension and maintains physical consistency with the facial posture in the spatial dimension. To this end, the system first pulls the target frame frame by frame in the original video decoding thread according to the frame index mapping table and calls a fixed face geometry template to perform multi-scale feature alignment; the template has determined a set of three-dimensional vertex coordinates that match the average head model height through air blowing experiments and laser scanning in the offline stage. During runtime, the template is projected onto the gray-level gradient extreme value region of the current frame through photometric consistency measurement and iterative nearest point algorithm, and the affine transformation is obtained with the goal of minimizing pixel residuals. Finally, a set of three-dimensional face feature points covering about sixty control points including the brow peak, orbital bone, bridge of the nose, philtrum, and chin is output. The system then uses the PnP solver in the perspective projection model to substitute the template coordinate system back to the camera coordinate system, obtaining the head rotation vector and head translation vector of the target frame and writing them into the head pose trajectory. Since each pose entry carries a native timestamp, subsequent interpolation can be performed directly on the actual sampling interval without resampling. Next, the system uses a hash mapping to assign each keyframe in the lip-sync keyframe sequence to the target frame number in the frame index mapping table. If the keyframe timestamp is not exactly equal to any target frame, a binary search is used to select the frame number with the smallest time difference for binding. If the absolute values ​​of the differences are the same, the smaller time difference is selected to ensure that the discrete frame numbers of the keyframes do not jump back and forth. After binding is completed, the pipeline enters the pose consistency stage: for the interval between two adjacent keyframes, the system calls the quaternion spherical linear interpolation function slerp() in rotation space to connect the two points along the unit four-dimensional sphere at a constant velocity, and at the same time calls the first-order linear interpolation function in Euclidean space to directly connect the translation vector. The interpolation interval strictly stops at the common boundary of adjacent phonemes to avoid cross-phoneme pose mixing. The resulting posture-consistent lip-sync sequence corresponds one-to-one with the original video target frame on the time axis. Each sequence unit carries three types of information: head rotation, head translation, and lip-sync category.

[0047] The rendering thread then uses a pose-consistent lip-shape sequence as a driving force to generate a local mouth region mesh on the corresponding target frame using 3D facial feature points as anchors. The mesh topology directly inherits the template oral cavity partition, and the vertex quality is controlled between two and four thousand through density adaptive sampling to balance deformation accuracy and raster partitioning efficiency. The system reads the lip shape category of the current frame from the lookup table and retrieves the parsed geometric deformation parameter set, which describes the displacement direction and scaling factor of the lip margin, oral fissure, buccinator muscle, and jawbone. After performing a vertex-by-vertex coordinate transformation on the mesh, the system multiplies the deformed coordinates by the head rotation matrix and adds the translation vector, finally writing the mouth region pixel block in the target frame image coordinate system. If the original video frame already contains a native mouth texture that matches the target lip shape category, the native texture is directly mapped to the deformed mesh surface using a differential motion field algorithm; otherwise, the system obtains the oral cavity geometry model and its high-resolution lip and tooth texture patch that best match the current head pitch and yaw angles from a fixed oral cavity geometry template library, performs pose transformation and resolution consistency, and then embeds it into the mouth mesh. To eliminate replacement traces, the system extends a fixed-width transition band outside the grid boundary. Within this band, the original and new pixels are weighted and fused using Gaussian weighting coefficients. The width of the transition band is linearly proportional to the target frame resolution, ensuring that high-definition scenes are not excessively blurred while avoiding hard edges in low-resolution scenes. Subsequently, bilateral texture statistical matching is performed using the area from the nose wing to the cheekbone in the same frame as a reference domain. Separate point mapping is performed on the luminance and chrominance channels to ensure that the mouth area and the surrounding skin maintain consistency in average brightness and hue distribution, thus resolving color temperature drift caused by uneven lighting or multiple cameras.

[0048] After all pixel operations are completed in the off-screen buffer, the buffer is written back to the corresponding target frame memory page using the original video's timestamp as the primary key, ensuring that the frame order is consistent with the original encoding. Since the phoneme timeline is used as the sole master clock throughout, the entire rendering pipeline requires no additional adjustments to the frame rate or sampling interval, thus avoiding lip-sync audio desynchronization caused by dual-clock drift. After the last frame is written, the encoder performs lossless repackaging according to the original video container format and resolution. The output video file visually retains the original detail, and the lip-sync shape at any given time strictly corresponds to the corresponding phoneme, meeting the dual requirements of synchronization and clarity for applications such as long dialogues, emotional performances, and low-latency live streaming.

[0049] Figure 2This invention details the construction principle and correspondence of the phoneme timeline and frame index mapping table. The diagram consists of three parts: the phoneme timeline, the original video target frame sequence, and the frame index mapping table. In the phoneme timeline, five phonemes are arranged in chronological order: phoneme A, phoneme B, phoneme C, phoneme D, and phoneme E, corresponding to time intervals t0-t1, t1-t2, t2-t3, t3-t4, and t4-t5, respectively. Each phoneme has a defined start and end time, forming a continuous phoneme sequence. Phoneme A occupies the time interval t0-t1 with a relatively short duration; phoneme B occupies the time interval t1-t2 with a longer duration; phoneme C occupies the time interval t2-t3 with a short duration; phoneme D occupies the time interval t3-t4 with a medium duration; and phoneme E occupies the time interval t4-t5 with a medium duration. The target frame sequence section of the original video displays 11 target frames from F1 to F11, arranged with the timestamp of the original video as the subordinate clock. Each target frame has a fixed time position and frame number, forming a discrete frame sequence. The distribution density of the target frames is related to the frame rate of the original video, and the frame density may vary in different time periods. The frame index mapping table, as a core technical component, establishes a one-to-one correspondence between the phoneme timeline and the target frames of the original video. This mapping table contains three key fields: target frame number, corresponding phoneme identifier, and frame timestamp. Specifically, target frames F1-F3 correspond to phoneme A, with timestamps t0-t1; target frames F4-F5 correspond to phoneme B, with timestamps t1-t2; and target frames F6-F9 correspond to a combination of phonemes C, D, and E, with timestamps t2-t5. Red arrows clearly indicate the mapping relationship between the target frames and their corresponding phoneme intervals, achieving a precise correspondence between the continuous phoneme timeline and the discrete video frame sequence. This mapping mechanism ensures the accuracy of time synchronization during subsequent lip-sync synthesis, providing a time reference and frame-level indexing support for phoneme-driven automatic video lip-sync synthesis.

[0050] Figure 3This invention systematically demonstrates the working principle of three complementary derivation chains, one of its core technologies. This mechanism combines a main chain with two side chains to achieve multi-dimensional mapping and conversion from phonemes to mouth shapes. The phoneme timeline, located at the top of the diagram, contains five consecutive phonemes: / a / , / i / , / p / , / l / , and / e / . These phonemes are arranged according to their pronunciation order and temporal sequence, forming the basis of a complete phoneme sequence. The main chain, as the first derivation chain, implements a one-to-one mapping function from phonemes to mouth shape categories. Based on a fixed and finite mouth shape dictionary, the main chain converts the input phoneme sequence into the corresponding mouth shape sequence: phoneme / a / maps to mouth shape A, phoneme / i / maps to mouth shape I, phoneme / p / maps to mouth shape P, phoneme / l / maps to mouth shape L, and phoneme / e / maps to mouth shape E. This mapping process strictly follows deterministic pronunciation rules, ensuring that each phoneme obtains a unique and accurate mouth shape category correspondence. The base mouth shape sequence generated by the main chain does not include transition effects, providing a stable benchmark for subsequent processing. The first side chain is specifically responsible for phonological connection processing, performing phase analysis and trajectory generation for phonemes belonging to the same syllable. This side chain divides the phoneme sequence by syllable and generates two types of segment labels for each syllable: monotonic opening and closing trajectories and lip change trajectories. Syllable 1 contains monotonic opening and closing trajectories with onset phase, transition phase, and steady-state phase; syllable 2 contains the corresponding lip change trajectories. The processing of the first side chain ensures smooth transitions and continuity of connections between phonemes within the same syllable. The second side chain processes articulatory contact events based on deterministic contact rules of the bilips, alveoli, hard palate, soft palate, and glottis. This side chain identifies phonemes requiring special articulatory actions, such as the phoneme / p / requiring bilip closure and the phoneme / l / requiring tongue tip contact. The second side chain labels discrete articulatory events at corresponding time points, including event point 1 and event point 2, which precisely indicate the timing of specific articulatory actions. The three-way chain merging table, as the final output, merges the lip-shape categories of the main chain, the segment labels of the first side chain, and the event sites of the second side chain along the phoneme timeline. The merging process employs a feedback-free, single-forward adjudication principle to resolve potential conflicts between different derivation chains, generating a unified "main chain + first side chain + second side chain merged result." This merged result provides a complete technical foundation for subsequent keyframe determination and lip-shape sequence generation.

[0051] Figure 4The experimental data curves detail the dynamic changes in lip opening and closing during phoneme phase slicing and the technical mechanism of phase calibration. The graph uses a two-dimensional coordinate system, with the horizontal axis representing time and the vertical axis representing the quantified value of lip opening. The time axis extends from t0 to t5, covering the complete articulation cycle of five consecutive phonemes. The duration of each phoneme is scientifically divided into three different phase types: onset phase, transition phase, and steady-state phase. The phoneme / a / corresponds to the time interval t0-t1, mainly containing the onset phase; the phoneme / i / corresponds to the time interval t1-t2, containing the transition phase; the phoneme / u / corresponds to the time interval t2-t3, containing the steady-state phase; the phoneme / o / corresponds to the time interval t3-t4, again containing the transition phase; and the phoneme / e / corresponds to the time interval t4-t5, containing the onset phase. This phase division method fully considers the physiological characteristics of human articulation and the dynamic laws of lip shape changes. The mouth opening / closing curve is represented by a smooth, continuous curve, ranging from 0 to 1.0, objectively reflecting the quantitative changes in the degree of mouth opening and closing during articulation. The curve starts at an opening / closing level of 0.2, gradually rising to 0.6 with the pronunciation of the phoneme / a / , reflecting the articulation characteristics of an open phoneme. In the / i / phase, the opening / closing further increases to a peak close to 0.8, corresponding to the high opening / closing requirement of this phoneme. Subsequently, in the steady-state phase of the phoneme / u / , the opening / closing remains at a relatively stable level of around 0.8, reflecting the characteristics of a steady-state phase. In the / o / phase, the opening / closing begins to decrease to around 0.4, finally falling back to the level of 0.6 in the / e / phase. Five keyframe points are marked on the experimental curve, located at characteristic moments of each phase. Keyframe 1 is located at a feature point in the initial phase, with an opening degree of approximately 0.2; keyframe 2 is located at the peak point of the transition phase, with an opening degree reaching 0.8; keyframe 3 is located in the plateau region of the steady-state phase, with the opening degree remaining at 0.8; keyframe 4 is located in the falling transition phase, with the opening degree decreasing to 0.4; and keyframe 5 is located in the final starting phase, with the opening degree stabilizing at 0.6. The determination of these keyframes follows the principle of preserving the shortest interval and the requirement of phase continuity, providing accurate time anchors and numerical references for subsequent lip-sync interpolation and video synthesis. Phase boundaries are clearly marked by vertical dashed lines, ensuring boundary traceability and the accuracy of phase calibration. This experimental data verifies the effectiveness of the phoneme phase slicing algorithm, providing important theoretical support and experimental basis for phoneme time-axis driven lip-sync synthesis technology.

[0052] The following example illustrates the complete implementation of a phoneme time-axis driven high-definition video lip-sync automatic synthesis method in a practical engineering scenario. Let the input be a segment of video with a duration of... The original video, resolution Frame rate , while giving the Chinese test statement "Mom holds the baby" synchronized with the video. The system uses the International Phonetic Alphabet, and the corresponding phoneme sequence is recorded as . This article will demonstrate the calculation process with a fixed phoneme duration model, and each phoneme duration is set to , with a total of phonemes, satisfying .

[0053] First, establish a phoneme timeline. Define the start and end times of the th phoneme record . The main clock of the phoneme timeline . Then, construct a frame index mapping table. Let the time interval of a single frame of the video be , and the timestamp of the th frame is .

[0054] According to , assign a unique phoneme identifier to each frame, and obtain the mapping . Next, perform phoneme slicing and phase calibration. Use the ratio . For the phoneme : . Mark the starting phase at , mark the transition phase at , and mark the steady-state phase at ; determine the phase label for each frame through the timestamp landing point, and form a sequence of phase-annotated frames .

[0055] Load the mouth configuration dictionary : , , , . Load the contact rules for the place of articulation : Bilabial consonants trigger lip closure events, plosives trigger plosion release events, and rounded vowels trigger lip constriction events. Traverse the main chain and write the basic mouth shape sequence in turn. The first side chain gives the start and end frame numbers of the opening and closing sections and the lip shape change sections within the syllables "ma", "ma", "bao", "bao"; the second side chain generates event sites for / m / , / b / , / / , for example, record lip closure at the start frame of / m / , and record plosion release at the end frame of / b / . After merging the three chains, perform conflict detection. Assume that a time overlap is found between the main chain lip closure segment and the opening section of the first side chain at the common boundary of / m / and / / The stitching strategy uses shorter segments, retaining the open section and truncating the closed segment to the boundary. Keyframes are then marked at the phase boundary, stitching boundary, and event site. The minimum keyframe interval threshold is set to... Short-duration segments are insufficient. The segments are merged into neighboring segments of the same type. This ultimately yields the keyframe number sequence. .

[0056] Establish head pose trajectory. Perform for each target frame. Solving for the rotation quaternion yields the result. With translation vector If the interval between two keyframes Interpolation time ,but , subscript This represents the keyframes at the interval endpoints. The poses of all non-keyframes are calculated using this formula, confined to the common boundary of the same phoneme and not crossing the boundary. During the rendering stage, a four-layer subdivided mouth region mesh is generated for each frame using 3D facial feature points as anchor points. For vertices Perform analytical deformation ,in The scaling matrix is ​​obtained by looking up a table according to lip shape category. Let be a rotation matrix. These are translation vectors, and all three are predefined in the template coordinate system. If the current frame requires "rounded lips," then... , , After deformation, the vertex is then subjected to head posture. The transformation is written to the image plane. If the original frame does not have a matching texture, a rounded lip template texture from the library is used, with a texture resolution of [texture resolution missing]. The opening raster is embedded through perspective mapping. Boundary feathering uses a Gaussian kernel with a full width at half maximum (FWHM). Color transfer achieves average brightness in the mouth region through piecewise linear mapping. Aligned with the average brightness of the cheek reference area, mapping formula .

[0057] Maintain the original frame timestamp during the output stage With the resolution unchanged, the processed frame stream is passed to the H.265 encoder at a bitrate of [missing information]. It is packaged as an MP4 file. The final synthesized video is... A continuous movement from lip closure to maximum opening was observed, precisely synchronized with the phoneme / mɑ / , with an error of [missing information]. Meets the live streaming synchronization threshold Require.

[0058] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A method for automatic lip-sync synthesis in high-definition video driven by a phoneme time axis, characterized in that, The method includes: Step 1: Obtain the input text or speech, generate a phoneme sequence and corresponding start and end time points using a fixed set of phonemes and deterministic pronunciation rules, and establish a phoneme timeline; using the timestamp of the original video as the subordinate clock, construct a frame index mapping table that corresponds one-to-one between the target frames of the original video and the phoneme timeline; output the phoneme timeline, the frame index mapping table, and the target frame set. Step 2: Under the constraints of the phoneme timeline and frame index mapping table, phoneme slicing and phase calibration are performed sequentially. Based on a fixed and finite lip shape dictionary and articulation rules, three complementary derivation chains are established simultaneously. Conflict resolution and keyframe determination are completed using a feedback-free, single-forward adjudication and the shortest interval preservation principle, resulting in a lip shape keyframe sequence that satisfies phase continuity and boundary traceability. Step 2, based on a fixed and finite lip shape dictionary and articulation rules, establishes three complementary derivation chains: a main chain, a first side chain, and a second side chain. The main chain is a one-to-one mapping from phonemes to lip shape categories, used to... Generate a basic mouth shape sequence without transitions; the first side chain is used for phonological connection, sequentially connecting the initial phase, transition phase, and steady-state phase belonging to the same syllable to generate segment labels for monotonic opening and closing trajectories and lip shape change trajectories; the second side chain corresponds to the articulation point touchpoint, and is used to mark discrete event sites of lip closure, tongue tip contact, and soft palate opening for phonemes that need to close, rub, or plosive, based on the deterministic touchpoint rules of the lips, alveoli, hard palate, soft palate, and glottis; merge the mouth shape category of the main chain, the segment labels of the first side chain, and the event sites of the second side chain on the same timeline according to the phoneme time axis to obtain a tri-chain merge table; Step 3: Extract 3D facial feature points and head pose trajectory from the original video, register the lip-sync keyframe sequence with the head pose trajectory according to the frame index mapping table to obtain a pose-consistent lip-sync sequence; output a video lip-sync synthesis result with the same resolution as the original video and synchronized with the phoneme time axis based on the pose-consistent lip-sync sequence.

2. The method for automatic lip-sync synthesis in high-definition video driven by a phoneme time axis as described in claim 1, characterized in that, In step 2, the process of sequentially performing phoneme slicing and phase calibration includes: dividing the duration of each phoneme into three types of phoneme slices: initial phase, transition phase, and steady-state phase; according to the frame index mapping table, uniquely assigning each target frame to its corresponding phoneme slice and recording the phase type label to form a phase-labeled frame column.

3. The method for automatic lip-sync synthesis in high-definition video driven by a phoneme time axis as described in claim 1, characterized in that, Step 2 first reads the phoneme timeline, which consists of phoneme records arranged in chronological order. Each phoneme record contains a phoneme identifier and start and end times. Then, it reads the frame index mapping table, which provides the correspondence between target frames and the phoneme timeline. Each record contains the target frame number, the corresponding phoneme identifier, and the frame timestamp. Finally, it loads the lip shape dictionary and articulation point rules. The lip shape dictionary is a fixed and finite mapping set, providing a one-to-one mapping relationship between phoneme categories and lip shape categories. The articulation point contact rules are a fixed set of rules, which give the contact conditions and corresponding event types for the lips, gums, hard palate, soft palate and glottis.

4. The method for automatic lip-sync synthesis in high-definition video driven by a phoneme time axis as described in claim 3, characterized in that, In step 2, the process of generating the main chain includes: traversing each phoneme record in the order of the phoneme timeline, searching for the mouth shape category corresponding to the phoneme category in the mouth shape dictionary, and generating basic mouth shape entries; adding time boundary information to each basic mouth shape entry, with the time boundary consistent with the start and end time points of the phoneme record; writing all basic mouth shape entries into the main chain in chronological order to obtain a basic mouth shape sequence without transitions; any entry in the main chain only covers the time range of its own phoneme record and does not cross the time boundaries of adjacent phonemes.

5. The method for automatic lip-sync synthesis in high-definition video driven by a phoneme time axis as described in claim 4, characterized in that, In step 2, the generation process of the first side chain includes: dividing the phoneme time axis into syllables, the time range of which is determined by the minimum start time point and the maximum end time point of the continuous phoneme records constituting the syllable; within each syllable, according to the order of the start phase, transition phase, and steady-state phase, concatenating the corresponding target frame numbers to generate a segment label for the monotonic opening and closing trajectory; this segment label records the segment type, the start target frame number, and the end target frame number; within each syllable, according to the deterministic order of lip shape from contraction to relaxation or from relaxation to contraction, generating a segment label for the lip shape change trajectory; this segment label records the lip shape change type, the start target frame number, and the end target frame number; writing the segment labels of the monotonic opening and closing trajectory and the segment labels of the lip shape change trajectory into the first side chain in chronological order; no segment label may exceed the time range of its syllable; the generation process of the second side chain includes: according to the articulation point contact rules... Then, the phoneme timeline is checked line by line to determine the set of phonemes that need to be closed, fricative, or plosive; for phonemes that are bilip-closed, discrete event sites of lip closure are marked within the time range of the phoneme recording; the time position of the event site is preferably one or both of the start and end time points of the phoneme recording; for phonemes that are tongue-tip contact, discrete event sites of tongue-tip contact are marked within the time range of the phoneme recording; the time position of the event site is preferably the target frame number closest to the middle of the phoneme; for phonemes that are soft palate open, discrete event sites of soft palate open are marked within the time range of the phoneme recording; the time position of the event site is preferably the target frame number near the start time point of the phoneme recording; the above discrete event sites are written into the second side chain in chronological order; each event site is bound to a target frame number, and if the event time point is not equal to any target frame timestamp, it is assigned to the target frame number closest in time.

6. The method for automatic lip-sync synthesis in high-definition video driven by a phoneme time axis as described in claim 5, characterized in that, In step 2, the lip-sync keyframe sequence is obtained through the following process: The overlapping regions of records at the temporal boundaries of adjacent phonemes are searched in the tri-chain merging table, and stitching segments are generated according to a fixed stitching strategy; stitching segments exist only within the common boundaries of adjacent phonemes and do not cross the common boundaries; keyframes are marked at the phase boundaries, stitching boundaries, and event points based on the segment labels of the first sidechain and the discrete event points of the second sidechain; keyframes have discrete values ​​in the target frame number; for short segments below the shortest duration standard appearing in the tri-chain merging table, they are merged into segments that are temporally adjacent and of the same category; the merging operation does not change the positions of the marked keyframes; all keyframes are combined in chronological order to form the lip-sync keyframe sequence.

7. The method for automatic lip-sync synthesis in high-definition video driven by a phoneme time axis as described in claim 6, characterized in that, Step 3 specifically includes: In each target frame of the original video, 3D facial feature points are calibrated based on a fixed facial geometry template; the head pose trajectory is obtained in chronological order of the target frames, the head pose trajectory including head rotation and head translation of the target frame; each keyframe in the lip-sync keyframe sequence is assigned to the corresponding target frame number according to the frame index mapping table; the head rotation and head translation of the target frame are read from the head pose trajectory and attached to the corresponding lip-sync keyframe; between adjacent lip-sync keyframes, spherical linear interpolation is used for head rotation, and linear interpolation is used for head translation to obtain a pose-consistent lip-sync sequence; a mouth region mesh is generated on the target frame using 3D facial feature points as anchor points; and the lip-sync region mesh is generated based on the pose. The lip shape configuration of the frame in the unified lip shape sequence is analyzed by performing analytical geometric deformation on the mesh and writing it into the mouth region. When the required lip shape combination does not exist in the original video, a fixed oral cavity geometry template and lip and teeth texture patch that match the current pose are selected and embedded into the mouth region after pose transformation. A transition band of fixed width is set at the boundary of the mouth region, and boundary feathering is performed. Color transfer is performed based on the facial reference region in the same frame to ensure that the mouth region and the reference region are consistent in brightness and chromaticity. Using the phoneme timeline as the sole master clock, the replaced and processed mouth region is written back to the corresponding target frame, keeping the frame timestamp and resolution unchanged. The output is a video lip shape synthesis result with the same resolution as the original video and synchronized with the phoneme timeline.

8. The method for automatic lip-sync synthesis in high-definition video driven by a phoneme time axis as described in claim 7, characterized in that, When the time position of a lip-sync keyframe is not equal to the timestamp of any target frame, it is assigned to the target frame number with the smallest time difference; if there are two equidistant target frames, the one with the smaller time is selected; the pose-consistent interpolation is performed between adjacent lip-sync keyframes and does not cross the common boundary of adjacent phonemes.

Images