Text-driven virtual character lip shape synchronization deep learning method

Abstract

The invention discloses a text-driven virtual character lip shape synchronization deep learning method, and relates to the technical field of artificial intelligence, and the method comprises the steps: recognizing format control characters in an original text to generate a perforation mask, calculating a perforation field, and extracting a visible sequence; inputting the visible sequence and the aligned perforation field into a depth sequence network, and outputting repair characterization and repair uncertainty; generating pronunciation unit embedding and basic duration based on the repair representation, and obtaining final unit duration by using repair uncertainty modulation; and finally, generating a frame-level mouth shape parameter sequence in combination with the restoration uncertainty, and mapping the frame-level mouth shape parameter sequence into a According to the invention, the high-frequency jump of the mouth shape can be suppressed, and the robustness of live broadcast digital human lip shape synchronization is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a deep learning method for text-driven virtual character lip-syncing. Background Technology

[0002] With the development of artificial intelligence and computer vision technologies, virtual digital humans have been widely used in scenarios such as live-streaming e-commerce, interactive bullet comments, and product information broadcasting. These systems typically receive text generated from live-streaming bullet comments or product information streams as input. They generate driving parameters through articulation unit modeling and duration prediction, thereby controlling the virtual character's mouth movements to achieve speech animation. However, in practical applications, the source of input text is complex and non-standard. In addition to conventionally visible characters, the text often contains Unicode format control characters invisible to the human eye. Although these characters are not easily detected at the display level, they alter the underlying discrete structure of the string, causing invisible breaks between the visual content and the computer processing sequence. This structural difference disrupts the continuity of word segmentation and sub-word splitting, inducing unknown segments or boundary anomalies in the articulation unit generation chain. Consequently, the final generated lip-sync animation exhibits unnatural jitter or temporal misalignment, severely impacting the viewing experience of the digital human.

[0003] Existing technologies typically employ rule-based preprocessing or simple regular expression filtering when processing such non-standard text. These methods often fail to maintain the integrity of the text index, easily leading to the loss of original positional information and causing inaccurate alignment in subsequent driving stages. Furthermore, existing deep learning models lack explicit measurement mechanisms for such hidden interference, making it impossible to effectively calculate the perturbation intensity of these invisible characters on neighboring sequences and to correct ambiguities. Due to the lack of quantification and modulation methods for this structural uncertainty, existing models cannot adaptively adjust their generation strategies based on the degree of interference during duration expansion and parameter generation. This results in high-frequency jumps or neutralization collapses in lip-sync parameters when facing noisy inputs, making it difficult to meet the requirements of high robustness and naturalness for virtual character lip-syncing in live streaming scenarios. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies that cannot effectively handle the disturbance of sequence structure caused by hidden perforations formed by format control characters in input text, resulting in ambiguity in the pronunciation unit modeling and duration expansion stages, and thus causing high-frequency jumps and unnatural timing in the generated virtual character lip-sync animation. The proposed invention is a text-driven deep learning method for virtual character lip-sync.

[0005] To address the problems existing in the prior art, the present invention adopts the following technical solution: Text-driven deep learning methods for lip-syncing in virtual characters include: S1. Obtain the original text character sequence and identify the format control characters in the original text character sequence to generate a punch mask; S2. Calculate the punch field based on the punch mask, and extract the visible sequence and the aligned punch field aligned with the visible sequence from the original text character sequence; S3. Input the visible sequence and the aligned perforation field into the depth sequence network, and output the repair characterization and repair uncertainty; S4. Generate pronunciation unit embeddings and unit base durations based on the repaired representation, and adjust the unit base durations according to the repair uncertainty to obtain the final unit durations; S5. Based on the final unit duration, the pronunciation unit is embedded in the time dimension and expanded, and combined with the repair of uncertainty, a frame-level lip-sync parameter sequence is generated. S6. Map the frame-level lip-sync parameter sequence to virtual character driving variables.

[0006] Preferably, identifying format control characters in the original text character sequence to generate a punch mask includes: Iterate through each character of the original text character sequence; Determine whether the current character belongs to the preset format control character set. If it does, set the value of the punch mask at the corresponding position to 1; otherwise, set the value of the punch mask at the corresponding position to 0.

[0007] Preferably, the process involves calculating the punch field based on the punch mask and extracting the visible sequence and the aligned punch field aligned with the visible sequence from the original text character sequence, including: The sliding window size is adaptively determined based on the length of the original text character sequence; For each position in the original text character sequence, calculate the cumulative value of the punch mask within the coverage area of ​​the sliding window; The accumulated values ​​are normalized to obtain the perforation field; Extract the characters corresponding to the positions where the punch mask is 0 to form a visible sequence, and extract the values ​​of the punch field at the same positions to form an aligned punch field.

[0008] Preferably, the visible sequence and the aligned perforation field are input into the depth sequence network, including: The visible characters in the visible sequence are embedded into vectors to obtain character embeddings, and corresponding time position codes are generated. Map the aligned perforation field to a perforation feature vector; Character embedding, temporal location encoding, and perforation feature vector are concatenated and input into a deep sequence network.

[0009] Preferably, the output includes the repair characterization and repair uncertainty, including: The repair probability distribution at each time step is output through a deep sequence network; The repair probability distribution is weighted and summed to obtain the repair characterization. Calculate the entropy of the repair probability distribution and use the entropy as the repair uncertainty.

[0010] Preferably, adjusting the unit base duration based on the repair uncertainty includes: Calculate the mean of the repair uncertainty for all time steps in the visible sequence; For each time step, calculate the ratio of the repair uncertainty of the current time step to the mean of the repair uncertainty to obtain the modulation factor; The final unit duration is obtained by multiplying the unit base duration by the modulation factor.

[0011] Preferably, the frame-level lip-sync parameter sequence is generated by combining the repair of uncertainties, including: The number of frames corresponding to each pronunciation unit is determined based on the final unit duration; Generate the relative phase of each frame within its respective phonological unit and convert it into a phase embedding; Map the repair of uncertainty to uncertainty embedding; The articulation unit embedding, phase embedding, and uncertainty embedding are concatenated and input into a deep generative network to obtain a frame-level lip shape parameter sequence.

[0012] Preferably, mapping the frame-level lip-sync parameter sequence to virtual character driving variables includes: By using a linear mapping matrix and a bias vector, the frame-level lip-sync parameter sequence is converted into the skeletal control variables or facial expression blending shape coefficients of the virtual character.

[0013] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention identifies format control characters in the original text and constructs a punch mask, enabling precise location of invisible but structurally disruptive hidden interference. Based on this, a sliding window is adaptively constructed according to the text length to calculate the punch field, and the normalized punch field is aligned to the visible character sequence. This transforms discrete, invisible punch perturbations into continuous, measurable punch feature vectors. This explicit modeling mechanism allows deep sequence networks to simultaneously perceive visible text content and neighborhood punch intensity when modeling phonetic units, effectively reducing repair ambiguities caused by abnormal input text structure, thereby improving the model's robustness to non-standard text input.

[0014] 2. This invention quantifies the repair uncertainty by calculating the entropy of the repair probability distribution and uses this uncertainty to dynamically modulate the unit base duration of the pronunciation unit. When a high level of repair ambiguity is detected, a larger modulation factor is automatically generated to extend the unit duration. This mechanism makes the pronunciation duration allocation more conservative and smooth in areas with severe piercing or high uncertainty. At the same time, in the frame-level generation stage, uncertainty is embedded as a conditional input, which explicitly suppresses high-frequency jumps in lip-shape parameters, ensuring the temporal continuity and naturalness of the generated frame-level lip-shape parameter sequence. This avoids lip-shape jitter or neutralization collapse caused by hidden piercings and significantly improves the lip-sync viewing experience of virtual characters. Attached Figure Description

[0015] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating a text-driven virtual character lip-syncing deep learning method provided in an embodiment of the present invention. Detailed Implementation

[0016] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0017] Example: This example provides a text-driven deep learning method for lip-syncing of virtual characters. See [link to example]. Figure 1 Specifically, including: S1. Obtain the original text character sequence and identify the format control characters in the original text character sequence to generate a punch mask; In embodiments of the present invention, identifying format control characters in the original text character sequence to generate a punch mask includes: Obtain the original text character sequence; Iterate through each character of the original text character sequence; Determine whether the current character belongs to the preset format control character set. If it does, set the value of the punch mask at the corresponding position to 1; otherwise, set the value of the punch mask at the corresponding position to 0. Specifically, the original text character sequence refers to the ordered discrete symbol sequence extracted character by character from text input such as live stream comments or product information streams in the order of appearance. Each character corresponds to a unique position index in the sequence. This sequence includes both user-visible alphanumeric characters, Chinese characters, and punctuation marks, as well as format control characters that may be inserted and are difficult to detect at the display level. This makes the same visual content appear as a position-dependent discrete structure on the computer processing side. The punch mask is a binary marker sequence aligned one-to-one with the original text character sequence. It is used to characterize whether there are hidden punch points formed by format control characters at each character position. The mask value of 1 indicates that the position is occupied by a format control character and forms a break source of text continuity. The mask value of 0 indicates that the position is a normally visible character and maintains the continuous carrying relationship of the text. The punch mask is generated by traversing the original text character sequence and performing set belonging judgment on each character, thereby providing a clear and reproducible position indication for subsequent quantitative calculation of punch distribution and perturbation suppression of lip-sync generation.

[0018] Specifically, after receiving the text to be broadcast from the live stream chat interface or the product information stream interface, the text to be broadcast is normalized into a Unicode encoding sequence according to the unified character encoding standard. This Unicode encoding sequence is then decoded one by one in code point order to obtain the original text character sequence. Each character in the original text character sequence corresponds to a sequence position index that increments from the beginning. Subsequently, a punch mask is initialized in memory as a binary sequence of the same length as the original text character sequence, and all positions are initialized to zero to ensure that positions not recognized as format control characters are treated as normal characters by default. Then, the original text character sequence is traversed character by character using the sequence position index as a loop variable. When the current character is encountered, its Unicode category attribute is read and compared with the format control... The character set is assigned to a specific category. Preferably, the format control character set is a set of characters marked as a format class in the Unicode general category, and the Unicode general category value is used as the basis for determining the set element. The characters corresponding to this category usually do not produce visible glyphs at the display level, but can change the text presentation or segmentation behavior, which is consistent with the actual source of the zero-width control character punch. When it is determined that the current character belongs to the format control character set, the value of the punch mask at the current sequence position index is written as one. Otherwise, the value of the punch mask at the current sequence position index is kept as zero or written as zero. Thus, after traversal, a punch mask that is aligned bit by bit with the original text character sequence is obtained. The punch mask is used to characterize whether there is a punch point formed by the format control character at each position and serves as the input for subsequent punch field calculation.

[0019] S2. Calculate the punch field based on the punch mask, and extract the visible sequence and the aligned punch field aligned with the visible sequence from the original text character sequence; In an embodiment of the present invention, the punch field is calculated based on the punch mask, and the visible sequence and the aligned punch field aligned with the visible sequence are extracted from the original text character sequence, including: The sliding window size is adaptively determined based on the length of the original text character sequence; For each position in the original text character sequence, calculate the cumulative value of the punch mask within the coverage area of ​​the sliding window; Specifically, after obtaining the original text character sequence and its corresponding punch mask, the length of the original text character sequence is first read and recorded as the sequence length. Then, the sliding window size is adaptively determined based on the sequence length so that the window coverage changes synchronously with the text size. Preferably, the sliding window radius is set to the square root of the sequence length rounded up, and the sliding window size is set to twice the window radius plus one. This is based on the fact that the square root growth law can provide sufficient neighborhood coverage for short texts to reflect the local punch clustering effect and avoid diluting local perturbations as the window length increases linearly for long texts. After the sliding window size is determined, each position of the original text character sequence is traversed using the sequence position index as the loop variable. When the current position is reached, the window center is aligned to the current position, and the left and right boundaries of the window coverage are determined. The left boundary is taken as the larger of the current position minus the window radius and the sequence start position, and the right boundary is taken as the smaller of the current position plus the window radius and the sequence end position, to ensure that the window coverage always falls within the effective index interval. Then, the punch mask values ​​within the coverage area are accumulated to obtain the punch cumulative value corresponding to the current position. The accumulation operation is obtained by adding the punch mask values ​​of each position within the coverage area one by one. The term with a punch mask of one indicates that there is a punch point formed by a format control character at that position and makes a unit contribution to the cumulative value. The term with a punch mask of zero indicates that there is a normal character at that position and does not contribute to the cumulative value. Thus, for each position of the original text character sequence, a cumulative value corresponding to the punch distribution intensity of its neighborhood is obtained and output as the input sequence for subsequent punch field normalization calculation.

[0020] The accumulated values ​​are normalized to obtain the perforation field; Extract the characters corresponding to the positions where the punch mask is 0 to form a visible sequence, and extract the values ​​of the punch field at the same positions to form an aligned punch field; Specifically, the punch field refers to the continuous numerical distribution formed by the punch mask at the sequence positions of the original text character sequence. It is used to characterize the intensity of the invisible punching effect caused by format control characters in the neighborhood of each position. The punch field is obtained by accumulating the punch mask values ​​within the coverage area of ​​each position through a sliding window adaptively determined according to the length of the original text character sequence and normalizing the accumulated results. This converts the discrete zero-values ​​of the punch mask into continuous perturbation weights that can be used for deep learning sequence modeling. The aligned punch field refers to the continuous numerical distribution obtained by mapping the punch field from the full position index space of the original text character sequence to the index space of the visible sequence. The visible sequence is extracted from the characters corresponding to the positions where the punch mask is zero in the original order. The aligned punch field is extracted synchronously from the values ​​of the punch field at the same position according to the same extraction rules, thereby ensuring that the aligned punch field is consistent with the visible sequence position by position.

[0021] Specifically, after obtaining the punch accumulation value sequence calculated for each position of the original text character sequence, in order to convert the punch accumulation value sequence into a continuous weight distribution that can be directly used as a conditional input for deep learning, the punch accumulation value sequence is normalized to obtain the punch field. The normalization process is to first sum all elements of the punch accumulation value sequence to obtain a normalized denominator, and then divide the punch accumulation value at each position by the normalized denominator to obtain the punch field value at that position. This results in the punch field forming a non-negative distribution with a sum of one across the entire sequence and maintaining the comparability of relative punch intensities.

[0022] Synchronous extraction is performed on the original text character sequence based on the punch mask to form a visible sequence and an aligned punch field. Specifically, the original text character sequence is traversed in its original order, and the punch mask is read at each position. When the punch mask value is zero, the character corresponding to the position is appended to the visible sequence, and the value of the punch field at that position is written to the aligned punch field in the same appending order. When the punch mask value is one, the position is skipped and not written to the visible sequence or the aligned punch field. After the traversal, a visible sequence consisting of all characters at positions where the punch mask is zero and an aligned punch field aligned with the visible sequence are obtained. The aligned punch field is used to characterize the intensity of the hidden punch effect in the neighborhood of the visible character position.

[0023] S3. Input the visible sequence and the aligned perforation field into the depth sequence network, and output the repair characterization and repair uncertainty; In embodiments of the present invention, the visible sequence and the aligned perforation field are input into a depth sequence network, and the repair characterization and repair uncertainty are output, including: The visible characters in the visible sequence are embedded into vectors to obtain character embeddings, and corresponding time position codes are generated. Map the aligned perforation field to a perforation feature vector; Specifically, after obtaining the visible sequence and its aligned punch field, each visible character in the visible sequence is first embedded into a vector to form a continuous representation that can be processed by a deep neural network. This embedding vectorization is achieved by establishing a character-to-vector mapping table and using the Unicode code point of the visible character as an index to read the corresponding embedding vector. The dimension of the embedding vector is preferably 256 to balance expressive power and inference overhead within the common character sets of live stream comments and product information streams. Simultaneously, to enable the sequence model to distinguish the semantic and rhythmic effects of the same character at different positions, a corresponding temporal position code is generated for the visible sequence. This temporal position code uses the sequence position index of the visible character in the visible sequence as the independent variable and is generated using a deterministic encoding in sine and cosine form. The 2j-th dimension and the 2j+1-th dimension at the t-th position are calculated as sine terms, respectively. The encoding dimension is consistent with the character embedding dimension, ensuring that it can be directly concatenated or added to the character embedding later. After completing the character embedding and temporal position encoding, the continuous scalar of the aligned piercing field at each position is mapped to a piercing feature vector to reflect the perturbation intensity of the hidden piercing in the neighborhood of that position. The mapping is preferably implemented using a two-layer feedforward neural network with the scalar of the aligned piercing field as input. The first layer is a linear transformation and is activated by Softplus to maintain a non-negative response. The second layer is a linear transformation to output a piercing feature vector with the same dimension as the character embedding. The weight parameters of the first and second layers are obtained through joint learning with the end-to-end training of the subsequent lip-sync task, so that the piercing feature vector can adaptively express the influence of piercing intensity on articulation unit modeling and mouth shape generation, and together with the character embedding and temporal position encoding, constitute the input of the subsequent repair representation generation network.

[0024] Character embeddings, temporal location encodings, and perforation feature vectors are concatenated and input into a deep sequence network; Specifically, after obtaining the character embeddings, temporal position codes, and perforation feature vectors corresponding to each position in the visible sequence, in order for the deep sequence network to simultaneously perceive the literal information of the visible characters, the sequence position information, and the intensity of the neighborhood perturbation introduced by the hidden perforations at the same position, the three types of vectors are first concatenated in the feature dimension direction to form a unified input vector. The concatenated vector corresponding to the t-th visible character is arranged in the order of character embedding first, temporal position code in the middle, and perforation feature vector last, and is denoted as the input vector. The deep sequence network preferably adopts a multi-layer self-attention encoder network to adapt to long sequence dependency modeling. Its specific structure includes an input projection layer, multiple attention blocks, and an output normalization layer. The input projection layer is used to linearly map the concatenated fused input vector to a unified model latent space dimension for subsequent attention calculation. The attention block includes at least a multi-head self-attention sublayer and a feedforward network sublayer, with residual connections and layer normalization used between the sublayers to stabilize gradient propagation. The multi-head self-attention sublayer generates a query matrix, a key matrix, and a value matrix from the input matrix and calculates the scaling dot product similarity between the query and the key to obtain attention weights. Then, the value matrix is ​​weighted and summed with the attention weights to obtain an information aggregation result containing the global context. The feedforward network sublayer uses two layers of linear transformation and nonlinear activation to achieve position-wise nonlinear representation enhancement. The encoder network preferably has four layers to cover the context dependencies under the common length of live barrage and is acceptable in terms of real-time inference overhead. The attention head is preferably eight heads to enhance the relationship modeling ability of different subspaces and improve the sensitivity to the location of punch perturbation. The final output context representation is used for subsequent generation of repair representations and repair of uncertainties.

[0025] The repair probability distribution at each time step is output through a deep sequence network; The repair probability distribution is weighted and summed to obtain the repair characterization. Calculate the entropy of the repair probability distribution and use the entropy as the repair uncertainty; Specifically, the repair representation refers to the continuous vector representation output by the deep sequence network for each visible character position after receiving the visible sequence and the aligned punch field. It is used to characterize the articulated content of the position under context conditions and its relationship with surrounding characters. The repair representation is obtained by the network performing probability calculations on the set of candidate repair symbols for the position and weighted summing of the embedding vectors of each candidate repair symbol according to its probability. This makes the repair representation simultaneously include the most likely repair result and the soft contribution to other feasible results, and facilitates the continuous calculation of subsequent articulation unit modeling and lip-sync generation. Repair uncertainty refers to the quantification result of the dispersion of the probability distribution of the above-mentioned candidate repair symbols for each visible character position. It is used to characterize the magnitude of repair ambiguity caused by punch perturbation at the position. The repair uncertainty is obtained by calculating the information entropy of the probability distribution of the candidate repair symbols. When the probability distribution is more concentrated, the repair uncertainty is smaller, indicating a clearer repair result. When the probability distribution is more dispersed, the repair uncertainty is larger, indicating stronger repair ambiguity at the position and potentially a higher risk of perturbation to subsequent duration unfolding and frame-level lip-sync generation.

[0026] Specifically, after obtaining the fused input matrix formed by concatenating character embeddings, temporal position encodings, and perforation feature vectors, the fused input matrix is ​​input into a deep sequence network to obtain the contextual hidden state sequence for each time step. Each time step corresponds to a character position in the visible sequence, and a set of hidden state vectors for predicting repair symbols is output. Subsequently, in the output layer, the hidden state vector is transformed into an unnormalized score vector of candidate repair symbols through a linear mapping. The candidate repair symbol set is preferably a set of symbols consistent with the character vocabulary to ensure that any visible character has a corresponding candidate space and to avoid introducing additional rule branches. After obtaining the unnormalized score vector, an exponential normalization operation is performed on it to obtain the repair probability distribution, so that the probability of each candidate repair symbol at that time step is non-negative and the sum is one, thereby outputting the repair probability distribution for each time step. After obtaining the repair probability distribution, the corresponding hidden state vector of each candidate repair symbol is read from the embedding table. The symbol embedding vectors are calculated, and a weighted summation is performed on the symbol embedding vectors with the repair probability distribution as weights to form the repair representation for that time step. The weighted summation is obtained by multiplying the symbol embedding vectors of each candidate repair symbol by its probability weight and summing over all candidates. This ensures that the repair representation not only reflects the most likely repair symbol but also includes the soft contribution to the second-best feasible symbol, thereby improving the stability of subsequent continuous modeling. At the same time, the information entropy of the repair probability distribution for that time step is calculated to quantify the repair ambiguity. The information entropy is obtained by taking the logarithm of each candidate probability in the repair probability distribution, multiplying it by itself, taking the negative, and then summing. When the probability is concentrated in a few candidate symbols, the information entropy is small, indicating a clear repair result. When the probability is dispersed among multiple candidate symbols, the information entropy is large, indicating strong ambiguity in the repair. The information entropy is used as the repair uncertainty output to characterize the repair risk caused by the punching disturbance at that time step and is used for subsequent duration modulation and frame-level lip generation.

[0027] S4. Generate pronunciation unit embeddings and unit base durations based on the repaired representation, and adjust the unit base durations according to the repair uncertainty to obtain the final unit durations; In embodiments of the present invention, the generation of pronunciation unit embeddings and unit base durations based on repaired characterization, and the adjustment of the unit base durations according to repair uncertainty, includes: Based on the repair representation, the pronunciation unit embedding and unit base duration are generated; Calculate the mean of the repair uncertainty for all time steps in the visible sequence; Specifically, after obtaining the repair representations and repair uncertainties corresponding to each time step of the visible sequence, all repair representations are stacked in chronological order to form a repair representation sequence, which is then input into the joint prediction network for articulation units and duration to generate the articulation unit embedding and unit base duration for each time step. The joint prediction network for articulation units and duration preferably adopts a deep sequence network structure similar to the repair network to maintain temporal dependency modeling capabilities and reduce implementation complexity. Specifically, it includes an input projection layer, multiple self-attention computation layers, and a feedforward enhancement layer. The input projection layer linearly maps the repair representations to the network's latent space dimension. The multiple self-attention computation layers perform global context aggregation on the repair representation sequence to capture the liaison effects of adjacent articulation units and smooth local anomalies caused by perforation. The feedforward enhancement layer performs nonlinear transformations on the latent states of each time step to enhance the expression of articulation-related patterns. Two parallel output branches are set at the network output end, where the first branch obtains the articulation units through linear mapping. The phonetic unit embedding vector carries the phonetic unit information corresponding to the time step and provides continuous conditional input for subsequent frame-level lip-sync generation. The dimension of the phonetic unit embedding vector is preferably set to 256 dimensions to maintain the same order of magnitude as the aforementioned character embedding dimension and facilitate subsequent network alignment. The second branch obtains a scalar prediction value of the unit base duration through linear mapping and performs a Softplus positive value operation on the scalar prediction value to ensure that the unit base duration is positive and facilitates subsequent duration expansion. The unit base duration represents the relative expansion length that the phonetic unit should be allocated to in the time step without considering the impact of punch repair ambiguity. At the same time, the repair uncertainty of all time steps in the visible sequence is averaged to obtain the global repair uncertainty mean. The mean calculation is obtained by adding the repair uncertainties of each time step one by one and dividing by the total number of time steps, thereby forming a global normalized benchmark for subsequent duration modulation, so that texts of different lengths remain comparable in terms of repair uncertainty.

[0028] For each time step, calculate the ratio of the repair uncertainty of the current time step to the mean of the repair uncertainty to obtain the modulation factor; The final unit duration is obtained by multiplying the unit base duration by the modulation factor. Specifically, the modulation factor refers to a continuous weight calculated for a specific time step in the visible sequence based on the relative magnitude of the repair uncertainty of that time step to the global repair uncertainty mean. It is used to quantify the strength of repair ambiguity caused by the zero-width control character punching at that time step and to transmit the strength of this ambiguity to the duration allocation process in an interpretable proportion. The modulation factor is obtained by adding one to the ratio of the repair uncertainty of the current time step to the global repair uncertainty mean. The modulation factor tends to be close to one when the repair uncertainty is low and greater than one when the repair uncertainty is high, thus reflecting a more conservative time expansion tendency for highly ambiguous positions. The final unit duration refers to the final duration value of the pronunciation unit of that time step for duration expansion after introducing the modulation factor on the basis of the unit base duration. The final unit duration is obtained by multiplying the unit base duration by the modulation factor. It represents the expansion length that the pronunciation unit of that time step should be allocated when generating frame-level lip shape parameters. The larger the value, the longer the time range occupied by that time step in the lip shape sequence, and it is used to reduce the risk of high-frequency lip shape switching caused by punching perturbation positions.

[0029] Specifically, after obtaining the unit base duration and the global repair uncertainty mean at each time step, duration modulation calculation is performed on each time step of the visible sequence to obtain the final unit duration. The time step index is used as a loop variable to traverse all time steps, and when traversing to the current time step, the repair uncertainty of that time step and the global repair uncertainty mean are read. Then, the ratio of the repair uncertainty of the current time step to the mean of the repair uncertainty is calculated, and this ratio is used to characterize the degree of repair ambiguity of the current time step relative to the global level, thereby obtaining the modulation factor. The modulation factor is preferably obtained by adding one to the ratio, so as to ensure that when the repair uncertainty of the current time step is comparable to the mean, the modulation factor is about two; when the repair uncertainty is low, the modulation factor approaches one; and when the repair uncertainty is high, the modulation factor is greater than two. This makes the duration unfolding more conservative at the position of high repair ambiguity and suppresses the propagation of punching perturbation to the frame-level lip sequence.

[0030] After obtaining the modulation factor, the unit base duration of the current time step is multiplied by the modulation factor to obtain the final unit duration. The multiplication operation makes the final unit duration change continuously with the degree of ambiguity repair and maintains differentiability for end-to-end training. At the same time, the final unit duration is always positive and can be directly used in the subsequent process of embedding the pronunciation units into the frame-level sequence by duration expansion.

[0031] S5. Based on the final unit duration, the pronunciation unit is embedded in the time dimension and expanded, and combined with the repair of uncertainty, a frame-level lip-sync parameter sequence is generated. In embodiments of the present invention, the pronunciation unit is embedded in the time dimension and expanded according to the final unit duration, and a frame-level lip-sync parameter sequence is generated by combining uncertainty repair, including: The number of frames corresponding to each pronunciation unit is determined based on the final unit duration; Generate the relative phase of each frame within its respective phonological unit and convert it into a phase embedding; Map the repair of uncertainty to uncertainty embedding; Specifically, after obtaining the final unit duration corresponding to each time step of the visible sequence and the embedding and repair uncertainty of the articulation units at the same time step, the final unit duration is used to determine the number of frames that should be allocated to each articulation unit during frame-level expansion. This involves traversing all articulation units using the time step index and reading the final unit duration of the current time step. To ensure the frame number is a positive integer and facilitate subsequent generation of lip-shape parameters in frame order, the final unit duration is rounded up to obtain the number of frames corresponding to the current articulation unit, and this is limited to at least one frame to avoid empty expansion, thus obtaining the frame number sequence corresponding to each articulation unit. After the frame number is determined, a relative phase is generated for each frame within the current articulation unit to characterize the normalized position of that frame within the expansion interval of that articulation unit. The relative phase is obtained by dividing the current frame's index within the articulation unit by the number of frames corresponding to that articulation unit, ensuring that the relative phase value is between zero and one and monotonically increases with the frame index, thereby forming a continuous sequence capable of expressing the time progress within a unit. The scalar is then converted into a phase embedding for use by the frame-level lip-sync generation network. The phase embedding preferably uses deterministic encoding in the form of sine and cosine, and the phase embedding dimension is set to be on the same order of magnitude as the articulation unit embedding dimension. This is because sine and cosine encoding can express the periodic phase in a continuously differentiable manner and facilitates the network's learning of the dynamic changes within the unit. At the same time, the repair uncertainty is mapped to uncertainty embedding to use the piercing repair ambiguity as a conditional input for frame-level generation. The uncertainty embedding is preferably composed of the ratio of the repair uncertainty to the mean of the global repair uncertainty and the square of the ratio, and is obtained by linear expansion along a predetermined dimension. This is because the ratio characterizes the relative ambiguity intensity, while the squared term enhances the discriminability of highly ambiguous positions and maintains continuous interpretability. Finally, the uncertainty embedding corresponding to each articulation unit is obtained and reused in all frames of that articulation unit, so that the frame-level lip-sync generation network can explicitly perceive the repair risk level of that unit when generating the lip-sync parameters for each frame of that unit.

[0032] The articulation unit embedding, phase embedding, and uncertainty embedding are concatenated and input into a deep generative network to obtain a frame-level lip shape parameter sequence. Specifically, the frame-level lip-sync parameter sequence refers to an ordered set of mouth motion control parameters output by a depth generation network in frame order after the visible sequence is expanded into continuous frames according to the final unit duration of the corresponding articulation units at each time step. Each frame of the sequence corresponds to the mouth shape state of the virtual character at that frame and is represented in the form of a continuous numerical vector. Each component of the continuous numerical vector corresponds to the independent degrees of freedom of the controllable deformation of the mouth and can be directly mapped to the mouth expression hybrid shape coefficient or mouth skeleton control amount, so that the virtual character's opening amplitude, lip closure degree, lip corner stretching and jaw opening and closing movements change continuously on the time axis. The frame-level lip-sync parameter sequence is arranged in ascending order of time and corresponds one-to-one with the frame timestamp. Its length is determined by the sum of the frame number of each articulation unit, so that the sequence can be used as a direct input to the rendering and driving module to achieve lip-sync animation output that matches the text content.

[0033] Specifically, after completing frame-level unrolling by the final unit duration, the phonological unit to which each frame belongs is determined, and the corresponding phonological unit embedding is read. Simultaneously, the phase embedding of the frame within its phonological unit is read, and the uncertainty embedding corresponding to that phonological unit is read as a piercing repair ambiguity condition. The phonological unit embedding characterizes the phonological driving information that the frame should present; the phase embedding characterizes the time progression within the unit to depict the continuous change of the lip shape from start to finish; and the uncertainty embedding characterizes the repair risk level of the unit to suppress high-frequency lip shape jumps caused by the piercing location. Subsequently, the phonological unit embedding, phase embedding, and uncertainty embedding are concatenated along the feature dimension to obtain the fusion condition vector for that frame. The fusion condition vectors of all frames are then stacked in chronological order to form a frame-level conditional input matrix. After the frame-level conditional input matrix is ​​constructed, it is input into a deep generation network to generate a frame-level lip shape parameter sequence. The deep generation network preferably employs an autoregressive sequence generation structure to explicitly model the smooth transition of lip shape parameters between adjacent frames and adapt to real-time generation requirements. The structure includes a conditional input projection layer, a multi-layer self-attention generation layer with causal masking, and an output head layer. The conditional input projection layer linearly maps the concatenated fused conditional vector to the latent space dimension of the generator network to unify the subsequent attention calculation scale. The generation layer consists of multi-layer causal self-attention sub-layers and feedforward nonlinear sub-layers, and residual connections and layer normalization are used between the sub-layers to stabilize training and inference. When calculating attention weights, the causal self-attention sub-layers apply invisible constraints to future time steps to ensure that the lip-sync parameters of each frame depend only on the conditions and generated states of the current and historical frames, thus forming a frame-by-frame recursive generation process. The feedforward nonlinear sub-layers perform position-by-position nonlinear transformations on the latent states of each frame to enhance the expression of lip-sync details. In the output head layer, the latent states of each frame output by the generation layer are linearly mapped to obtain the corresponding frame-level lip-sync parameter vector. The frame-level lip-sync parameter vector is preferably a mouth expression hybrid shape coefficient or a mouth keypoint driving coefficient and is represented by continuous values ​​to directly interface with the virtual character. Finally, a sequence of frame-level lip-sync parameters arranged in chronological order is obtained and output for subsequent character-driven mapping and rendering playback.

[0034] S6. Map the frame-level lip-sync parameter sequence to virtual character driving variables; In embodiments of the present invention, mapping frame-level lip-sync parameter sequences to virtual character driving variables includes: By using a linear mapping matrix and a bias vector, the frame-level lip-sync parameter sequence is converted into the skeletal control variables or facial expression blending shape coefficients of the virtual character. Specifically, after obtaining the frame-level lip-sync parameter sequence arranged in chronological order, to ensure that the frame-level lip-sync parameters can directly match the binding structure of the target virtual character and be stably executed by the rendering engine, the lip-sync drive channel definition is first read from the character assets to determine whether the target output type is skeletal control quantity or facial expression blending shape coefficient. Based on this, the dimensions of the linear mapping matrix and the bias vector are determined. The dimension of the frame-level lip-sync parameter vector is denoted as P, and the number of target drive channels is denoted as K. The linear mapping matrix is ​​then set to K rows and P columns, and the bias vector is set to K dimensions. Subsequently, each frame of the frame-level lip-sync parameter sequence is traversed using the frame index as the loop variable. The frame-level lip-sync parameter vector of the current frame is read, and a linear transformation is performed to obtain the target drive quantity vector of the current frame. The linear transformation is achieved by multiplying the linear mapping matrix by the frame-level lip-sync parameter vector and adding the bias vector, thereby obtaining the skeletal control quantity corresponding to the current frame. Alternatively, a mixed shape coefficient for facial expressions can be used, where the mixed shape coefficient directly controls the weights of the shape keys related to the character's mouth, while the skeletal control amount controls the rotation angle or displacement of the mouth bones. To ensure that the output driving amount is consistent with the controllable range of the character, amplitude constraint processing is further performed on each component of the target driving amount vector. The amplitude constraint is obtained by truncating each component according to the minimum and maximum allowable values ​​of the corresponding channel in the character assets, so as to avoid mesh interpolation or mouth shape distortion caused by out-of-range driving. After completing the linear mapping and amplitude constraint of each frame, the target driving amount vectors of each frame are arranged into a driving sequence in chronological order and output to the animation playback module or rendering module, so that the virtual character updates the mouth shape in each frame according to the skeletal control amount or the mixed shape coefficient for facial expressions, thereby unambiguously converting the frame-level mouth shape parameter sequence into an executable character driving output.

[0035] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A text-driven deep learning method for lip-syncing of virtual characters, characterized in that, Includes the following steps: S1. Obtain the original text character sequence and identify the format control characters in the original text character sequence to generate a punch mask; S2. Calculate the punch field based on the punch mask, and extract the visible sequence and the aligned punch field aligned with the visible sequence from the original text character sequence; S3. Input the visible sequence and the aligned perforation field into the depth sequence network, and output the repair characterization and repair uncertainty; S4. Generate pronunciation unit embeddings and unit base durations based on the repaired representation, and adjust the unit base durations according to the repair uncertainty to obtain the final unit durations; S5. Based on the final unit duration, the pronunciation unit is embedded in the time dimension and expanded, and combined with the repair of uncertainty, a frame-level lip-sync parameter sequence is generated. S6. Map the frame-level lip-sync parameter sequence to virtual character driving variables.

2. The text-driven virtual character lip-syncing deep learning method according to claim 1, characterized in that, Identifying format control characters in the original text character sequence to generate a punch mask includes: Iterate through each character of the original text character sequence; Determine whether the current character belongs to the preset format control character set. If it does, set the value of the punch mask at the corresponding position to 1; otherwise, set the value of the punch mask at the corresponding position to 0.

3. The text-driven virtual character lip-syncing deep learning method according to claim 2, characterized in that, The punch field is calculated based on the punch mask, and the visible sequence and the aligned punch field aligned with the visible sequence are extracted from the original text character sequence, including: The sliding window size is adaptively determined based on the length of the original text character sequence; For each position in the original text character sequence, calculate the cumulative value of the punch mask within the coverage area of ​​the sliding window; The accumulated values ​​are normalized to obtain the perforation field; Extract the characters corresponding to the positions where the punch mask is 0 to form a visible sequence, and extract the values ​​of the punch field at the same positions to form an aligned punch field.

4. The text-driven virtual character lip-syncing deep learning method according to claim 1, characterized in that, Input the visible sequence and aligned perforation field into the depth sequence network, including: The visible characters in the visible sequence are embedded into vectors to obtain character embeddings, and corresponding time position codes are generated. Map the aligned perforation field to a perforation feature vector; Character embedding, temporal location encoding, and perforation feature vector are concatenated and input into a deep sequence network.

5. The text-driven virtual character lip-syncing deep learning method according to claim 1, characterized in that, Output repair characterization and repair uncertainties, including: The repair probability distribution at each time step is output through a deep sequence network; The repair probability distribution is weighted and summed to obtain the repair characterization. Calculate the entropy of the repair probability distribution and use the entropy as the repair uncertainty.

6. The text-driven virtual character lip-syncing deep learning method according to claim 1, characterized in that, Adjusting the unit base duration based on the repair uncertainty includes: Calculate the mean of the repair uncertainty for all time steps in the visible sequence; For each time step, calculate the ratio of the repair uncertainty of the current time step to the mean of the repair uncertainty to obtain the modulation factor; The final unit duration is obtained by multiplying the unit base duration by the modulation factor.

7. The text-driven virtual character lip-syncing deep learning method according to claim 1, characterized in that, The frame-level lip-sync parameter sequence is generated by combining the repair of uncertainties, including: The number of frames corresponding to each pronunciation unit is determined based on the final unit duration; Generate the relative phase of each frame within its respective phonological unit and convert it into a phase embedding; Map the repair of uncertainty to uncertainty embedding; The articulation unit embedding, phase embedding, and uncertainty embedding are concatenated and input into a deep generative network to obtain a frame-level lip shape parameter sequence.

8. The text-driven virtual character lip-syncing deep learning method according to claim 1, characterized in that, Mapping frame-level lip-sync parameter sequences to virtual character driving variables includes: By using a linear mapping matrix and a bias vector, the frame-level lip-sync parameter sequence is converted into the skeletal control variables or facial expression blending shape coefficients of the virtual character.

Images