Digital human lip sync driving method and system

Abstract

The application belongs to the technical field of image processing, and discloses a digital human lip synchronization driving method and system. The method comprises the following steps: constructing a continuous audio playing time axis according to an audio stream sampling rate and actual output sampling frame numbers in an audio processing callback; generating a mouth shape advancing clock according to the continuous audio playing time axis; generating a mouth shape parameter sequence with a time label according to speech frame features and pronunciation context; writing the mouth shape parameter sequence with the time label into a mouth shape parameter ring buffer in chronological order and setting a target occupation depth; querying the mouth shape parameter corresponding to a current output time according to the mouth shape advancing clock, and performing a prediction advancing when the mouth shape parameter has been generated; determining callback jitter according to a deviation ratio, and performing a change amplitude limitation and adjusting the target occupation depth when the callback jitter occurs. The application improves the lip synchronization precision by constructing the continuous audio playing time axis and generating the mouth shape advancing clock.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a digital human lip-sound synchronization driving method and system. Background Technology

[0002] In real-time rendering scenarios of 2D digital humans across multiple platforms, to ensure that lip movements correspond to the speech playback process, existing technologies typically utilize a separate audio processing thread within the browser's audio processing architecture to process audio blocks. The timing of audio processing callbacks, the arrival of audio blocks at their corresponding rhythms, or the playback sequence related to callbacks are used as the basis for lip movement updates. This approach is relatively straightforward and can achieve basic lip movement control when the processing chain is stable and the device load is relatively steady.

[0003] However, in the actual operating environment of mobile or lightweight terminals, audio processing callbacks do not always arrive at the ideal fixed intervals. When browser load fluctuates, thread scheduling changes, or device resources are momentarily strained, audio processing callbacks are prone to merging, delaying, sudden increases in interval, or local jumps. In such cases, if lip movements are still directly driven by the callback trigger rhythm, the lip movement reference will change with the density of callbacks, failing to stably represent the actual audio output process. As a result, there may be pauses in lip movement during certain periods, followed by concentrated acceleration in subsequent periods, ultimately causing a timing deviation between the mouth movements and the actual sound production process.

[0004] Furthermore, existing technologies typically focus on displaying lip-shape parameters based on the query time. However, they lack a targeted, continuous connection mechanism for situations where the query time exceeds the existing range of lip-shape parameters or where callback fluctuations result in insufficient remaining lip-shape parameters for subsequent queries. Consequently, interruptions in lip-shape parameter queries can easily occur in the processing chain, leading to situations where the current rendering cycle can only directly use old values ​​or abruptly switch to new values. This results in jumps, jitters, or localized inconsistencies in the digitized human's mouth shape. Existing technologies also tend to passively follow callback fluctuations, lacking a linkage control mechanism to further apply the anomaly determination results to lip-shape output limitations and query margin adjustments, even if abnormal callback rhythms are detected.

[0005] Therefore, how to drive lip-sync progression based on the actual audio output process under unstable audio processing callback rhythm, and maintain continuous and stable lip-sync parameter output when the lip-sync parameter query margin is insufficient or the callback jitter occurs, has become a technical problem that urgently needs to be solved by those skilled in the art.

[0006] In view of this, the present invention proposes a digital human lip-sound synchronization driving method and system to solve the above problems. Summary of the Invention

[0007] To overcome the shortcomings of existing technologies, such as unstable lip-sync reference due to audio processing callback fluctuations, discontinuous lip-sync parameter lookup, and abrupt jumps and jitters in mouth shape, this invention provides a digital human lip-sync driving method and system. By constructing a continuous audio playback timeline, generating a lip-sync clock, managing lip-sync parameter buffers, and combining predictive advance and jitter suppression control, the timing accuracy, output continuity, and operational stability of the digital human lip-sync process are improved. To achieve the above objectives, this invention provides the following technical solution:

[0008] A digital human lip-sound synchronization driving method includes:

[0009] Obtain the audio stream sampling rate and the actual number of output sampling frames corresponding to each audio processing callback, and construct a continuous audio playback timeline based on the audio stream sampling rate and the actual number of output sampling frames.

[0010] A lip-sync advance clock is generated based on the continuous audio playback timeline;

[0011] The speech to be played is framed to extract speech frame features, and the text to be played is expanded to obtain the pronunciation context. Based on the speech frame features and pronunciation context, a time-labeled mouth shape parameter sequence is generated, where the mouth shape parameter is a parameter that characterizes the changes in the shape of the digital human mouth.

[0012] Write the time-tagged lip-sync parameter sequence into the lip-sync parameter circular buffer in chronological order and set the target occupancy depth;

[0013] The current output time is determined based on the lip-sync advance clock, and the lip-sync parameters corresponding to the current output time are retrieved from the lip-sync parameter circular buffer.

[0014] When the current output time exceeds the range of generated mouth shape parameters, perform predictive propulsion to determine the mouth shape parameters corresponding to the current output time.

[0015] Obtain the actual callback interval and the theoretical expected callback interval, and determine whether callback jitter has occurred based on the deviation ratio between the actual callback interval and the theoretical expected callback interval;

[0016] When a callback jitter is detected, the change range of the lip-sync parameter corresponding to the current output moment is limited, and the target occupancy depth is adjusted according to the degree of jitter.

[0017] Specifically, methods for constructing a continuous audio playback timeline include:

[0018] Before audio playback begins, the sampling rate of the current audio stream and the processing block frame length of the audio processing thread are read, and the cumulative output sample frame count register is set in the audio processing thread. The cumulative output sample frame count register is used to store the total number of sample frames that have been sent to the audio output link since the start of playback.

[0019] Each time an audio processing callback occurs, the number of sampled frames actually output this time is read and added to the cumulative output sampled frame count register.

[0020] After the accumulation is completed, the cumulative number of output sample frames is divided by the sampling rate to obtain the position of the continuous audio playback timeline corresponding to the current audio playback moment.

[0021] Specifically, the method for generating the lip-sync clock is as follows:

[0022] During the initial playback phase, the execution time of the current rendering frame is continuously recorded according to a fixed rendering beat. The fixed rendering beat is the time beat at which the graphics rendering side performs a lip-sync update once at a preset time interval. The position of the continuous audio playback timeline closest to the current rendering frame is recorded in the audio processing thread. The time difference between the execution time of the current rendering frame and the corresponding moment of the continuous audio playback timeline is recorded as a delay sample.

[0023] After the data collection is completed, the maximum and minimum values ​​in the delayed samples are removed, and the average value of the remaining delayed samples is calculated. The resulting average value is then used as the synchronization offset.

[0024] Subtract the synchronization offset from the continuous audio playback timeline to obtain the lip-sync advance clock.

[0025] Specifically, after generating the lip-sync advance clock, the lip-sync advance clock is kept consistent with the actual rendering state during continuous operation, and the synchronization offset is updated.

[0026] The specific implementation method for updating the synchronization offset is as follows:

[0027] Using a preset time period as a monitoring window, the average rendering latency is continuously calculated over N monitoring windows.

[0028] When the absolute value of the difference between the average rendering latency of N consecutive monitoring windows and the current synchronization offset is greater than the preset offset threshold, it is determined that the current synchronization offset can no longer represent the fixed delay compensation relationship between the audio output link and the graphics rendering link.

[0029] Collect all delay samples across N consecutive monitoring windows, and take the median of all delay samples as the updated synchronization offset.

[0030] Specifically, methods for generating time-labeled lip shape parameter sequences include:

[0031] The audio to be played is framed to obtain a time-ordered audio frame sequence and the basic acoustic features are extracted.

[0032] The text to be played is expanded into a sound, resulting in a sequence of basic phonetic units arranged in chronological order.

[0033] Time-align the speech frame sequence with the basic speech unit sequence;

[0034] Based on the time alignment results, the basic articulatory units are classified into categories, including closed-lip articulatory units, open-lip articulatory units, and rounded-lip articulatory units.

[0035] The basic mouth shape and initial values ​​of each dimension of mouth shape parameters are determined based on the type of articulatory unit.

[0036] The initial values ​​of each dimension of mouth shape parameters are corrected based on the basic acoustic characteristics to obtain the amplitude correction results of each dimension of mouth shape parameters. The amplitude correction results of each dimension of mouth shape parameters are then transitionally corrected based on the adjacent articulatory unit categories to obtain the transitional correction results of each dimension of mouth shape parameters. Finally, a mouth shape parameter sequence with time tags is generated by combining the corresponding time tags.

[0037] Specifically, methods for obtaining the amplitude correction results of each dimension of the mouth shape parameters include:

[0038] The short-time energy, fundamental frequency variation and spectral envelope features in the basic acoustic characteristics are normalized to obtain the energy correction factor, dynamic correction factor and spectral correction factor.

[0039] The height of the lips opening and closing and the amount of mandibular displacement are corrected based on the energy correction factor; the lateral traction of the corners of the mouth and the height of the lips opening and closing are corrected based on the dynamic correction factor; and the degree of lip rounding is corrected based on the spectral correction factor.

[0040] The correction values ​​of each mouth shape parameter are superimposed onto the initial values ​​of the corresponding mouth shape parameters to obtain the amplitude correction results of each mouth shape parameter.

[0041] Specifically, methods for obtaining the transition correction results for each dimension of mouth shape parameters include:

[0042] Using the switching boundary between the current articulatory unit category and the adjacent articulatory unit category as the center, the range of transitional speech frames before and after the switching boundary is determined, and the front weight corresponding to the previous articulatory unit category and the back weight corresponding to the current articulatory unit category are determined respectively.

[0043] Multiply the mouth shape parameter value corresponding to the previous articulatory unit category and the mouth shape parameter value corresponding to the current articulatory unit category by the front weight and the back weight respectively, and add the weighted results to obtain the transition correction result of each dimension of mouth shape parameter.

[0044] Specifically, the methods for writing into the circular buffer of the entry type parameter and setting the target occupancy depth include:

[0045] A mouth shape parameter record is formed by combining a time stamp and mouth shape parameters of each dimension. Each time a mouth shape parameter record is generated, the current write position of the mouth shape parameter circular buffer is written, and the write position is moved to the right by one record unit.

[0046] When the write position reaches the end of the mouth shape parameter circular buffer, it returns to the starting position to continue writing;

[0047] Count the actual interval of a preset number of consecutive audio processing callbacks and sort them from smallest to largest;

[0048] The actual interval corresponding to the preset high percentile value is used as the benchmark for callback fluctuation coverage.

[0049] The target occupancy depth is obtained by overlaying a preset safety margin on the basis of the pullback fluctuation coverage benchmark.

[0050] Specifically, the methods for performing predictive propagation to determine the lip shape parameters corresponding to the current output moment include:

[0051] When the current output time is between the two sets of lip shape parameter records, the weights of the previous and subsequent records are reassigned in reverse according to the distance from the current output time to the time tags of the two sets of records. The lip shape parameters of each dimension of the two sets are weighted and summed according to the weights of the previous and subsequent records to obtain the lip shape parameters corresponding to the current output time.

[0052] When the current output time exceeds the time stamp of the latest set of lip shape parameter records in the circular buffer of lip shape parameters, the time difference between the current output time and the time stamp of the latest set of lip shape parameter records is used as the lead amount;

[0053] When the lead amount reaches the preset lead threshold, short-time extrapolation is performed based on the weighted summation result of the change rate of the continuously preset number of lip shape parameter records to determine the lip shape parameters corresponding to the current output time.

[0054] Specifically, the methods for determining callback jitter include:

[0055] The theoretical expected callback interval is obtained by dividing the processing block frame length by the sampling rate.

[0056] During operation, the actual time interval is recorded for each pair of adjacent audio processing callbacks, and the absolute value of the difference between the actual time interval and the theoretical expected callback interval is divided by the theoretical expected callback interval to obtain the deviation ratio.

[0057] Compare the deviation ratio with the jitter ratio threshold;

[0058] When the deviation ratio is greater than the jitter ratio threshold, a callback jitter is determined to have occurred; when the deviation ratio is not greater than the jitter ratio threshold, a callback jitter is determined to have not occurred.

[0059] Specifically, the methods for implementing the variation range limit include:

[0060] Calculate the difference between the lip shape parameter values ​​of the current output time and the output values ​​of the previous rendering cycle;

[0061] Compare the absolute value of the difference with the maximum permissible variation corresponding to each dimension of the mouth shape parameter;

[0062] When the absolute value of the difference corresponding to a one-dimensional lip shape parameter is not greater than the maximum allowable change, the corresponding lip shape parameter value will be used as the output value of the current rendering cycle.

[0063] When the absolute value of the difference corresponding to a one-dimensional lip shape parameter is greater than the maximum allowable change, the difference is truncated to the maximum allowable change while maintaining its original sign, and added to the corresponding lip shape parameter value output in the previous rendering cycle to obtain the restricted output value of the corresponding lip shape parameter.

[0064] A digital human lip-sound synchronization driving system, implementing the aforementioned digital human lip-sound synchronization driving method, includes:

[0065] The time construction module is used to obtain the audio stream sampling rate and the actual number of output sampling frames corresponding to each audio processing callback, and to construct a continuous audio playback timeline based on the audio stream sampling rate and the actual number of output sampling frames.

[0066] Clock generation module: used to generate a lip-sync clock based on the continuous audio playback timeline;

[0067] The lip-shape generation module is used to perform frame processing on the speech to be played to extract speech frame features, to perform pronunciation expansion on the text to be played to obtain the pronunciation context, and to generate a lip-shape parameter sequence with time labels based on the speech frame features and pronunciation context.

[0068] Buffer writing module: Used to write the sequence of lip shape parameters with timestamps into the circular buffer of lip shape parameters in chronological order and set the target occupancy depth;

[0069] Parameter query module: used to determine the current output time based on the lip shape advance clock, and query the lip shape parameters corresponding to the current output time from the lip shape parameter circular buffer;

[0070] Predictive propulsion module: Used to perform predictive propulsion to determine the lip parameters corresponding to the current output time when the current output time exceeds the range of generated lip parameters;

[0071] Jitter Detection Module: Used to obtain the actual callback interval and the theoretical expected callback interval, and to determine whether callback jitter has occurred based on the deviation ratio between the actual callback interval and the theoretical expected callback interval;

[0072] Jitter control module: When a callback jitter is detected, it limits the change in the lip shape parameters corresponding to the current output moment and adjusts the target occupancy depth according to the degree of jitter.

[0073] The technical effects and advantages of the digital human lip-voice synchronization driving method and system proposed in this invention are as follows:

[0074] Compared to existing technologies, this invention does not directly use the audio processing callback rhythm as the basis for lip-sync advancement. Instead, it acquires the audio stream sampling rate and the actual output sampling frame number corresponding to each audio processing callback to construct a continuous audio playback timeline. Based on this, it determines the synchronization offset and generates a lip-sync advancement clock, decoupling the lip-sync advancement benchmark from the callback trigger rhythm and ensuring it corresponds to the actual audio output process. This improves the lip-sync advancement pauses, concentrated advancements, and timing misalignments caused by callback merging, delays, or jumps in existing technologies, enhancing the timing accuracy and advancement stability of the digital human lip-sync process.

[0075] This invention further extracts speech frame features by framing the speech to be played, expands the pronunciation of the text to be played to obtain the pronunciation context, and generates a time-stamped sequence of lip-shape parameters. The lip-shape parameter sequence is then written into a circular buffer of lip-shape parameters in chronological order, with a target occupancy depth set. This creates a continuous connection between lip-shape parameter generation, buffer writing, and subsequent queries. This allows for a stable query margin between lip-shape parameter generation and usage, improving the problems of read interruptions, reuse of old values, or abrupt switching to new values ​​caused by insufficient subsequent queryable lip-shape parameters in existing technologies, thus enhancing the continuity of lip-shape output.

[0076] This invention further determines the current output time based on the lip-shape advance clock and queries the lip-shape parameter corresponding to the current output time from the lip-shape parameter circular buffer. When the current output time exceeds the range of generated lip-shape parameters, predictive advancement is performed to determine the lip-shape parameter corresponding to the current output time, ensuring that the query path and the connection path in the out-of-range case both fall on the result object of the lip-shape parameter corresponding to the current output time. This maintains continuous changes in mouth shape in both normal query and out-of-range compensation cases, improving the problems of local discontinuity and abrupt changes in mouth shape caused by query interruption or direct switching in existing technologies.

[0077] This invention further obtains the actual callback interval and the theoretically expected callback interval, and determines whether callback jitter has occurred based on the deviation ratio between the two. When callback jitter is determined to have occurred, the change amplitude of the lip-shape parameters corresponding to the current output moment is limited, and the target occupancy depth is adjusted according to the degree of jitter. This further applies the abnormal callback determination result to the lip-shape output limitation and query margin adjustment. This can suppress sudden jumps in lip shape when abnormal fluctuations occur, and simultaneously increase the subsequent query margin, improving the problem in existing technologies where callback anomalies are directly amplified to the final lip-shape output, and improving the smoothness, continuity, and overall operational stability of the digital human's lip movements. Attached Figure Description

[0078] Figure 1 This is a flowchart of a digital human lip-sound synchronization driving method according to Embodiment 1 of the present invention;

[0079] Figure 2 This is a schematic diagram of the method for obtaining the lip shape parameters corresponding to the current moment in Embodiment 1 of the present invention;

[0080] Figure 3 This is a block diagram of a digital human lip-sound synchronization driving system according to Embodiment 2 of the present invention. Detailed Implementation

[0081] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0082] Example 1

[0083] See Figure 1 As shown, this embodiment provides a digital human lip-sound synchronization driving method, including:

[0084] A continuous audio playback timeline is constructed based on the audio stream sampling rate and the actual number of output sampling frames in the audio processing callback. The continuous audio playback timeline is a playback time reference that continuously increases with the actual number of output sampling frames, so that the lip-sync advancement reference is consistent with the actual audio output process.

[0085] The method for constructing a continuous audio playback timeline includes:

[0086] Before audio playback begins, the sampling rate of the current audio stream and the processing block frame length of the audio processing thread are read, and the cumulative output sample frame count register is set in the audio processing thread. The cumulative output sample frame count register is used to store the total number of sample frames that have been sent to the audio output link since the start of playback.

[0087] Each time an audio processing callback occurs, the actual number of sampled frames output is read and added to the cumulative output sampled frame count register. After accumulation, the cumulative output sampled frame count is divided by the sampling rate to obtain the continuous audio playback timeline position corresponding to the current audio playback moment.

[0088] Since the position of the continuous audio playback timeline is determined by the actual number of sampled frames that have been output, when the browser or terminal merges two small callbacks into one large callback when the instantaneous load increases, or skips a callback and makes up the output in the subsequent callback, the cumulative number of output sampled frames will continue to increase with the actual number of sampled frames sent, and the time base will not stop due to changes in the density of callbacks.

[0089] A synchronization offset is determined based on the continuous audio playback timeline, and a lip-sync advance clock is generated. The synchronization offset is a fixed delay compensation amount between the audio output link and the graphics rendering link. The lip-sync advance clock is a control clock used to determine the current output moment. The current output moment refers to the target time point mapped onto the continuous audio playback timeline based on the lip-sync advance clock during the current graphics rendering update process, used to determine the lip-sync parameters that should be read at the current time. During operation, when the rendering delay continuously deviates from the synchronization offset, the synchronization offset is updated.

[0090] The specific implementation method for determining the synchronization offset is as follows:

[0091] First, during the initial playback phase, the execution time of the current rendering frame is continuously recorded according to a fixed rendering beat. The fixed rendering beat is the time beat at which the graphics rendering side performs a lip-sync update once at a preset time interval. The position of the continuous audio playback timeline closest to the current rendering frame is recorded in the audio processing thread. The time difference between the execution time of the current rendering frame and the corresponding moment on the continuous audio playback timeline is recorded as a delay sample.

[0092] The number of delayed samples should be set to no less than 20, preferably 30, to reduce the impact of thread preemption, device wake-up, and first frame texture upload on the initial measurement results. After acquisition, the maximum and minimum values ​​in the delayed samples are removed, and the average value of the remaining delayed samples is calculated. The resulting average value is determined as the synchronization offset.

[0093] Subtract the synchronization offset from the continuous audio playback timeline to obtain the lip-sync advance clock.

[0094] After generating the lip-sync advance clock, the synchronization offset needs to be updated to ensure that the lip-sync advance clock remains consistent with the actual rendering state during continuous operation.

[0095] The specific implementation method for updating the synchronization offset is as follows: A preset time period, such as 500 milliseconds, is used as a monitoring window. Within each monitoring window, all latency samples are collected, and the arithmetic mean of all latency samples is taken as the average rendering latency of that monitoring window. The average rendering latency is then calculated continuously over N monitoring windows (N ranges from 3 to 5, preferably 3). When the absolute value of the difference between the average rendering latency of each of the N consecutive monitoring windows and the current synchronization offset is greater than a preset offset threshold, it is determined that the current synchronization offset can no longer accurately represent the fixed latency compensation relationship between the audio output link and the graphics rendering link. All latency samples from the N consecutive monitoring windows are collected, and the median of all latency samples is taken as the updated synchronization offset.

[0096] The offset threshold is set by those skilled in the art based on the relationship between the actual audio output link and the graphics rendering link.

[0097] The system performs frame processing on the speech to be played to extract speech frame features, expands the text to be played to obtain a sequence of basic phonetic units, and uses the adjacent basic phonetic units as the pronunciation context of the current basic phonetic unit. Based on the speech frame features and pronunciation context, a sequence of mouth shape parameters with time labels is generated to form a mouth shape parameter data stream corresponding to the current speech pronunciation state.

[0098] After extracting speech frame features and obtaining the pronunciation context, a time-labeled sequence of lip shape parameters is generated. These lip shape parameters characterize changes in the morphology of the digital human's mouth, including lip closure degree, lip opening and closing height, lip rounding degree, lateral traction of the corners of the mouth, and mandibular displacement. Specifically, lip closure degree characterizes the degree of contact between the upper and lower lips; a larger value indicates closer closure. Lip opening and closing height characterizes the vertical opening of the mouth fissure; a larger value indicates a larger vertical opening. Lip rounding degree characterizes the degree to which the lips retract forward; a larger value indicates a more pronounced rounded lip shape. Lateral traction of the corners of the mouth characterizes the lateral stretching of the left and right corners of the mouth relative to the midline. Mandibular displacement characterizes the vertical offset of the mandibular control point relative to the neutral position; a larger value indicates a greater downward displacement of the mandible. Preferably, lip closure degree, lip opening and closing height, lip rounding degree, and mandibular displacement are represented by normalized values ​​ranging from 0 to 1, while lateral traction of the corners of the mouth is represented by a two-vector centered on the neutral position, ranging from -1 to 1. To ensure that the combination of mouth shape parameters conforms to the actual mouth movement patterns, combination constraints are also set for the mouth shape parameters. Preferably, when the lip closure degree is greater than 0.8, the lip opening and closing height is not greater than 0.2; when the lip rounding degree is greater than 0.7, the lateral traction of the corners of the mouth is not greater than 0.1; when the lip opening and closing height is greater than 0.75, the mandibular displacement is not less than 0.5.

[0099] The method for generating a time-stamped sequence of lip shape parameters includes:

[0100] The audio to be played is framed to obtain a sequence of audio frames arranged in chronological order, and basic acoustic features are extracted, including the short-time energy, fundamental frequency variation, and spectral envelope features of each audio frame. The audio frame division and basic acoustic feature extraction process can be implemented using existing audio analysis techniques, which will not be elaborated on here.

[0101] The text to be played is expanded to produce a sequence of basic pronunciation units arranged in chronological order. The framing and expansion processes can be implemented using existing text front-end processing technologies, which will not be elaborated on here.

[0102] After obtaining the basic articulation unit sequence, the speech frame sequence is time-aligned with the basic articulation unit sequence, so that each speech frame corresponds to the current basic articulation unit and the adjacent contextual basic articulation units. Specifically, the center time of the speech frame is used as the alignment reference to determine the basic articulation unit interval falling within that time, and the corresponding basic articulation unit is identified as the current basic articulation unit, while the basic articulation units immediately before and after it are identified as the contextual basic articulation units. Through this process, each speech frame simultaneously possesses acoustic features, current basic articulation unit information, and contextual basic articulation unit information.

[0103] After time alignment, the current basic articulation units are categorized according to mouth shape driving requirements to obtain corresponding articulation unit categories. These articulation unit categories are mouth shape driving categories used to characterize different mouth articulation forms, including closed-lip articulation units, open-lip articulation units, and rounded-lip articulation units. Specifically, based on the mouth closure, open-lip, and rounded-lip states corresponding to the current basic articulation unit, the current basic articulation units are categorized; when the current basic articulation unit corresponds to a closed-lip articulation form, it is classified as a closed-lip articulation unit; when the current basic articulation unit corresponds to an open-lip articulation form, it is classified as an open-lip articulation unit; and when the current basic articulation unit corresponds to a rounded-lip articulation form, it is classified as a rounded-lip articulation unit.

[0104] After obtaining the articulatory unit category, the basic mouth shape is determined based on the current articulatory unit category. When the current articulatory unit category belongs to the closed-lip articulatory unit category, the basic mouth shape is determined to be the closed-lip shape; when the current articulatory unit category belongs to the open-lip articulatory unit category, the basic mouth shape is determined to be the open-lip shape; when the current articulatory unit category belongs to the rounded-lip articulatory unit category, the basic mouth shape is determined to be the rounded-lip shape.

[0105] After determining the basic mouth shape morphology, the initial values ​​of each dimension of mouth shape parameters under the basic mouth shape morphology are further determined. The initial values ​​of mouth shape parameters refer to the reference parameter values ​​directly given by the basic mouth shape morphology corresponding to the current vocal unit category before the basic acoustic feature amplitude correction and the transition correction of adjacent vocal unit categories are performed. The method for obtaining the initial values ​​of each dimension of mouth shape parameters is as follows: a correspondence table between vocal unit categories and basic mouth shape parameters is established in advance. The correspondence table records the initial values ​​of bilip closure degree, bilip opening and closing height, lip rounding degree, lateral traction of the corner of the mouth, and mandibular displacement corresponding to different vocal unit categories. After determining the current vocal unit category, the initial values ​​of each dimension of mouth shape parameters corresponding to the current vocal unit category are found in the correspondence table.

[0106] Preferably, the correspondence table is obtained by statistically analyzing typical mouth shapes in standard pronunciation samples. The standard pronunciation samples refer to reference pronunciation samples that have been pre-collected and labeled with pronunciation unit categories. These standard pronunciation samples include mouth shape data aligned with the speech signal time and corresponding pronunciation unit category identifiers, used to characterize the typical mouth shape of the corresponding pronunciation unit category under stable pronunciation conditions. Specifically, standard pronunciation samples corresponding to different pronunciation unit categories are collected, and the lip closure degree, lip opening and closing height, lip rounding degree, lateral traction of the corners of the mouth, and mandibular displacement of each standard pronunciation sample under stable pronunciation conditions are extracted. Representative values ​​of each dimension of mouth shape parameters are statistically analyzed according to the pronunciation unit category, and these representative values ​​are then written into the correspondence table as initial values ​​of the basic mouth shape parameters for the corresponding pronunciation unit category. Further preferably, the initial value of lip closure degree corresponding to closed-lip pronunciation units is higher than that corresponding to open-lip pronunciation units; the initial value of lip opening and closing height corresponding to open-lip pronunciation units is higher than that corresponding to closed-lip pronunciation units; and the initial value of lip rounding degree corresponding to rounded-lip pronunciation units is higher than that corresponding to non-rounded-lip pronunciation units.

[0107] The same set of basic mouth shape parameters can be used to determine the basic mouth shape direction within the same articulation unit category. Differences in mouth opening amplitude, corner traction, lip rounding, and mandibular displacement during the specific articulation process are reflected through basic acoustic feature amplitude correction and transition correction between adjacent articulation unit categories.

[0108] After determining the basic lip shape and initial values ​​of each lip shape parameter, the basic acoustic features corresponding to the current speech frame are read, and the initial values ​​of each lip shape parameter are corrected in amplitude based on these basic acoustic features. To eliminate the influence of differences in the dimensions and value ranges of different acoustic features on the amplitude correction results of the lip shape parameters, the short-time energy, fundamental frequency change, and spectral envelope features are first normalized to obtain the corresponding energy correction factor, dynamic correction factor, and spectrum correction factor. Specifically, the upper and lower reference limits for the short-time energy, fundamental frequency change, and spectral envelope features are determined respectively. Then, the acoustic feature value corresponding to the current speech frame is subtracted from the lower reference limit and divided by the difference between the upper and lower reference limits to obtain the normalization result. When the normalization result is greater than the preset upper limit, it is truncated to the preset upper limit; when the normalization result is less than the preset lower limit, ... The data is truncated to a preset lower limit; preferably, the preset upper limit is set to 1 and the preset lower limit is set to 0; more preferably, the reference upper limit and reference lower limit are determined primarily by statistically analyzing the corresponding acoustic feature value range in the current speech segment, where the current speech segment refers to a local continuous speech window formed by taking 5 consecutive speech frames forward and backward from the current speech frame as the center; when the length of the current speech segment is insufficient to form a stable statistical result, it is determined by the corresponding acoustic feature value range in the current speaker sample; when the current speaker sample is unavailable, it is determined by the corresponding acoustic feature value range in the preset reference speech set.

[0109] After normalization, using the initial values ​​of each mouth shape parameter as a baseline, short-term energy correction is applied to the bilip opening height and mandibular displacement. Specifically, the correction amounts for bilip opening height and mandibular displacement are calculated based on the energy correction factor, and the bilip opening height correction is added to the initial value of bilip opening height, and the mandibular displacement correction is added to the initial value of mandibular displacement, resulting in the bilip opening height and mandibular displacement after short-term energy correction. The bilip opening height correction is determined by the energy correction factor and the bilip opening height energy correction weight, and the mandibular displacement correction is also determined by the energy correction factor and the mandibular displacement energy correction weight. Through this process, the mouth opening amplitude and mandibular displacement amplitude increase synchronously relative to the basic mouth shape when the vocal intensity is high, and decrease synchronously relative to the basic mouth shape when the vocal intensity is low. Preferably, the energy correction weight for the bilip opening and closing height is set to 0.10 to 0.30, and the energy correction weight for the mandibular displacement is set to 0.08 to 0.25.

[0110] After short-term energy correction, fundamental frequency variation correction is applied to the lateral traction of the corners of the mouth and the height of lip opening / closing. Specifically, the correction amounts for the lateral traction of the corners of the mouth and the dynamic correction amounts for the height of lip opening / closing are calculated based on a dynamic correction factor. The lateral traction correction amount is then added to the current value of the lateral traction of the corners of the mouth, and the dynamic correction amount for the height of lip opening / closing is added to the current value of the height of lip opening / closing, resulting in the lateral traction of the corners of the mouth and the height of lip opening / closing corrected by the fundamental frequency variation. The lateral traction correction amount is determined by both the dynamic correction factor and the dynamic correction weight for the lateral traction of the corners of the mouth, and the dynamic correction amount for the height of lip opening / closing is also determined by both the dynamic correction factor and the dynamic correction weight for the height of lip opening / closing. Through this process, the dynamic traction of the corners of the mouth and the opening dynamics are enhanced relative to the basic mouth shape when the tone of voice changes significantly, while the mouth movements remain relatively stable when the tone of voice changes gently. Preferably, the dynamic correction weight for the lateral traction of the corners of the mouth is set to 0.05 to 0.20, and the dynamic correction weight for the height of lip opening / closing is set to 0.03 to 0.15.

[0111] After correcting for the fundamental frequency variation, spectral envelope feature correction is performed on the lip rounding degree. Specifically, the proportion of low-frequency components in the spectral envelope feature is extracted, and this proportion is normalized to a spectral correction factor. The lip rounding degree correction amount is then calculated based on this spectral correction factor and superimposed on the initial value of the lip rounding degree to obtain the lip rounding degree after spectral envelope feature correction. The lip rounding degree correction amount is jointly determined by the spectral correction factor and the lip rounding degree spectral correction weight. Through this process, lip rounding-related articulation exhibits a more pronounced frontal convergence feature when the proportion of low-frequency components is high. Preferably, the lip rounding degree spectral correction weight is set to 0.08 to 0.25.

[0112] In the above-mentioned correction processes, the correction weights are determined based on the statistical results of the changes in each dimension of mouth shape parameters with acoustic features in the accompanying speech and mouth shape annotation samples, and are set separately according to the type of articulatory unit, so that closed-lip articulatory units, open-lip articulatory units, and rounded-lip articulatory units have different correction sensitivities under the same acoustic change conditions. To avoid the corrected mouth shape parameters exceeding the allowable range of mouth shape parameters, after completing the basic acoustic feature correction, boundary truncation processing is also performed on each dimension of mouth shape parameters; when the correction result of any dimension of mouth shape parameter is greater than the upper limit of that dimension of mouth shape parameter, the correction result of that dimension of mouth shape parameter is truncated to the upper limit of that dimension of mouth shape parameter; when the correction result of any dimension of mouth shape parameter is less than the lower limit of that dimension of mouth shape parameter, the correction result of that dimension of mouth shape parameter is truncated to the lower limit of that dimension of mouth shape parameter.

[0113] After completing short-time energy correction, fundamental frequency change correction, spectral envelope feature correction, and boundary truncation processing, the amplitude correction results of each dimension of lip shape parameters corresponding to the current speech frame are obtained.

[0114] After completing the basic acoustic feature amplitude correction and obtaining the amplitude correction results of each dimension of mouth shape parameters corresponding to the current speech frame, the amplitude correction results of each dimension of mouth shape parameters are transitionally corrected according to the adjacent articulatory unit categories. Specifically, it is first determined whether there is a mouth shape switching between the current articulatory unit category and the adjacent articulatory unit categories. When there is a switching between closed and open lip state, the degree of lip closure and the height of lip opening and closing are boundary transitionally corrected. When there is a switching between non-rounded and rounded lip state, the degree of lip rounding is boundary transitionally corrected. When there is a difference in the mouth opening direction or closing direction between adjacent articulatory unit categories, the lateral traction of the corner of the mouth and the displacement of the mandible are coordinated and corrected.

[0115] During transition correction, the transition speech frame range before and after the switching boundary between the current articulatory unit (AMU) category and the adjacent AMU category is determined, using this boundary as the center. The preceding weights for the previous AMU category and the following weights for the current AMU category are then determined. These weights are distributed in reverse order based on the relative distances from the current speech frame to the reference positions on both sides, and are normalized so that the sum of the preceding and following weights is 1. Subsequently, the lip shape parameter values ​​corresponding to the previous and current AMU categories are multiplied by the preceding and following weights, respectively, and the weighted results are summed to obtain the transition correction results for each dimension of the lip shape parameters corresponding to the current speech frame. More preferably, the transition speech frame range is defined as two frames before and after the switching boundary.

[0116] After completing the basic acoustic feature amplitude correction and the adjacent vocal unit category transition correction, the transition correction results of each dimension of mouth shape parameters corresponding to the current speech frame are used as the final mouth shape parameters of the current speech frame, and the center time of the corresponding speech frame is used as the time label of the final mouth shape parameters, thereby forming a time-labeled mouth shape parameter sequence.

[0117] Write the sequence of lip-shape parameters with time tags into the circular buffer of lip-shape parameters in chronological order, and set the target occupancy depth to ensure sufficient query margin when querying lip-shape parameters by advancing the clock according to lip shape.

[0118] After obtaining the sequence of lip-sync parameters with time tags, the lip-sync parameters corresponding to each frame are written into the circular buffer of lip-sync parameters in ascending order of time tags. The circular buffer of lip-sync parameters is a circular storage area that caches lip-sync parameters and their time tags in chronological order. During writing, a combination of time tags and lip-sync parameters of each dimension is used as a lip-sync parameter record. Each time a lip-sync parameter record is generated, it is written into the current writing position of the circular buffer of lip-sync parameters, and the writing position is moved forward by one record unit. When the writing position reaches the end of the circular buffer of lip-sync parameters, it wraps back to the starting position and continues writing.

[0119] After writing the lip-sync parameter records, a target occupancy depth is set. This target occupancy depth refers to the length of time corresponding to unread lip-sync parameter records pre-reserved in the lip-sync parameter circular buffer to ensure the continuity of subsequent queries in the current lip-sync advance clock. The target occupancy depth is determined based on the statistical results of the actual intervals of the most recent consecutive audio processing callbacks. Specifically, the actual intervals of a preset number of consecutive audio processing callbacks are statistically analyzed, sorted from smallest to largest, and a preset high-quantile interval value is taken as the callback fluctuation coverage benchmark. A preset safety margin is then added to obtain the target occupancy depth. This method ensures that the target occupancy depth covers the normal fluctuation range of audio processing callbacks, thereby improving the continuity of subsequent lip-sync parameter queries and the anti-jitter capability.

[0120] Preferably, the actual interval between consecutive preset number of audio processing callbacks is 32 consecutive times; the preset high percentile interval value is the actual interval value corresponding to the 90th percentile value to the 95th percentile value; and the preset safety margin is 8 milliseconds to 16 milliseconds.

[0121] After the target occupancy depth is set, the length of the unread area in the lip-shape parameter circular buffer is adjusted according to the target occupancy depth, and the overwriting conditions of the lip-shape parameter records are controlled. The unread area length refers to the number of lip-shape parameter records in the lip-shape parameter circular buffer whose timestamps are not less than the current lip-shape advance clock and have not yet been read by the current lip-shape advance clock. Specifically, when the number of lip-shape parameter records in the unread area meets the target occupancy depth requirement, lip-shape parameter records are written in the current writing order. When the number of lip-shape parameter records in the unread area does not meet the target occupancy depth requirement, the earliest unread record is not immediately overwritten, and newly generated lip-shape parameter records continue to be written until the number of lip-shape parameter records in the unread area recovers to meet the target occupancy depth requirement. When the lip-shape parameter circular buffer is full, and the timestamp corresponding to the earliest record is earlier than the current lip-shape advance clock reaching a preset overwrite time threshold, overwriting of the earliest record is allowed. Through this method, the lip-shape parameter circular buffer prioritizes replenishing unread records when subsequent query space is insufficient, and releases storage space only when the earliest record has expired and the overwrite condition is met.

[0122] Preferably, the capacity of the lip-shape parameter circular buffer corresponds to a lip-shape parameter record length of 80 milliseconds to 120 milliseconds; when the time interval between adjacent lip-shape parameter records is 10 milliseconds, the lip-shape parameter circular buffer is used to store 8 to 12 consecutive lip-shape parameter records. More preferably, the preset coverage time threshold is set to 40 milliseconds.

[0123] Through the above processing, continuous writing of lip shape parameters, storage in chronological order, orderly overwriting of expired records, and dynamic maintenance of query margins are achieved, providing a stable data foundation for determining the current output time and reading the corresponding lip shape parameters based on the lip shape advance clock.

[0124] The current output time is determined based on the lip-shape advance clock, and the lip-shape parameters corresponding to the current output time are queried from the lip-shape parameter circular buffer. When the current output time is between two sets of lip-shape parameter records, the lip-shape parameters corresponding to the current output time are determined by time interpolation. When the current output time exceeds the time tag of the latest set of lip-shape parameter records in the lip-shape parameter circular buffer, a preset advance threshold is used to determine whether to perform predictive advance, so as to determine the lip-shape parameters corresponding to the current output time and maintain the continuous change of mouth shape.

[0125] See Figure 2 As shown, after obtaining the lip-sync advance clock and the lip-sync parameter circular buffer, when each fixed rendering beat arrives, the current output time is first determined based on the lip-sync advance clock, and then the lip-sync parameters corresponding to the current output time are queried.

[0126] Specifically, the current output time is first determined based on the current lip shape advance clock, and then the previous set of lip shape parameter records and the next set of lip shape parameter records with time tags located before and after the current output time are retrieved in the lip shape parameter circular buffer.

[0127] When two sets of lip-shape parameter records are retrieved, the distance from the current output time to each of the two time tags is calculated. The weights of the preceding and following records are then reassigned in reverse order of distance, ensuring that the closer the current output time is to a particular time tag, the greater its corresponding weight. The sum of the weights of the preceding and following records is equal to 1. Subsequently, the lip-shape parameters corresponding to each dimension of the two sets of lip-shape parameter records are weighted and summed according to the weights of the preceding and following records to obtain the lip-shape parameters corresponding to the current output time.

[0128] When the current output time exceeds the time stamp of the latest set of lip shape parameter records in the circular buffer of lip shape parameters, the time difference between the current output time and the time stamp of the latest set of lip shape parameter records is used as the lead amount.

[0129] The lead amount is compared with the preset lead threshold. When the lead amount does not reach the lead threshold, the latest set of lip shape parameters is output and the corresponding lip shape parameters are recorded for a fixed rendering cycle.

[0130] The lead threshold is determined based on the time interval corresponding to the fixed rendering beat, and is a preset proportion of the time interval corresponding to the fixed rendering beat. Preferably, the lead threshold is set to 0.5 to 1 times the time interval corresponding to the fixed rendering beat.

[0131] When the lead time reaches the lead threshold, a preset number of consecutive sets of mouth shape parameter records are read. The differences between adjacent sets of mouth shape parameter records in terms of lip closure, lip opening and closing height, lip rounding, lateral traction of the corners of the mouth, and mandibular displacement are calculated. The differences are then divided by the corresponding time interval to obtain the rate of change of each dimension of mouth shape parameter in each adjacent interval. The rate of change of the same dimension of mouth shape parameter in each adjacent interval is then weighted and summarized to obtain the current trend value of the dimension of mouth shape parameter. The current trend value is an estimate of the short-term change direction and magnitude of the dimension of mouth shape parameter near the current mouth shape advance clock. Finally, the dimension of mouth shape parameter corresponding to the latest set of mouth shape parameter records is used as the extrapolation starting point. The changes of the current trend value of each dimension of mouth shape parameter within a fixed rendering beat time length are superimposed on the extrapolation starting point to obtain the predicted mouth shape parameter at the current moment.

[0132] The number of consecutive preset quantities is set by those skilled in the art based on the fixed number of rendering beats and the stability requirements of the lip-shape parameter change trend. It is used to cover multiple consecutive lip-shape parameter change intervals before the current moment, and the preset value is 3.

[0133] The weights in the weighted aggregation are determined by the reciprocal of the distance from the end time stamp of each adjacent interval to the current lip-shape advance clock. Specifically, the time distance between the end time stamp of each adjacent interval and the current lip-shape advance clock is first calculated. Then, the reciprocal of each time distance is taken as the original weight of the change rate of the corresponding adjacent interval. Subsequently, the original weights are normalized so that the sum of the weights of each adjacent interval is 1, thus obtaining the final weight corresponding to the change rate of each adjacent interval. In this way, the adjacent interval closer to the current lip-shape advance clock has a higher proportion of its change rate in the current change trend value; the adjacent interval farther from the current lip-shape advance clock has a lower proportion of its change rate in the current change trend value.

[0134] After determining the mouth shape parameters corresponding to the current output moment through predictive propagation, to ensure that these mouth shape parameters remain within the allowable range for that dimension, boundary truncation is applied to the lip closure degree, lip opening and closing height, lip rounding degree, lateral traction of the corners of the mouth, and mandibular displacement. Specifically, when the lip closure degree, lip opening and closing height, lip rounding degree, and mandibular displacement are greater than 1, they are truncated to 1; when they are less than 0, they are truncated to 0. When the lateral traction of the corners of the mouth is greater than 1, it is truncated to 1; when it is less than -1, it is truncated to -1.

[0135] The callback jitter is determined based on the deviation ratio between the actual callback interval and the theoretically expected callback interval. When callback jitter occurs, the change amplitude of the lip-shape parameter corresponding to the current output time obtained by the aforementioned query or determined by prediction is limited, and the target occupancy depth is adjusted to suppress sudden jumps in lip-shape parameters and improve the query stability of the lip-shape parameter circular buffer.

[0136] The method for determining callback jitter based on the deviation ratio between the actual callback interval and the theoretically expected callback interval includes:

[0137] First, the theoretical expected callback interval is calculated based on the processing block frame length and sampling rate. The actual callback interval is the time interval between two adjacent audio processing callback trigger times, obtained by recording the current audio processing callback trigger time and subtracting it from the previous audio processing callback trigger time. The theoretical expected callback interval is equal to the processing block frame length divided by the sampling rate. Then, during operation, the actual time interval is recorded for each pair of adjacent audio processing callbacks, and the absolute value of the difference between the actual time interval and the theoretical expected callback interval is divided by the theoretical expected callback interval to obtain the deviation ratio. The deviation ratio is compared with a jitter ratio threshold; when the deviation ratio is greater than the jitter ratio threshold, callback jitter is determined to have occurred; when the deviation ratio is not greater than the jitter ratio threshold, callback jitter is determined not to have occurred.

[0138] The method for setting the jitter ratio threshold is as follows: During the stable operation phase, collect the most recent preset number of actual callback interval samples, calculate the deviation ratio of each sample relative to the theoretically expected callback interval, sort them from smallest to largest, take the value corresponding to the preset percentile, and add a preset compensation amount to obtain the jitter ratio threshold; the preset compensation amount is a fixed value of 0.05, used to add a safety margin based on the historical normal fluctuation range; then, upper and lower limit constraints are applied to the jitter ratio threshold. Preferably, the preset percentile is the 95th percentile, the preset compensation amount is from 0.02 to 0.08, more preferably 0.05, and the lower limit of the jitter ratio threshold is 0.25, and the upper limit is 0.50.

[0139] The stable operation phase refers to the operation phase in which, as judged by those skilled in the art based on the changes in the actual callback intervals in multiple consecutive audio processing callbacks, the actual callback intervals remain continuous, the changes in adjacent intervals are gradual, and there are no obvious sudden increases or decreases in intervals.

[0140] After determining that a callback jitter has occurred, the change range of the lip-sync parameter corresponding to the current output moment is limited.

[0141] The method for implementing the variation range limit includes:

[0142] The differences between the lip shape parameter values ​​at the current output time and the lip shape parameter values ​​of the previous rendering cycle are calculated for lip closure degree, lip opening and closing height, lip rounding degree, lateral traction of the corners of the mouth, and mandibular displacement. The absolute value of the difference for each lip shape parameter is then compared with the maximum allowable change for that lip shape parameter. When the absolute value of the difference for a certain lip shape parameter is not greater than the maximum allowable change for that lip shape parameter, that lip shape parameter value is used as the output value for the current rendering cycle. When the absolute value of the difference for a certain lip shape parameter is greater than the maximum allowable change for that lip shape parameter, the difference is truncated to the maximum allowable change while retaining its original sign, and then added to the lip shape parameter value of the previous rendering cycle to obtain the restricted output value for that lip shape parameter in the current rendering cycle. The restricted output value is the output value of the current rendering cycle after applying the variation amplitude limit. Through the above processing, the lip shape parameters obtained by the time interpolation and the lip shape parameters obtained by the prediction advancement remain continuously changing within the current rendering cycle.

[0143] The method for obtaining the maximum allowable change is as follows: Collect the most recent preset number of frame lip-shape parameter samples in the stable lip-shape segment; calculate the absolute value of the difference between adjacent frames for each dimension of the lip-shape parameter; and take the value corresponding to a preset quantile as the initial maximum allowable change for that dimension of the lip-shape parameter. When a callback jitter is detected, tighten the initial maximum allowable change for that dimension of the lip-shape parameter to a preset proportion range, and apply the aforementioned change amplitude limit according to the tightened maximum allowable change. Preferably, the change amplitude quantile is taken as the 95th percentile, and the preset proportion range is 60% to 80% of the initial maximum allowable change.

[0144] The stable lip-sync segment refers to the lip-sync output segment that is not within the two speech frames before and after the boundary of the adjacent vocal unit category switching, the current lip-sync advance clock does not exceed the range of the lip-sync parameters that have been generated, and no callback jitter has been determined to have occurred.

[0145] When callback jitter is detected, the target occupancy depth is adjusted upwards. Specifically, a preset compensation depth is added to the current target occupancy depth to allow for a larger query margin in the lip-shape parameter circular buffer. When no callback jitter is detected, the target occupancy depth is obtained by using the callback fluctuation coverage benchmark and the preset safety margin.

[0146] The method for obtaining the preset compensation depth is as follows: During historical operation, the gap between the length of the unread area of ​​the lip-shape parameter circular buffer and the target occupied depth during the period when callback jitter occurs is statistically determined, and the compensation depth percentile value corresponding to the query margin gap obtained by converting the gap is used as the preset compensation depth. Specifically, the difference between the number of records corresponding to the target occupied depth and the actual unread area length during each period when callback jitter occurs is recorded, and then the difference is converted into the corresponding time length to obtain the query margin gap corresponding to each callback jitter; subsequently, the query margin gaps are sorted in ascending order, and the compensation depth percentile value is taken as the preset compensation depth. Preferably, the compensation depth percentile is taken from the 90th percentile to the 95th percentile. More preferably, the preset compensation depth is also subject to a lower limit constraint, making it not less than the time length corresponding to a fixed rendering cycle.

[0147] Through the above processing, when the audio processing callback fluctuation increases, the target occupancy depth and the change amplitude limit are synchronized, thereby reducing the lip-sync jumps and subsequent query gaps caused by callback jitter.

[0148] Example 2

[0149] See Figure 3 As shown, this embodiment provides a digital human lip-sound synchronization driving system, and the implementation of the digital human lip-sound synchronization driving method includes:

[0150] The time construction module is used to obtain the audio stream sampling rate and the actual number of output sampling frames corresponding to each audio processing callback, and to construct a continuous audio playback timeline based on the audio stream sampling rate and the actual number of output sampling frames.

[0151] Clock generation module: used to generate a lip-sync clock based on the continuous audio playback timeline;

[0152] The lip shape generation module is used to perform frame processing on the speech to be played to extract speech frame features, to perform pronunciation expansion on the text to be played to obtain the pronunciation context, and to generate a time-labeled lip shape parameter sequence based on the speech frame features and pronunciation context. The lip shape parameters are parameters that characterize the changes in the shape of the digital human's mouth.

[0153] Buffer writing module: Used to write the sequence of lip shape parameters with timestamps into the circular buffer of lip shape parameters in chronological order and set the target occupancy depth;

[0154] Parameter query module: used to determine the current output time based on the lip shape advance clock, and query the lip shape parameters corresponding to the current output time from the lip shape parameter circular buffer;

[0155] Predictive propulsion module: Used to perform predictive propulsion to determine the lip parameters corresponding to the current output time when the current output time exceeds the range of generated lip parameters;

[0156] Jitter Detection Module: Used to obtain the actual callback interval and the theoretical expected callback interval, and to determine whether callback jitter has occurred based on the deviation ratio between the actual callback interval and the theoretical expected callback interval;

[0157] Jitter control module: When a callback jitter is detected, it limits the change in the lip shape parameters corresponding to the current output moment and adjusts the target occupancy depth according to the degree of jitter.

[0158] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0159] In conclusion, the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A digital human lip-sound synchronization driving method, characterized in that, include: Obtain the sampling rate of the audio stream and the actual number of output sampling frames corresponding to each audio processing callback, and construct a continuous audio playback timeline based on the sampling rate of the audio stream and the actual number of output sampling frames; A lip-sync advance clock is generated based on the continuous audio playback timeline; The speech to be played is framed to extract speech frame features, and the text to be played is expanded to obtain the pronunciation context. Based on the speech frame features and pronunciation context, a time-labeled mouth shape parameter sequence is generated, where the mouth shape parameter is a parameter that characterizes the changes in the shape of the digital human's mouth. Write the time-tagged lip-sync parameter sequence into the lip-sync parameter circular buffer in chronological order and set the target occupancy depth; The current output time is determined based on the lip-sync advance clock, and the lip-sync parameters corresponding to the current output time are retrieved from the lip-sync parameter circular buffer. When the current output time exceeds the range of generated mouth shape parameters, perform predictive propulsion to determine the mouth shape parameters corresponding to the current output time. Obtain the actual callback interval and the theoretical expected callback interval, and determine whether callback jitter has occurred based on the deviation ratio between the actual callback interval and the theoretical expected callback interval; When a callback jitter is detected, the change range of the lip-sync parameter corresponding to the current output moment is limited, and the target occupancy depth is adjusted.

2. The digital human lip-sound synchronization driving method according to claim 1, characterized in that, The method for constructing a continuous audio playback timeline includes: Before audio playback begins, the sampling rate of the current audio stream and the processing block frame length of the audio processing thread are read, and the cumulative output sample frame count register is set in the audio processing thread. The cumulative output sample frame count register is used to store the total number of sample frames that have been sent to the audio output link since the start of playback. Each time an audio processing callback occurs, the number of sampled frames actually output this time is read and added to the cumulative output sampled frame count register. After the accumulation is completed, the cumulative number of output sample frames is divided by the sampling rate to obtain the position of the continuous audio playback timeline corresponding to the current audio playback moment.

3. The digital human lip-sound synchronization driving method according to claim 1, characterized in that, The method for generating the lip-sync advance clock is as follows: During the initial playback phase, the execution time of the current rendering frame is continuously recorded according to a fixed rendering beat. The fixed rendering beat is the time beat at which the graphics rendering side performs a lip-sync update once at a preset time interval. The position of the continuous audio playback timeline closest to the current rendering frame is recorded in the audio processing thread. The time difference between the execution time of the current rendering frame and the corresponding moment of the continuous audio playback timeline is recorded as a delay sample. After the data collection is completed, the maximum and minimum values ​​in the delayed samples are removed, and the average value of the remaining delayed samples is calculated. The resulting average value is then used as the synchronization offset. Subtract the synchronization offset from the continuous audio playback timeline to obtain the lip-sync advance clock.

4. The digital human lip-sound synchronization driving method according to claim 3, characterized in that, After generating the lip-sync advance clock, ensure that the lip-sync advance clock remains consistent with the actual rendering state during continuous operation and update the synchronization offset; The specific implementation method for updating the synchronization offset is as follows: Using a preset time period as a monitoring window, the average rendering latency is continuously calculated over N monitoring windows. When the absolute value of the difference between the average rendering latency of N consecutive monitoring windows and the current synchronization offset is greater than the preset offset threshold, it is determined that the current synchronization offset can no longer represent the fixed delay compensation relationship between the audio output link and the graphics rendering link. Collect all delay samples across N consecutive monitoring windows, and take the median of all delay samples as the updated synchronization offset.

5. The digital human lip-sound synchronization driving method according to claim 1, characterized in that, The method for generating a time-stamped sequence of lip shape parameters includes: The audio to be played is framed to obtain a time-ordered audio frame sequence and the basic acoustic features are extracted. The text to be played is expanded into a sound, resulting in a sequence of basic phonetic units arranged in chronological order. Time-align the speech frame sequence with the basic speech unit sequence; Based on the time alignment results, the basic articulatory units are classified into categories, including closed-lip articulatory units, open-lip articulatory units, and rounded-lip articulatory units. The basic mouth shape and initial values ​​of each dimension of mouth shape parameters are determined based on the type of articulatory unit. The initial values ​​of each dimension of mouth shape parameters are corrected based on the basic acoustic characteristics to obtain the amplitude correction results of each dimension of mouth shape parameters. The amplitude correction results of each dimension of mouth shape parameters are then transitionally corrected based on the adjacent articulatory unit categories to obtain the transitional correction results of each dimension of mouth shape parameters. Finally, a mouth shape parameter sequence with time tags is generated by combining the corresponding time tags.

6. The digital human lip-sound synchronization driving method according to claim 5, characterized in that, The method for obtaining the amplitude correction results of each dimension of mouth shape parameters includes: The short-time energy, fundamental frequency variation and spectral envelope features in the basic acoustic characteristics are normalized to obtain the energy correction factor, dynamic correction factor and spectral correction factor. The height of the lips opening and closing and the amount of mandibular displacement are corrected based on the energy correction factor; the lateral traction of the corners of the mouth and the height of the lips opening and closing are corrected based on the dynamic correction factor; and the degree of lip rounding is corrected based on the spectral correction factor. The correction values ​​of each mouth shape parameter are superimposed onto the initial values ​​of the corresponding mouth shape parameters to obtain the amplitude correction results of each mouth shape parameter.

7. The digital human lip-sound synchronization driving method according to claim 5, characterized in that, The method for obtaining the transition correction results of each dimension of mouth shape parameters includes: Using the switching boundary between the current articulatory unit category and the adjacent articulatory unit category as the center, the range of transitional speech frames before and after the switching boundary is determined, and the front weight corresponding to the previous articulatory unit category and the back weight corresponding to the current articulatory unit category are determined respectively. Multiply the mouth shape parameter value corresponding to the previous articulatory unit category and the mouth shape parameter value corresponding to the current articulatory unit category by the front weight and the back weight respectively, and add the weighted results to obtain the transition correction result of each dimension of mouth shape parameter.

8. The digital human lip-sound synchronization driving method according to claim 1, characterized in that, The method for writing into the circular buffer of type parameters and setting the target occupancy depth includes: A mouth shape parameter record is formed by combining a time stamp and mouth shape parameters of each dimension. Each time a mouth shape parameter record is generated, the current write position of the mouth shape parameter circular buffer is written, and the write position is moved to the right by one record unit. When the write position reaches the end of the mouth shape parameter circular buffer, it returns to the starting position to continue writing; Count the actual interval of a preset number of consecutive audio processing callbacks and sort them from smallest to largest; The actual interval corresponding to the preset high percentile value is used as the benchmark for callback fluctuation coverage. The target occupancy depth is obtained by overlaying a preset safety margin on the basis of the pullback fluctuation coverage benchmark.

9. A digital human lip-sound synchronization driving method according to claim 8, characterized in that, The method for determining the lip shape parameters corresponding to the current output moment by performing predictive propagation includes: When the current output time is between the two sets of lip shape parameter records, the weights of the previous and subsequent records are reassigned in reverse according to the distance from the current output time to the time tags of the two sets of records. The lip shape parameters of each dimension of the two sets are weighted and summed according to the weights of the previous and subsequent records to obtain the lip shape parameters corresponding to the current output time. When the current output time exceeds the time stamp of the latest set of lip shape parameter records in the circular buffer of lip shape parameters, the time difference between the current output time and the time stamp of the latest set of lip shape parameter records is used as the lead amount; When the lead amount reaches the preset lead threshold, short-time extrapolation is performed based on the weighted summation result of the change rate of the continuously preset number of lip shape parameter records to determine the lip shape parameters corresponding to the current output time.

10. A digital human lip-sound synchronization driving method according to claim 2, characterized in that, The method for determining callback jitter includes: The theoretical expected callback interval is obtained by dividing the processing block frame length by the sampling rate. During operation, the actual time interval is recorded for each pair of adjacent audio processing callbacks, and the absolute value of the difference between the actual time interval and the theoretical expected callback interval is divided by the theoretical expected callback interval to obtain the deviation ratio. Compare the deviation ratio with the jitter ratio threshold; When the deviation ratio is greater than the jitter ratio threshold, a callback jitter is determined to have occurred; when the deviation ratio is not greater than the jitter ratio threshold, a callback jitter is determined to have not occurred.

11. A digital human lip-sound synchronization driving method according to claim 5, characterized in that, Methods for implementing variation range limits include: Calculate the difference between the lip shape parameter values ​​of the current output time and the output values ​​of the previous rendering cycle; Compare the absolute value of the difference with the maximum permissible variation corresponding to each dimension of the mouth shape parameter; When the absolute value of the difference corresponding to a one-dimensional lip shape parameter is not greater than the maximum allowable change, the corresponding lip shape parameter value will be used as the output value of the current rendering cycle. When the absolute value of the difference corresponding to a one-dimensional lip shape parameter is greater than the maximum allowable change, the difference is truncated to the maximum allowable change while maintaining its original sign, and added to the corresponding lip shape parameter value output in the previous rendering cycle to obtain the restricted output value of the corresponding lip shape parameter.

12. A digital human lip-sound synchronization driving system, implementing the digital human lip-sound synchronization driving method according to any one of claims 1-11, characterized in that, include: The time construction module is used to obtain the audio stream sampling rate and the actual number of output sampling frames corresponding to each audio processing callback, and to construct a continuous audio playback timeline based on the audio stream sampling rate and the actual number of output sampling frames. Clock generation module: used to generate a lip-sync clock based on the continuous audio playback timeline; The lip-shape generation module is used to perform frame processing on the speech to be played to extract speech frame features, to perform pronunciation expansion on the text to be played to obtain the pronunciation context, and to generate a lip-shape parameter sequence with time labels based on the speech frame features and pronunciation context. Buffer writing module: Used to write the sequence of lip shape parameters with timestamps into the circular buffer of lip shape parameters in chronological order and set the target occupancy depth; Parameter query module: used to determine the current output time based on the lip shape advance clock, and query the lip shape parameters corresponding to the current output time from the lip shape parameter circular buffer; Predictive propulsion module: Used to perform predictive propulsion to determine the lip parameters corresponding to the current output time when the current output time exceeds the range of generated lip parameters; Jitter Detection Module: Used to obtain the actual callback interval and the theoretical expected callback interval, and to determine whether callback jitter has occurred based on the deviation ratio between the actual callback interval and the theoretical expected callback interval; Jitter control module: When a callback jitter is detected, it limits the change in the lip shape parameters corresponding to the current output moment and adjusts the target occupancy depth according to the degree of jitter.

Images