Audio processing method and apparatus, training method and apparatus, device, medium

Abstract

The present disclosure provides an audio processing method and device, a training method and device, equipment and a medium, relates to the technical field of data processing, and in particular to the technical field of audio processing, lip generation, model training and the like. The specific implementation scheme is: receiving input audio, performing speech activity detection on the input audio, and separating at least one mute segment and at least one audio segment; performing audio feature extraction on the at least one mute segment and the at least one audio segment independently, respectively, to obtain mute features corresponding to each mute segment and audio features corresponding to each audio segment; merging the mute features and the audio features according to the time sequence of each segment in the input audio, and generating a lip video corresponding to the input audio based on the merged features by using a pre-trained lip generation model.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and more particularly to the fields of audio processing, lip-shape generation, and model training. Specifically, this disclosure relates to an audio processing method and apparatus, a training method and apparatus for a lip-shape generation model, an electronic device, and a computer-readable storage medium. Background Technology

[0002] In recent years, with the rapid development of deep learning technology, audio-driven lip-syncing technology has become increasingly mature and has been widely used in the field of digital human generation, including but not limited to real-time interview digital humans, live streaming digital humans, e-commerce digital humans, and educational digital humans.

[0003] In the aforementioned real-world business scenarios, digital humans often spend a lot of time waiting or remaining silent. In such cases, the ideal behavior for a digital human is to naturally shut up. Summary of the Invention

[0004] This disclosure provides an audio processing method and apparatus, a training method and apparatus for a lip-shape generation model, an electronic device, and a computer-readable storage medium.

[0005] According to a first aspect of this disclosure, an audio processing method is provided, the method comprising: Receive input audio, perform speech activity detection on the input audio, and separate at least one silent segment and at least one spoken audio segment; Audio features are extracted independently from the at least one silent segment and the at least one audio segment with sound, to obtain the silence feature corresponding to each silent segment and the audio feature corresponding to each audio segment with sound. According to the time sequence of each segment in the input audio, the silence feature and the sound audio feature are merged, and a pre-trained lip-shape generation model is used to generate a lip-shape video corresponding to the input audio based on the merged features.

[0006] According to a second aspect of this disclosure, a method for training a lip shape generation model is provided, the method comprising: Obtain silent video samples; Obtain non-silent video samples; Based on the acquired silent video samples and non-silent video samples, a training dataset is constructed; The lip shape generation model is trained based on the training dataset.

[0007] According to a third aspect of this disclosure, an audio processing apparatus is provided, the apparatus comprising: An audio separation module is used to receive input audio, perform speech activity detection on the input audio, and separate at least one silent segment and at least one audible audio segment. The feature extraction module is used to independently extract audio features from the at least one silent segment and the at least one audio segment, to obtain the silence feature corresponding to each silent segment and the audio feature corresponding to each audio segment. The feature splicing module is used to merge the silence feature and the sound audio feature according to the time order of each segment in the input audio, and use a pre-trained lip shape generation model to generate the lip shape video corresponding to the input audio based on the merged features.

[0008] According to a fourth aspect of this disclosure, a training apparatus for a lip shape generation model is provided, the apparatus comprising: The video sample acquisition module is used to acquire silent video samples and non-silent video samples. The model training module is used to construct a training dataset based on the acquired silent video samples and non-silent video samples; and to train the lip-shape generation model based on the training dataset.

[0009] According to a fifth aspect of this disclosure, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to at least one of the aforementioned processors; wherein, The memory stores instructions that can be executed by at least one processor, which, when executed by at least one processor, enable the at least one processor to perform the audio processing method and / or the training method for the lip-shape generation model.

[0010] According to a sixth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the above-described audio processing method and / or the training method for the lip-shape generation model.

[0011] According to a seventh aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the above-described audio processing method and / or the training method for the lip-shape generation model.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a schematic flowchart of an audio processing method provided in an embodiment of this disclosure; Figure 2 This is a flowchart illustrating some steps of another audio processing method provided in this embodiment of the present disclosure; Figure 3a It is the silent audio waveform of a long-duration silent audio segment; Figure 3b This is a schematic diagram of the energy distribution of a feature sequence obtained by dividing a long silent audio segment into multiple sub-segments, extracting features from each sub-segment through an audio encoder, and splicing them together in chronological order. Figure 4 This is a flowchart illustrating some steps of another audio processing method provided in this embodiment of the present disclosure; Figure 5 This is a flowchart illustrating some steps of another audio processing method provided in this embodiment of the present disclosure; Figure 6 This is a flowchart illustrating a training method for a lip shape generation model provided in an embodiment of this disclosure; Figure 7 This is a flowchart illustrating some steps of a training method for another lip shape generation model provided in this embodiment of the disclosure; Figure 8 This is a flowchart illustrating some steps of a training method for another lip shape generation model provided in this embodiment of the disclosure; Figure 9 This is a schematic diagram of the structure of an audio processing device provided in an embodiment of this disclosure; Figure 10 This is a schematic diagram of the structure of a training device for a lip shape generation model provided in an embodiment of this disclosure; Figure 11 This is a block diagram of an electronic device used to implement the audio processing method and / or the training method of the lip-shape generation model according to the embodiments of this disclosure. Detailed Implementation

[0014] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0015] In some related technologies, a video of a person and the target audio are input, an audio encoder is used to extract audio features, and these features are fed together with the face image into a lip-shape generation model. The output is a synthesized lip shape that is aligned with the audio. Finally, the synthesized face or lip area is replaced back into the original video through image fusion, thereby producing a highly realistic lip-shape video.

[0016] In short-term silent scenarios, when receiving silent input, the lip shape generation model cannot reliably generate the lip shape of the closed mouth state. The lip shape often remains in a half-open or trembling state, which is seriously inconsistent with the real static closed mouth state.

[0017] In long-duration silent scenarios, when the duration of silence exceeds the maximum single processing time of mainstream audio feature encoders, the lip shape generation model will generate obvious abnormal lip shapes near the processing boundary of the encoder and will be unable to maintain a stable closed mouth state.

[0018] In scenarios where speech and silence are interspersed within a spoken audio clip, relying on a lip-shape generation model for direct inference can easily result in lip tremors or abnormal mouth opening. If an external fixed closed-mouth frame is used for forced replacement, abrupt changes in lip shape, skin color, and facial details are likely to occur at the moment of speech-silence transition. Both methods lead to a mismatch between the visual and audio content, severely reducing the naturalness and spatiotemporal coherence of digital human videos.

[0019] The audio processing methods and apparatus, lip-shape generation model training methods and apparatus, electronic devices, and computer-readable storage media provided in this disclosure are intended to solve at least one of the above-mentioned technical problems of the prior art.

[0020] The audio processing method and lip-shape generation model training method provided in this disclosure can be executed by electronic devices such as terminal devices or servers. Terminal devices can be in-vehicle devices, user equipment (UE), mobile devices, user terminals, terminals, cellular phones, cordless phones, personal digital assistants (PDAs), handheld devices, computing devices, in-vehicle devices, wearable devices, etc. The method can be implemented by a processor calling computer-readable program instructions stored in memory. Alternatively, the method can be executed by a server.

[0021] Figure 1 A schematic flowchart of an audio processing method provided in an embodiment of this disclosure is shown. Figure 1 As shown in the figure, the audio processing method provided in this embodiment includes the following steps: S110, S120, and S130.

[0022] S110. Receive input audio, perform voice activity detection on the input audio, and separate at least one silent segment and at least one audible audio segment.

[0023] The input audio can be the target audio data for generating lip-sync videos of the digital human to be driven. The input audio can come from an audio stream generated by a text-to-speech system, real-time microphone audio data, or pre-recorded audio files.

[0024] The duration and content of the input audio depend on the specific application scenario. For example, in a real-time interview digital human scenario, the input audio can be a mute signal received during the interviewer's questioning intervals or when the digital human is listening, or it can be a response voice signal to be played after the interviewer finishes speaking and before the digital human is ready to answer.

[0025] Understandably, in actual business operations, input audio is often not a continuous signal in a single state, but a mixed signal with alternating silent and audible audio segments. For example, when a digital human is listening to a user speak, the input audio may be a relatively long period of silence; when a digital human is responding continuously, there may also be brief silence gaps in the input audio due to natural pauses between sentences.

[0026] Speech activity detection is a technique used to detect the presence of human speech signals in audio signals. Its purpose is to distinguish between silent intervals (intervals with no sound or only background noise) and audible audio intervals (intervals with valid speech signals) from continuous input audio.

[0027] Voice activity detection can be implemented using the VAD (Voice Activity Detection) algorithm of WebRTC (Web Real-Time Communications), a detection algorithm based on energy threshold and zero-crossing rate, or a detection algorithm based on a deep learning model. This disclosure does not specifically limit the implementation of such algorithms.

[0028] The detection results output by the speech activity detection can indicate whether each audio frame is valid speech or silent. For example, a detection threshold can be set. When the energy value or speech probability of an audio frame is greater than the threshold, it is determined to be valid speech; otherwise, it is determined to be silent.

[0029] A silence segment refers to a continuous audio interval in the input audio that is determined by speech activity detection to contain no valid speech signal. Depending on the actual application scenario, the duration of a silence segment varies considerably: when the digital human is listening, a silence segment may last for tens of seconds or even several minutes; while during the normal response process of the digital human, a silence segment may only be a short pause between sentences, typically lasting a few seconds.

[0030] An audio segment refers to a continuous audio interval in the input audio that is determined by speech activity detection to contain a valid speech signal. An audio segment can include signals with acoustic energy such as human voice, music, and ambient sounds. In a typical application scenario of this disclosure embodiment, the audio segment mainly refers to the speech signal for which the digital human needs lip-syncing based on its content.

[0031] An input audio may contain multiple silent segments and multiple audio segments with sound. For example, the input audio may be composed of alternating sequences such as "silent segment A - audio segment B - silent segment C - audio segment D".

[0032] S120. Extract audio features independently from at least one silent segment and at least one audio segment with sound, to obtain the silence feature corresponding to each silent segment and the audio feature corresponding to each audio segment with sound.

[0033] The separate extraction of audio features refers to extracting features from the separated silent segments and the audio segments with sound separately during the call process of different encoders.

[0034] In some specific implementations, a separate audio encoder is invoked to extract features from each silent segment; similarly, a separate audio encoder is invoked to extract features from each audible audio segment. The encoding processes for each segment are independent and do not interfere with each other, thus ensuring that the silence features are not affected by the audible audio features.

[0035] For example, if the input audio is separated into N silent segments and M audible audio segments, then N independent silent feature extractions and M independent audible audio feature extractions are performed respectively, with each extraction using only the audio data of a single segment as the input to the encoder.

[0036] The audio encoder used for audio feature extraction can be a pre-trained deep learning audio coding model, such as the Whisper model, the Wav2Vec model, or the HuBERT model. Taking the Whisper encoder as an example, this encoder is based on the Transformer (a deep learning architecture based on self-attention mechanisms) architecture and can extract deep hidden state features from the input audio. These features contain semantic information, acoustic prosody, and emotional features of the audio, which are the key basis for subsequent lip-shape generation models to predict lip shapes. The features output by the encoder are usually in time series form, with each time step corresponding to a time segment in the original audio. The dimension of the feature vector at each time step is determined by the design of the encoder.

[0037] The specific implementation method of audio feature extraction will be explained in more detail in the subsequent specification with specific embodiments.

[0038] Silence features are the feature sequences obtained after extracting audio features from silent segments. Ideally, the audio features of a pure silence signal should remain stable over time, without abnormal energy fluctuations, so that the downstream lip-shape generation model can generate a stable closed lip shape.

[0039] However, in actual processing, due to factors such as the maximum processing time limit of the encoder and the segment truncation boundary effect, the silent feature may exhibit abnormal energy spikes or jitters, which need to be repaired through subsequent smoothing processing.

[0040] Spoken audio features refer to the feature sequences obtained after extracting audio features from spoken audio segments. Spoken audio features should be able to accurately reflect information such as the phoneme composition, prosodic changes, and speech rate rhythm of the input speech, so that the lip-shape generation model can generate a sequence of lip movements that are highly synchronized with the speech.

[0041] Because silent segments and spoken audio segments have fundamentally different signal characteristics—silent segments contain no valid speech content information, and their ideal lip shape should be a naturally closed, still state—while spoken audio segments require the generation of a precisely synchronized sequence of lip movements based on specific speech content. If both are fed into an audio encoder for continuous encoding, the features corresponding to silent segments may be "contaminated" by features from adjacent spoken audio segments due to the global context modeling performed by the attention-based audio encoder during encoding. This means that residual speech activation signals are mixed into the silent features, causing the downstream generation model to produce erroneous weak mouth openings or lip tremors during silent segments, thus affecting the quality of subsequent lip generation. By separating silent segments and spoken audio segments, the information transmission from spoken segments to silent segments during attention calculation can be effectively blocked, fundamentally eliminating feature contamination.

[0042] S130. According to the time order of each segment in the input audio, merge the silence features and the sound audio features, and use the pre-trained lip shape generation model to generate the lip shape video corresponding to the input audio based on the merged features.

[0043] In this embodiment of the disclosure, merging the silence feature and the audio feature means restoring the silence feature sequence and the audio feature sequence obtained after separate extraction into a complete feature sequence aligned with the time axis of the original input audio.

[0044] In some specific implementations, the merging operation is arranged according to the original time order of each segment in the input audio, so that the merged feature sequence maintains a strict correspondence with the original input audio in the time dimension.

[0045] For example, if the input audio is composed of "silent segment A → audio segment B → silent segment C" in sequence, the merged feature sequence is also spliced ​​in the order of "silent feature A → audio segment B → silent feature C", and the start and end positions of each feature segment on the time axis are consistent with their corresponding original audio segments.

[0046] Understandably, since each segment is encoded independently in S120, the length of the feature sequence output by each segment encoder in the time dimension may not be directly proportional to the duration of its original audio segment (depending on parameters such as the encoder's downsampling rate). Therefore, when merging features, it is necessary to map the corresponding feature sequences onto a unified time axis based on the timestamp information of the original audio of each segment to ensure the temporal continuity of the merged feature sequences, so that the downstream lip-shape generation model can generate lip-shape images frame by frame in the correct temporal order.

[0047] In this embodiment of the disclosure, the lip-shape generation model refers to a pre-trained neural network model capable of generating lip-shape images synchronized with audio based on audio features. The lip-shape generation model may be, but is not limited to, a model based on a generative adversarial network (GAN), a model based on a diffusion model, or a lip-shape generation model based on other neural network architectures.

[0048] This lip shape generation model has learned the complex mapping relationship between audio features and lip shape states, as well as the reasonable transition rules of lip shape states over time, during the training phase. The training method of this lip shape generation model will be explained in detail in the subsequent instructions.

[0049] During the inference phase, the merged feature sequence is input into the lip-shape generation model. This feature sequence fully covers the entire duration of the input audio in the time dimension, including the stationary silence features corresponding to each silent segment and the speech features corresponding to each spoken audio segment. Based on this feature sequence, the lip-shape generation model generates corresponding lip-shape images frame by frame, and all frames are arranged in chronological order to form a complete lip-shape video.

[0050] For the part corresponding to the audio segment, the lip shape image generated by the lip shape generation model is precisely synchronized with the speech content; for the part corresponding to the silent segment, since the input is the independently encoded and stable silence feature, the lip shape generation model can continuously generate a stable lip shape image of a natural closed mouth state without producing abnormal lip trembling or half-open state.

[0051] The final generated lip-sync video refers to a dynamic sequence of frames representing the lip region (or the lower half of the face including the lips) of a digital human. This lip-sync video can be fused with the base facial video stream of the digital human using image fusion technology, seamlessly replacing the generated lip region with the corresponding position in the original digital human video, ultimately outputting a complete digital human speaking video.

[0052] Through the processing flow of S110 to S130 described above, this embodiment of the present disclosure provides an audio-driven lip-syncing scheme that can reliably generate a natural closed-mouth state under silent input. By performing speech activity detection on the input audio, it extracts audio features from silent segments and spoken audio segments independently, effectively preventing the silent features from being contaminated by the features of adjacent spoken audio segments. When the input audio contains silent segments, it ensures that the generated lip-syncing video maintains a natural and stable closed-mouth state during the silent period, while maintaining accurate audio-visual synchronization during the spoken period. It effectively solves the problems of lip trembling, abnormal mouth opening, and visual abrupt changes when switching between silence and speaking in related technologies, significantly improving the naturalness and spatiotemporal coherence of digital human videos.

[0053] The audio processing method provided in the embodiments of this disclosure will be described in detail below.

[0054] In some possible implementations, after separating the silence segment and before extracting audio features, the silence segment can be preprocessed in the temporal domain to further eliminate the interference of possible ambient noise on the silence features. Specifically, the temporal audio signal value corresponding to the silence segment is replaced with a zero value.

[0055] In this embodiment of the disclosure, the time-domain audio signal value refers to the amplitude value of each sampling point of the audio signal in the time domain representation. An audio signal is essentially a time sequence composed of a series of sampling points, each corresponding to the sound pressure level amplitude value at a given moment. The time domain, in contrast to the frequency domain and feature domain, refers to the original representation of the audio signal without Fourier transform or other mathematical transformations. Replacing the time-domain audio signal value with zero values ​​means forcibly setting the amplitude values ​​of all audio sampling points corresponding to the silent segment to zero.

[0056] The purpose of this operation is that the portion of the input audio that is determined to be silent by speech activity detection may not be an absolutely zero signal in the real environment, but may contain weak ambient noise. Ambient noise refers to the continuous background sounds in the environment, such as the sound of a fan running, an air conditioner running, electrical noise, street traffic noise, etc. Although the energy of these noises is usually very low, if they are directly fed into the audio encoder for feature extraction, the encoder may still extract weak, non-stationary feature signals from them. These weak feature signals may be amplified in the downstream lip-shape generation model, causing undue slight changes or tiny jitters in the generated lip shape, affecting the naturalness and stability of the lip shape in the silent state.

[0057] By uniformly replacing the time-domain audio signal values ​​of silent segments with zero values, it can be ensured that the silent segments fed into the encoder are absolutely pure zero signals. The feature sequence extracted by the encoder from the zero signal is completely stable in the time dimension, without any minute fluctuations. The downstream lip-shape generation model receives completely consistent and stable features, thus enabling it to continuously output natural and jitter-free closed lip shapes.

[0058] It should be noted that the temporal preprocessing step and the audio feature extraction of the silence segment are two independent processing steps. Temporal preprocessing is performed before feature extraction, and its function is to transform the silence segment into a clean zero signal at the audio signal level; while audio feature extraction encodes the processed zero signal (or the original audible audio signal) to obtain the corresponding feature sequence. The order of the two is as follows: temporal preprocessing is performed first (replacing the temporal audio signal values ​​corresponding to the silence segment with zero values), and then audio feature extraction is performed on the processed silence segment.

[0059] Understandably, this temporal preprocessing operation is also applicable to the generation of silent video samples during the training phase. When constructing the training dataset, after selecting consecutive closed silent segments from the video, the temporal audio signal value corresponding to the silent segment is replaced with a zero value to generate silent video samples. This ensures that the silent samples in the training data maintain consistency in signal characteristics with the silent signals actually processed during the inference phase, which is beneficial for improving the model's generalization performance in silent scenarios.

[0060] This embodiment of the disclosure eliminates the interference of environmental noise on the extraction of silent features by replacing the temporal audio signal value corresponding to the silent segment with a zero value before feature extraction, ensuring that the silent feature sequence output by the encoder is completely stable; the downstream lip shape generation model, based on the completely stable silent features, can continuously generate natural and non-shaking closed lip shape states, further improving the stability and realism of digital human lip shape videos in silent scenes.

[0061] Figure 2The diagram illustrates a flowchart of one method for extracting audio features from at least one silent segment to obtain the silence features corresponding to each silent segment. Figure 2 As shown, it may include step S210, S210. In response to the existence of a silent segment whose duration exceeds a preset threshold, the silent segment is divided into multiple sub-segments whose duration is less than the preset threshold.

[0062] In this embodiment, the preset threshold can be an upper limit of duration set according to the maximum single-processing capability of the audio encoder. Taking the Whisper encoder as an example, its maximum single-input duration is 30 seconds; audio exceeding this duration will not be processed normally by the encoder. Therefore, the preset threshold can be set to the maximum input duration supported by the audio encoder, or a safe value slightly less than that maximum input duration, such as 25 seconds or 28 seconds.

[0063] The embodiments disclosed herein do not limit the specific value of the preset threshold; those skilled in the art can make adaptive settings according to the type of audio encoder actually used.

[0064] A sub-segment is a series of consecutive audio segments obtained by dividing an extremely long silent segment in the time domain. The segmentation operation can be performed by dividing the segments into equal lengths according to a fixed duration, or an adaptive segmentation method based on audio segment detection can be used.

[0065] In some possible implementations, the duration of each segment after segmentation should be less than a preset threshold to ensure that each segment can be processed normally by the audio encoder. At the same time, the segmentation operation should ensure the continuity between segments as much as possible to avoid introducing additional signal discontinuities due to improper selection of segmentation points.

[0066] It's important to note that the phrase "in response to" means that not all silent segments require segmentation. Segmentation is only triggered when a silent segment exceeding a preset threshold in duration is detected. Silent segments shorter than the threshold do not require segmentation and can be directly fed into the audio encoder for feature extraction. This conditional logic ensures both the rationality of the processing flow and computational efficiency.

[0067] S220. Extract audio features from each sub-segment to obtain the features corresponding to each sub-segment.

[0068] In this embodiment of the disclosure, each segment is processed independently using an audio encoder for feature extraction. Each segment, after being processed by the encoder, outputs a corresponding feature sequence. The feature extraction processes for each segment are independent of each other, with each extraction using only the audio data of a single segment as input to the encoder.

[0069] Because the segmentation operation divides the originally continuous silence signal into multiple independent sub-segments, the encoder cannot perceive the contextual information of adjacent sub-segments when processing each sub-segment. For silence segments, since the silence signal itself does not contain any semantic information, the lack of context does not affect the accuracy of the silence feature.

[0070] Although the segmentation process solves the problem of input time limitation of the encoder, it introduces a new technical problem: at the feature splicing point, due to the lack of context of the truncated edge by the encoder and the edge effect of the convolution receptive field, abnormal feature energy spikes will be generated, namely the phenomenon of "boundary distortion".

[0071] Figure 3 schematically illustrates this boundary distortion phenomenon. Among them, Figure 3b The image shows the waveform of a long-duration silent audio clip. Figure 3b The image shows a schematic diagram illustrating the energy distribution of a feature sequence obtained by segmenting a long period of silence audio into multiple sub-segments, extracting features from each sub-segment through an audio encoder, and then splicing them together in chronological order.

[0072] in, Figure 3a The horizontal axis, Times(s), represents time, and the vertical axis, Amplitude, represents the audio amplitude. Ideally, the audio amplitude of a long period of silence is 0. Figure 3b The horizontal axis, Times(s), represents time, and the vertical axis, Whisper FeatureEnergy, represents the feature energy value. As can be observed in the figure, near the splicing boundary of adjacent sub-segments (the area marked by the dashed box in the figure, approximately between 29 and 31 seconds), there is a significant abnormal spike in the feature energy.

[0073] This non-speech-related feature distortion can be misinterpreted by the downstream lip-shape generation model as a sudden speech signal, causing the digital human to make incorrect mouth opening or lip trembling movements during periods when it should be silent and closed.

[0074] To address the aforementioned boundary distortion problem, this embodiment of the present disclosure performs weighted smoothing on the features of two temporally adjacent sub-segments, i.e., step S230.

[0075] S230. Perform weighted smoothing on the portion of the adjacent end regions in the corresponding features of two temporally adjacent sub-segments.

[0076] In this embodiment of the disclosure, "two temporally adjacent sub-segments" refers to two sub-segments that are in a sequential position within the original silent segment before segmentation, meaning that the end time of the preceding sub-segment and the start time of the following sub-segment are temporally continuous. Since once a silent segment is segmented, it inevitably generates multiple sub-segments that are temporally consecutive.

[0077] "Adjacent end regions" refer to the adjacent transition interval formed by the end portion of the feature sequence of the preceding sub-segment and the beginning portion of the feature sequence of the following sub-segment. This end region extends a certain range to both sides, referencing the splicing boundary of the corresponding features of the two sub-segments. The purpose of weighted smoothing is to ensure that the feature values ​​within the end region smoothly transition from one side of the transition interval to the other, eliminating abnormal energy spikes caused by encoder edge effects.

[0078] The specific implementation of weighted smoothing will be explained in detail in the subsequent instructions.

[0079] S240. According to the time order of each sub-segment in the silent segment, the features corresponding to each smoothed sub-segment are spliced ​​together to obtain the silent feature corresponding to the silent segment.

[0080] In this embodiment of the disclosure, splicing refers to combining the smoothed sub-segment features end-to-end according to their chronological order in the original silence segment. Since the end regions of each sub-segment feature have been smoothed, the feature values ​​at the splicing point transition smoothly without abnormal energy spikes. Therefore, the complete silence feature sequence after splicing remains stable in the time dimension.

[0081] Unlike weighted smoothing, which only applies to the end regions, the splicing operation covers all sub-segments generated from the segmentation of the silence segment. The splicing is based on the temporal order of the sub-segments within the original silence segment. The resulting silence feature sequence accurately characterizes the feature state of long-term silence signals, providing stable and distortion-free feature input for downstream lip-shape generation models.

[0082] Through the processing steps S210 to S240 described above, this embodiment of the present disclosure achieves effective feature extraction for long-duration silence segments. Specifically, by dividing the ultra-long silence segment into multiple sub-segments and extracting features from each sub-segment, the limitation of the maximum input duration of the audio encoder on the processing of ultra-long silence segments is solved, enabling this embodiment of the present disclosure to support silence input of arbitrary duration. By performing weighted smoothing processing on the end regions of features of adjacent sub-segments, the boundary feature energy spikes caused by segmentation and splicing are eliminated, avoiding the problem of incorrect mouth opening or lip trembling caused by feature distortion during long-duration silence in the digital human. The silence feature sequence after smoothing remains stable in the time dimension, ensuring that the downstream lip shape generation model can generate a continuously closed natural lip shape state based on this stable feature.

[0083] Figure 4 The diagram illustrates a flowchart of one implementation method for weighted smoothing of portions of adjacent end regions in the corresponding features of two temporally adjacent sub-segments, such as... Figure 4As shown, steps S410, S420, S430, and S440 may be included.

[0084] S410. Taking the splicing boundary position of the corresponding features of the two sub-segments as the center, take feature frames of a preset length before and after the splicing boundary position to form a local smooth window as the end region.

[0085] In this embodiment, the splicing boundary position refers to the intersection between the last frame of the previous sub-segment feature sequence and the first frame of the subsequent sub-segment feature sequence. This position is the area where boundary distortion is most concentrated and is also the area that needs to be focused on for smoothing. Taking the splicing boundary position as the center of the end region means that the end region expands symmetrically to both sides with this position as the reference.

[0086] A local smoothing window refers to a set of local feature frames extending forward and backward by a preset length from the splicing boundary position as the center of symmetry in the time dimension. The preset length can be set according to actual application requirements and encoder characteristics, for example, taking the number of feature frames corresponding to 1 second before and after the boundary. If the encoder's feature output frame rate is 50 frames / second, then 50 frames are taken before and after, for a total of 100 frames in the local smoothing window. The area covered by this local smoothing window is the "adjacent end region" mentioned in S230.

[0087] It should be noted that the size of the local smoothing window needs to be selected while balancing smoothing effect and computational efficiency. If the window is too small, it cannot effectively cover the entire affected area of ​​the boundary distortion, resulting in insufficient smoothing effect; if the window is too large, it may cause unnecessary modifications to the normal silence features outside the boundary distortion region, affecting the overall stability of the silence features.

[0088] Experiments have shown that taking 1 second before and after the boundary for the corresponding number of feature frames can achieve a good balance between smoothing effect and feature fidelity.

[0089] S420. Calculate reference feature values ​​based on feature frames outside the local smoothing window, and construct interpolation features within the local smoothing window.

[0090] In this embodiment of the disclosure, the feature frames outside the local smoothing window refer to stable feature frames located outside the range of the local smoothing window and close to the boundaries of both ends of the window. Since these frames are far from the splicing boundary, they are basically unaffected by the boundary distortion effect and can represent the normal feature level of the silence signal.

[0091] The reference feature value refers to a benchmark calculated based on the stable feature frames outside the aforementioned window, used to guide the construction of interpolation features. An exemplary calculation method is as follows: extract several stable feature frames from the outer front side of the local smoothing window (i.e., before the start position of the smoothing window in the previous sub-segment) and calculate their average value; simultaneously extract several stable feature frames from the outer back side of the local smoothing window (i.e., after the end position of the smoothing window in the next sub-segment) and calculate their average value; use the mean of the two average values ​​as the reference feature value. This reference feature value represents the normal silence feature level unaffected by boundary distortion.

[0092] Interpolation features refer to a sequence of gradual features constructed within a local smooth window that smoothly transitions from one end of the window to a reference feature value and then smoothly transitions from the reference feature value to the other end of the window.

[0093] For example, the interpolation features can be constructed as follows: the local smoothing window is divided into two halves. The first half constructs a gradient feature that linearly transitions from the actual feature value at the end of the previous sub-segment feature to the reference feature value. The second half constructs a gradient feature that linearly transitions from the reference feature value to the actual feature value at the beginning of the next sub-segment feature. The linear transition can be achieved using a linear interpolation algorithm, that is, by mixing the starting feature value and the target feature value proportionally according to the frame number, so that the change in feature value between adjacent frames is uniform. The interpolation feature sequence constructed in this way forms a continuous and smooth gradient transition within the local smoothing window, eliminating possible feature value jumps at the splicing boundary.

[0094] S430. Construct a Gaussian weighted distribution centered on the splicing boundary position.

[0095] In this embodiment of the disclosure, Gaussian weight distribution refers to assigning a weight value to each time position within a locally smoothing window, following the shape of a Gaussian function (normal distribution function). The standard form of the Gaussian function is: in, This is the mean (i.e., the center position) of the Gaussian function. The standard deviation is used to control the rate at which the weights decay from the center outwards. In this embodiment of the disclosure, the center of the Gaussian function is... This is set as the splicing boundary position. This means that the closer to the boundary, the greater the proportion of interpolated features in the smoothing process (i.e., the stronger the correction to the original features); the farther away from the boundary, the smaller the proportion of interpolated features in the smoothing process (i.e., more of the original features are preserved).

[0096] Boundary distortion is most severe at the splicing boundary, requiring maximum smoothing correction; however, as the distance from the boundary increases, the distortion gradually weakens, and trust in the original features should be gradually restored. The bell-shaped distribution of Gaussian weights perfectly matches this characteristic, enabling accurate repair of boundary distortion while preserving the original features far from the boundary to the greatest extent possible.

[0097] S440. Based on Gaussian weight distribution, the original features and interpolated features within the local smoothing window are weighted and mixed to obtain the smoothed features.

[0098] In this embodiment of the disclosure, weighted mixing refers to mixing the original features and interpolated features proportionally for each frame feature within the local smoothing window, based on the Gaussian weight value corresponding to that frame position. For example, for the i-th frame feature within the local smoothing window, its smoothed feature value can be expressed as: in, These are the raw features directly output by the encoder. The interpolation features constructed in S420, Let be the Gaussian weight value corresponding to the position of the i-th frame, with a value range of [0, 1].

[0099] As can be seen from the above formula, near the splicing boundary, Approaching the peak of the Gaussian function, the interpolation feature dominates, effectively eliminating anomalous energy spikes; at the two edges of the local smoothing window, Approaching zero, the original features are basically preserved, achieving a natural transition from the smooth repair area to the normal feature area.

[0100] In some specific implementations, the Gaussian weight distribution has a set upper limit on the mixing ratio, so that the weight of the interpolation feature within a preset range centered on the splicing boundary position does not exceed the upper limit.

[0101] For example, the upper limit of this mixing ratio can be set to 80% (i.e., 0.8).

[0102] Within a preset range centered on the splicing boundary, lies the core region closest to the boundary within the local smoothing window; this region represents the area with the most severe boundary distortion. The upper limit of the blending ratio ensures that at least 20% of the original feature information is retained even at the location with the most severe boundary distortion, preventing the original features from being completely replaced by interpolated features. If the original features are completely replaced by interpolated features (i.e., the blending ratio reaches 100%), the smoothed features may deviate from the overall feature distribution output by the encoder at the boundary, affecting the inference accuracy of the downstream lip-shape generation model.

[0103] By introducing a mixing ratio cap, an optimal balance can be achieved between eliminating distortion and maintaining feature authenticity, ensuring that the smoothed features both eliminate boundary energy spikes and maintain consistency with the feature distribution of the encoder's original output.

[0104] After the processing steps S410 to S440 described above, the feature sequence within the local smoothing window is restored from its original spliced ​​state with energy spikes to a stable and smooth transition state. The smoothed feature sequence changes continuously and gently over time, without any abnormal energy abrupt changes.

[0105] Specifically, by constructing a local smoothing window centered on the splicing boundary, the influence range of boundary distortion is accurately located, ensuring that smoothing only applies to the end regions that need repair, avoiding unnecessary modifications to normal silent features. By calculating reference feature values ​​based on stable feature frames outside the window and constructing interpolated features, a reasonable transition reference is provided for the boundary region, allowing the smoothed features to naturally connect with the normal features on both sides. By constructing a Gaussian weight distribution centered on the splicing boundary, spatial adaptive adjustment of smoothing intensity is achieved—maximum smoothing correction is given in the core boundary region with the most severe distortion, while the weights of the original features are gradually restored in the edge regions with weaker distortion, thus achieving the best balance between eliminating distortion and maintaining feature fidelity. By setting an upper limit for the mixing ratio, the feature distribution deviation problem that may be caused by completely replacing the original features with interpolated features in the core boundary region is avoided, further ensuring the authenticity and usability of the smoothed features.

[0106] In this embodiment of the disclosure, the maximum processing time limit of the audio encoder also needs to be considered when extracting audio features from audio segments. Unlike silent segments, audio segments contain rich speech and semantic information, and the features of different parts have temporal continuity and contextual relevance. Therefore, special attention needs to be paid to the coherence of features after segmentation and extraction.

[0107] Figure 5 This diagram illustrates a flowchart of one implementation method for extracting audio features from at least one audio segment to obtain the audio features corresponding to each audio segment. Figure 5 As shown, steps S510, S520, and S530 may be included.

[0108] S510. In response to the existence of an audio segment in the audio segment whose duration exceeds a preset threshold, the audio segment is divided into multiple sub-segments whose duration is less than the preset threshold.

[0109] In this embodiment, the basic logic for segmenting audio segments is similar to that for segmenting silent segments. The preset threshold refers to an upper limit on the duration set based on the maximum single-processing capacity of the audio encoder; it can be set to the maximum input duration supported by the audio encoder or a safe value slightly less than that maximum input duration. An audio segment refers to a continuous audio interval in the input audio that is determined by speech activity detection to contain valid speech signals. In a typical application scenario of this embodiment, the audio segment mainly refers to the speech signal for which the digital human needs lip-syncing based on its content.

[0110] Each segment after segmentation has a duration less than the preset threshold, ensuring that each segment can be processed normally by the audio encoder. The considerations differ from those for segmenting silent segments: spoken audio segments contain continuous semantic information, and the segmentation position should avoid truncation in the middle of semantic units (such as words or phrases) as much as possible to reduce the disruption of contextual coherence.

[0111] In some possible implementations, short pauses or low-energy segments detected in speech activity detection can be used as segmentation reference positions, so that segmentation occurs at natural speech gaps.

[0112] S520. Extract audio features from each sub-segment. When extracting audio features from a sub-segment, the information of the sub-segments that are temporally adjacent to the sub-segment is used as context information. Based on the information of the sub-segment and the context information, the features corresponding to the sub-segment are extracted.

[0113] In this embodiment of the disclosure, when extracting audio features from each sub-segment, each sub-segment is not fed into the audio encoder in complete isolation. Instead, a context-aware feature extraction method is adopted. Context information refers to the audio information of sub-segments that are temporally adjacent to the current sub-segment. Using the information of temporally adjacent sub-segments as context information means that during feature extraction, the encoder can refer to the information of adjacent sub-segments to assist in the feature encoding of the current sub-segment.

[0114] The phrase "extracting features corresponding to a sub-segment based on its information and contextual information" clarifies the operational relationship between contextual information and feature extraction: contextual information is not an isolated label, but rather serves as one of the input bases for feature extraction, participating in the feature extraction process together with the information of the sub-segment itself. This approach enables the encoder to perceive the signal continuity between the sub-segment and its preceding and following adjacent segments during the encoding process, thereby outputting a feature sequence that remains coherent in the time dimension.

[0115] Taking three temporally consecutive sub-segments A, B, and C as an example: When extracting features from sub-segment B, the end portion of sub-segment A (e.g., the last 0.5 seconds of audio information) and the beginning portion of sub-segment C (e.g., the first 0.5 seconds of audio information) are used as context information and input into the encoder along with the complete audio information of sub-segment B. The encoder extracts the features corresponding to sub-segment B based on its own information and the context information of sub-segments A and C. Because the encoder can capture the transition relationships between B and A, and between B and C, the feature sequence of sub-segment B can maintain a reasonable connection with the features of adjacent sub-segments at its beginning and end, thus maintaining the temporal continuity of the features of each sub-segment.

[0116] S530. According to the time order of each sub-segment in the audio segment, the features corresponding to each sub-segment are spliced ​​together to obtain the audio features corresponding to the audio segment.

[0117] In this embodiment, concatenation refers to combining the features corresponding to each sub-segment according to their chronological order in the original audio segment. Since the feature extraction of each sub-segment incorporates contextual information from adjacent sub-segments, the transition between features at the concatenation point is natural and smooth, avoiding feature jumps caused by contextual breaks. The resulting complete audio feature sequence can accurately and coherently represent the speech information of the entire audio segment in the time dimension, providing high-quality feature input for the downstream lip-shape generation model.

[0118] Through the processing steps S510 to S530 described above, this embodiment of the present disclosure achieves effective feature extraction of ultra-long audio segments. By segmenting the ultra-long audio segment into multiple sub-segments and extracting features from each sub-segment, the limitation of the maximum input duration of the audio encoder on the processing of ultra-long audio segments is solved. By introducing information from adjacent sub-segments as contextual information when extracting features from each sub-segment, the context breakage problem caused by forced segmentation is effectively alleviated, and the temporal continuity of the features of each sub-segment is maintained. Combined with the aforementioned processing scheme for silent segments, this embodiment of the present disclosure can reasonably process segments of any length and type in the input audio, ensuring that the final merged feature sequence is both accurate and stable as a whole.

[0119] Figure 6 The diagram illustrates the steps of a training method for a lip shape generation model provided in this embodiment of the present disclosure, as shown below. Figure 6 As shown, the training method for the lip shape generation model provided in this embodiment may include steps S610, S620, S630, and S640.

[0120] S610, Obtain a silent video sample.

[0121] In this embodiment of the disclosure, a silent video sample refers to video data used to train the lip-shape generation model, in which the audio is silent and the person's lips are in a naturally closed state.

[0122] Unlike related technologies that only use spoken videos as training data, this disclosure specifically constructs silent video samples. The introduction of silent video samples is to address a fundamental deficiency in related technologies: existing training data is almost entirely spoken videos, with a severe lack of high-quality silent training samples, resulting in unpredictable model behavior when encountering silent input—the model cannot reliably generate a closed-mouth state, and the lip shape often remains in a half-open or trembling state.

[0123] The audio portion of the silent video samples consists of a pure silent signal (all-zero or near-all-zero signal), while the video portion comprises a continuous sequence of frames showing a person with their lips naturally closed. A single silent video sample contains several frames, in which the person's lips are naturally closed, corresponding to an audio signal value of zero. By including these samples in the training dataset, the model can learn the mapping relationship between silent input and closed lip shape during training, fundamentally solving the problem of lip shape instability in silent states.

[0124] The specific method for obtaining the silent video sample will be explained in detail in the subsequent instructions.

[0125] S620, Obtain non-silent video samples.

[0126] In this embodiment of the disclosure, non-silent video samples refer to video data containing a person's speaking actions and corresponding audio, used to train the lip-shape generation model. Non-silent video samples are conventional speaking video samples, whose audio portion contains valid speech signals, and whose video portion is a sequence of lip-shape changes when a person speaks, with a temporal synchronization relationship between the audio and video.

[0127] Non-silent video samples can be derived from publicly available speaker video datasets, such as HDTF (High Definition Talking Face) and VFHQ (Video Face High Quality), or from speaking video footage recorded for a specific digital human avatar. The methods for obtaining non-silent video samples can follow conventional practices in related technologies, such as downloading directly from publicly available datasets or capturing data using recording devices; these will not be elaborated upon here.

[0128] S630. Based on the acquired silent video samples and non-silent video samples, construct a training dataset.

[0129] In this embodiment of the disclosure, the training dataset consists of both silent and non-silent video samples. Unlike related technologies that only use speaking videos (i.e., non-silent video samples in this embodiment of the disclosure) to construct the training dataset, this embodiment of the disclosure adds dedicated silent video samples to the training dataset.

[0130] The process of constructing the training dataset includes: summarizing all acquired silent and non-silent video samples to form a sample set; performing uniform preprocessing operations on the samples according to training requirements, such as uniform resolution, uniform frame rate, and uniform audio sampling rate; and labeling the samples, with labeling information including the type of each sample (silent sample or non-silent sample), the start and end times of silence in the audio, and the coordinates of lip-shaped key points in the video.

[0131] The composition of the training dataset ensures that the data distribution more closely reflects the real-world situation where silence and speech alternate in actual business scenarios. During the actual construction process, the ratio of silent video samples to non-silent video samples in the training dataset can be reasonably set based on the statistical distribution of the proportion of silence in the business scenario. For example, if the digital human spends approximately 15% of its time in a silent listening state in actual business, the proportion of silent video samples in the training dataset can be set accordingly to around 15%, so that the data distribution encountered by the model during the training phase matches the actual situation during the inference phase. Of course, this disclosure does not limit the specific ratio, and those skilled in the art can flexibly adjust it according to the actual application scenario.

[0132] S640. Train the lip shape generation model based on the training dataset.

[0133] In this embodiment of the disclosure, a lip-shape generation model is trained based on a constructed training dataset. The lip-shape generation model can be, but is not limited to, a model based on a generative adversarial network, a model based on a diffusion model, or a lip-shape generation model based on other neural network architectures. During training, the model receives video samples from the training dataset as input and learns the mapping relationship from audio features to lip-shape images.

[0134] For non-silent video samples, the model learns the mapping relationship between spoken audio features and lip shapes, that is, it generates a sequence of lip shape changes that are precisely synchronized with the speech content based on the input speech features. For silent video samples, the model learns the mapping relationship between silence features and closed lip shapes, that is, it generates a stable, non-shaky, natural closed lip shape based on the input silence features.

[0135] By mixing silent and non-silent video samples for training, the model fully learns lip-shape generation capabilities in both states during the training phase. This avoids the unpredictable silence behavior problem caused by models in related technologies that only learn from spoken data and have never encountered silent data. Specific training details, such as sampling strategies, loss function design, and optimizer configuration, will be explained in more detail in subsequent specifications.

[0136] The lip-shape generation model training method provided in this disclosure fundamentally solves the deficiency of high-quality silent, closed-mouth samples in related technologies by specifically acquiring and introducing silent video samples. During the training phase, the model has the opportunity to fully learn the mapping relationship between silent input and closed-mouth lip shape, thus reliably generating a natural closed-mouth state when faced with silent input during the inference phase, rather than remaining in a half-open or trembling state. By constructing a training dataset using both silent and non-silent video samples, the distribution of the training data more closely resembles real-world business scenarios. In actual digital human applications, silence and speaking are two alternating basic states. The mixed training dataset enables the model to simultaneously master lip-shape generation capabilities in both states, providing a foundation for subsequent differentiated processing. By training the lip-shape generation model on a hybrid training dataset, the model establishes a correct response pattern to silent input during the training phase. This, combined with the independent encoding of silent segments during the inference phase, forms a closed-loop "training-inference" technology. The training phase lays the foundation for the model's ability to correctly process silent features, while the inference phase ensures that silent features are not contaminated through VAD separation and independent encoding. The two work together to achieve stable closed-lip shape output of the digital human in silent scenarios of any duration.

[0137] The training method for the lip shape generation model provided in this embodiment will be described in detail below.

[0138] Figure 7 A flowchart illustrating one method for obtaining silent video samples is shown, such as... Figure 7 As shown, steps S710, S720, S730, and S740 may be included.

[0139] S710: Acquire videos containing spoken and silent segments.

[0140] In this embodiment of the disclosure, a video containing both speaking and silent segments refers to video footage that includes both speaking actions and natural silence. A speaking segment refers to the time period in the video where a person is speaking and their lips are moving accordingly; a silent segment refers to the time period in the video where a person stops speaking and their lips are naturally closed. In normal speaking videos, people naturally close their mouths during pauses in speaking (such as between sentences or during periods of thought), and these silent segments are the source material for constructing silent video samples.

[0141] These videos can originate from publicly available speaker video datasets or from video footage specifically recorded for a particular digital human avatar. The video capture environment can be an indoor recording studio, an office environment, or other practical application scenarios. It should be understood that the spoken segments in the video provide material for subsequently obtaining non-silent video samples (when spoken segments are extracted individually, they can be used as non-silent video samples), while the silent segments in the video provide material for obtaining silent video samples.

[0142] S720: Perform face detection on the video, determine the face region, perform facial key point detection within the face region, and obtain lip key points.

[0143] In this embodiment, face detection refers to the technique of locating faces from video frames. Face detection can process each frame of the video frame by frame, outputting the bounding box of the face in each frame, i.e., the face region. Face detection can be implemented using deep learning face detection algorithms, such as MTCNN (Multi-task Cascaded Convolutional Networks) algorithm, RetinaFace algorithm, etc. This embodiment does not limit the specific type of face detection algorithm. When multiple faces are detected in a video frame, the face region of the target person can be determined by heuristic rules such as face size and face position in the frame, for example, selecting the face in the center of the frame with the largest face size as the target person.

[0144] After determining the face region, facial landmark detection is further performed within the face region. Facial landmark detection refers to the technique of locating a predetermined number of facial feature points from a face image. These feature points typically include the contour points of facial organs such as eyebrows, eyes, nose, and lips. Facial landmark detection can be implemented based on algorithms such as MediaPipe and FaceAlignment. This disclosure does not limit the specific type of facial landmark detection algorithm.

[0145] Lip keypoints refer to the keypoints located in the lip region of the face. Taking a common 68-point facial keypoint model as an example, lip keypoints are typically the 48th to 67th keypoints, totaling 20 points. These keypoints cover the upper edge of the upper lip, the lower edge of the lower lip, and the left and right corners of the mouth, among other lip contour positions. After obtaining the lip keypoints, subsequent quantitative analysis of the mouth's state can be performed based on the coordinate information of these points.

[0146] It should be noted that facial landmark detection is performed within the face region determined by face detection, and the two processes are sequentially dependent: first, face detection locates the area containing the face from the entire video frame, and then facial landmark detection further locates facial feature points such as the lips within that area. This two-step detection approach improves the accuracy and efficiency of landmark detection.

[0147] S730: Calculate the mouth aspect ratio based on lip key points, and identify frames with a mouth aspect ratio lower than a preset closure threshold as closed frames.

[0148] In this embodiment, MAR (Mouth Aspect Ratio) refers to a numerical indicator calculated based on lip key points to quantify the degree of mouth opening. Compared to simple mouth opening (which only calculates the absolute distance between the upper and lower lips, such as 1 cm), MAR, by calculating the ratio of mouth height to mouth width, can more stably represent the closed mouth state and has better robustness to changes in mouth shape and size caused by factors such as speaking, changes in head posture, and changes in the distance between the face and the camera.

[0149] MAR can be calculated in several ways. One exemplary method is as follows: Take the vertical distance between the midpoint of the upper lip (the key point at the center of the upper edge of the upper lip) and the midpoint of the lower lip (the key point at the center of the lower edge of the lower lip) as the mouth height; take the horizontal distance between the left and right corners of the mouth as the mouth width; divide the mouth height by the mouth width to obtain the MAR value. The calculation formula can be expressed as: in, Let these be the coordinates of the midpoint of the upper lip. Let these be the coordinates of the midpoint of the lower lip. The coordinates of the left corner of the mouth are... The coordinates of the right corner of the mouth are... This represents the Euclidean distance between two points.

[0150] When the mouth is fully closed, the vertical distance between the midpoints of the upper and lower lips is minimal, and the MAR value approaches 0. When the mouth is open, the vertical distance between the midpoints of the upper and lower lips increases, and the MAR value increases accordingly. The MAR value is positively correlated with the degree of mouth opening.

[0151] In this embodiment of the disclosure, the preset closure threshold refers to the upper limit of the MAR value used to determine whether a person is in a closed-mouth state. When the MAR value of a frame is lower than the preset closure threshold, the frame is identified as a closed frame. A closed frame refers to a frame image in a video where a person is in a natural closed-mouth state.

[0152] For example, based on actual measurements of a large amount of video data, setting the preset closure threshold to 0.40 can accurately distinguish between a natural closed mouth state and non-closed mouth states such as speaking or slightly open mouth: when MAR<0.40, the person is in a closed mouth state; when MAR≥0.40, the person is in a speaking or slightly open mouth state.

[0153] It is worth noting that the specific value of the preset closure threshold can be adjusted according to actual application needs. Different digital human images (such as different lip thicknesses), different key point detection algorithm accuracies, and different application scenarios' criteria for judging the closed mouth state may all affect the selection of the optimal threshold. The value of 0.40 in this embodiment is an optimal value verified by a large number of actual tests, but this application does not impose a rigid limit on the specific value of the closure threshold.

[0154] S740: Filter out silent segments from the video that consist of consecutive closed frames and last for a duration exceeding a preset time; generate silent video samples based on the silent segments.

[0155] In this embodiment of the disclosure, the purpose of filtering consecutive closed frames is to extract the continuous time periods during which a person maintains a naturally closed mouth from the video. A single frame's closed state may only be a momentary closure during speech (such as the brief lip closure before or after uttering the plosive sounds "b" or "p"), and the training value of such momentary states is limited—it does not represent a truly natural state of silence. What truly has training value are those long-lasting, stable states of naturally closed mouth.

[0156] Therefore, this embodiment sets a preset duration as a filtering condition. A consecutive closed frame refers to a sequence of frames in a video where multiple consecutive frames are identified as closed frames; that is, these frames are arranged in chronological order, and the MAR value of each frame is lower than a preset closure threshold. Only when the duration of consecutive closed frames exceeds the preset duration is the video segment extracted as a valuable silent segment.

[0157] For example, the preset duration can be set to 2 seconds. According to actual testing, segments with lips closed for at least 2 consecutive seconds are needed to form a stable, natural closed-mouth state; closed states shorter than 2 seconds are usually just short pauses or gaps in normal speech, insufficient as qualified training samples for silence. By setting this preset duration constraint, momentary closed-mouth states during speech gaps can be effectively filtered out, ensuring that the extracted silent segments have sufficient continuity and stability. A silent segment is a selected video clip consisting of long, continuous closed frames, characterized by a person maintaining a natural closed mouth and audio characteristics of the corresponding original ambient audio (which may be actual silence or may include ambient noise).

[0158] Based on the selected silent segments, silent video samples are further generated for model training. Silent segments are naturally occurring silent periods in the video, where the video portion shows the person speaking quietly, but the audio portion may not be a pure silent signal—the original audio of the silent segment may contain weak signals such as ambient noise and microphone background noise. Therefore, the audio portion needs to be processed to generate training samples consistent with the characteristics of the silent signals actually processed during the inference stage.

[0159] The specific method for generating silent video samples will be explained in detail in the subsequent instructions.

[0160] Through the processing steps S710 to S740 described above, this embodiment of the present disclosure realizes a complete process for automatically extracting and generating silent video samples from ordinary videos containing speaking and silent segments. This automated process can efficiently construct silent training data from massive amounts of video data without manual annotation, solving the problem of a severe lack of high-quality silent training samples in the prior art.

[0161] In some possible implementations, the step of generating silent video samples based on silent segments is specifically achieved by replacing the time-domain audio signal value corresponding to the silent segment with a zero value to generate silent video samples.

[0162] In this embodiment, the time-domain audio signal value refers to the amplitude value of each sampling point of the audio signal in the time domain representation. Replacing the time-domain audio signal value corresponding to a silent segment with a zero value means that the amplitude values ​​of all audio sampling points corresponding to that silent segment are forcibly set to zero. This process makes the audio portion of the silent segment an absolutely pure zero signal, which, together with its video portion (the person naturally closing their mouth), constitutes a complete silent video sample.

[0163] The reason for performing temporal zeroing on the audio of silent segments is that the audio of silent segments in the original video is not an absolutely zero signal. In a real recording environment, even if the person is not speaking, the microphone will still pick up weak ambient noise (such as air conditioner noise, fan noise, electrical noise, etc.) and microphone background noise. If audio containing this weak noise is directly used as training data, the trained model may exhibit behavioral bias when faced with absolutely pure silent input during the inference stage. This is because the weak noise in the "silent samples" during the training stage may be learned by the model as "normal features" in a silent state, while the silent signal actually processed during the inference stage is completely zero after temporal zeroing, resulting in a difference in distribution between the two.

[0164] By replacing the time-domain audio signal value corresponding to the silent segment with a zero value, it can be ensured that the silent audio in the training data is completely consistent with the silent audio actually processed in the inference stage (the silent segments are also zeroed in the time domain during the inference stage), thereby eliminating the data distribution difference between training and inference and improving the generalization performance of the model in silent scenarios.

[0165] The generated silent video sample consists of: a video portion, which is a selected sequence of consecutive closed frames (the person's mouth remains naturally closed); and an audio portion, which is a zero-signal with a duration corresponding to the video portion. This silent video sample can be directly used to construct the training dataset.

[0166] This automated process extracts silent samples from ordinary videos containing spoken and silent segments, efficiently constructing large-scale silent training data without manual annotation. The entire process, including face detection, facial landmark detection, MAR calculation, closed frame recognition, and continuous closed frame selection, can be completed automatically, effectively reducing the manual cost of training data construction.

[0167] By introducing the mouth width-to-height ratio (MAR) index and setting a preset closure threshold (e.g., 0.40), an objective and quantitative determination of the closed mouth state is achieved. Compared with the simple mouth opening degree (absolute distance), MAR, as a relative ratio index, has better robustness to factors such as changes in the distance between the face and the camera and changes in head posture, and can maintain stable determination accuracy under different scenarios and different human figures.

[0168] By setting a preset duration (e.g., 2 seconds) for consecutive closed frames as a filtering condition, the system effectively filters out momentary lip-closure states caused by pronunciation needs during speech, ensuring that the extracted silent segments represent genuine, natural silence. This filtering condition guarantees the quality of the silent training data, enabling the model to learn stable, continuous lip-closure features, rather than momentary, unnatural lip-closure actions.

[0169] By replacing the temporal audio signal value corresponding to the silent segment with a zero value, the generated silent video sample is ensured to be completely consistent with the silent signal actually processed in the inference stage in terms of audio characteristics (the silent segment is also zeroed in the temporal domain in the inference stage, as described in claim 5), thus eliminating the data distribution difference between training and inference, and effectively improving the generalization performance and lip-shape generation stability of the model in silent scenarios.

[0170] Figure 8 The diagram illustrates a flowchart of one method for training a lip shape generation model based on a training dataset. Figure 8 As shown, a differentiated sampling strategy can be used to sample from the training dataset, which may specifically include steps S810, S820, and S830.

[0171] The differentiated sampling strategy refers to designing different sampling rules for silent and non-silent video samples, rather than using a uniform sampling method for all samples in the training dataset. The fundamental reason for this differentiated design is that non-silent video samples and silent video samples have fundamentally different data characteristics. If the same sampling strategy is used, the model will learn incorrect mapping relationships in some scenarios.

[0172] S810. For non-silent video samples, select multiple frames as training batches in a way that maximizes the difference in lip shape between frames.

[0173] In this embodiment of the disclosure, the non-silent video sample is a spoken video sample, in which the lip shape state between adjacent frames in the video frame sequence dynamically changes with the spoken content. Inter-frame lip shape difference refers to the degree of difference in lip shape state between different frames selected from the non-silent video sample. The difference in lip shape state can be quantified by comparing the distance between the coordinates of the lip keypoints in corresponding frames: the greater the sum of the Euclidean distances between the corresponding lip keypoint coordinates of two frames, the greater the lip shape difference between the two frames.

[0174] Maximizing inter-frame lip-shape difference refers to selecting a set of consecutive frames from non-silent video samples during sampling that have the greatest possible difference in lip shape among themselves. For example, the number of consecutive frames selected can be 5 or 16. The technical purpose is to prevent the "lip-shape leakage" problem.

[0175] Lip leakage refers to a situation where the model learns a "shortcut" during training—simply copying the lip shape of the input reference frame as the output, instead of regenerating the lip shape based on the input audio. This problem arises because if random sampling is used for the spoken video, the model easily samples adjacent frames with minimal lip shape differences (e.g., the lip shapes are almost identical between adjacent frames). In this case, even without relying on the input audio, the model can achieve a small difference between the output and the target simply by copying the lip shape of the input frame, resulting in a low loss value during training. This discourages the model from learning the true mapping between audio features and lip shape changes. During inference, when the model needs to modify the lip shape in the reference image based on the input audio, the generated lip shape often doesn't match the input audio because the model has never truly learned this audio-based modification ability, leading to audio-visual desynchronization.

[0176] By forcibly sampling the frame combination with the largest lip shape differences, the lip shape difference between the input reference frame and the target frame is made large enough that the model cannot simply copy the input frame to "get away with it." It must learn to modify the lip shape in the reference image according to the input audio. In this way, the model establishes a true mapping relationship between audio features and lip shape changes during training, and can accurately generate a matching lip shape sequence based on the input audio during the inference phase.

[0177] S820. For silent video samples, multiple frames are selected as training batches using random sampling.

[0178] In this embodiment, all frames of the silent video sample exhibit a natural closed lip shape with almost no variation between frames. If the same difference-maximizing sampling strategy is applied to the silent video sample as to the non-silent video sample, another technical problem arises: because the lip shapes of all frames in the silent video sample are highly similar, the difference-maximizing strategy forces the sampling algorithm to "forcefully" find the frame with the largest difference in a sequence of almost identical frames. This process may amplify false lip shape differences caused by minor changes in lighting, slight jitter in the face detection bounding box, and random errors in keypoint detection, causing the model to learn unnecessary and non-existent lip shape change trends during silent training. During the inference phase, when the model encounters consistently stable silent features, the false change trends learned during training are activated, causing the model to produce abnormal lip shape jitter instead of a stable closed lip shape.

[0179] Therefore, a random sampling strategy is adopted for the silent video samples. Random sampling refers to randomly selecting a number of frames from the silent video samples as training batches, without setting any constraints on the differences between frames. Since the lip shape in all frames of the silent samples is in a naturally closed state, random sampling enables the model to learn the characteristic that the lip shape remains stable in the silent state—regardless of which frames are selected, the input and output lip shapes are in a closed state. Therefore, the model learns the correct mapping relationship that "silent input corresponds to closed lip shape output".

[0180] S830. Mix the training batches corresponding to non-silent video samples and the training batches corresponding to silent video samples, and feed them into the lip-shape generation model for training.

[0181] In this embodiment of the disclosure, "mixing" refers to the training batches sampled from non-silent video samples in a manner that maximizes the inter-frame lip shape difference, and the training batches sampled from silent video samples in a random manner, which together constitute the input data for a training iteration.

[0182] A training batch refers to the set of samples used for a single model parameter update. A training batch can contain frame groups from multiple non-silent video samples and frame groups from multiple silent video samples. The mixing method can be flexibly set: each training batch can simultaneously include frame groups of both types of samples, allowing the model to learn the lip-shape generation capability under two states in a single parameter update; alternatively, the two types of samples can be used alternately in different training batches, for example, one batch can consist entirely of non-silent sample frame groups, and the next batch can consist entirely of silent sample frame groups. This disclosure does not limit the specific mixing method.

[0183] Through hybrid training, the model can fully learn lip-shape generation capabilities under two states within the same training process. For inputs driven by spoken audio, the model learns to generate lip-shape change sequences that are precisely synchronized with speech; for inputs driven by silence, the model learns to generate stable, non-jerky, natural closed-lip shapes.

[0184] Furthermore, to further enhance the model's generalization performance, data augmentation can be performed on both types of samples during training. Data augmentation refers to randomly altering the visual attributes of samples before feeding them into the model to increase the diversity of the training data.

[0185] For example, data augmentation may include: brightness adjustment (randomly changing the brightness of the image), saturation adjustment (randomly changing the color saturation of the image), contrast adjustment (randomly changing the brightness contrast of the image), etc. Data augmentation can be performed on two types of samples separately, and the augmented samples constitute the final training batch and are fed into the model for training.

[0186] In some possible implementations, selecting multiple frames as training batches in a way that maximizes the inter-frame lip shape difference can be specifically achieved as follows: calculate the lip shape key point difference values ​​between candidate frames from non-silent video samples; and select a set of consecutive frames that maximizes the sum of the difference values ​​as the training batch.

[0187] In this embodiment of the disclosure, a candidate frame refers to a video frame selected from non-silent video samples as a training batch. In practice, multiple candidate frame groups can be generated from a continuous frame sequence of a non-silent video sample according to a certain step size or window sliding method. Each candidate frame group contains a preset number of continuous frames (such as 5 or 16 frames).

[0188] The lip keypoint difference value is a numerical indicator used to quantify the degree of difference in lip shape between two frames. An exemplary way to calculate the lip keypoint difference value is as follows: for each pair of frames in a candidate frame group, calculate the Euclidean distance between the corresponding lip keypoint coordinates, and use the sum of the distances between the corresponding keypoints of all pairs of frames as the lip keypoint difference value of the candidate frame group.

[0189] Specifically, suppose a candidate frame group contains K frames, each with L lip keypoints. For any two frames i and j, the difference in their lip keypoints can be expressed as: in, Let L be the coordinates of the l-th lip keypoint in the i-th frame. Let L be the coordinates of the l-th lip keypoint in the j-th frame. This represents the Euclidean distance between two points. The lip keypoint difference value for the entire candidate frame group can be expressed as the sum of the difference values ​​of all pairwise frame combinations: After calculating the sum of the differences in lip shape keypoints across all candidate frame groups, the candidate frame group with the largest sum of differences is selected as the training batch corresponding to the non-silent video sample. This selection method means that among all possible frame groups, the group with the most significant and richest lip shape changes between frames is selected as the training data.

[0190] The maximum sum of the difference values ​​indicates that the set of frames covers the range of most dramatic lip shape changes in non-silent video samples (such as changes from closed to open, or from one lip shape to another). Training with such a set of frames forces the model to learn to modify lip shapes based on audio features, rather than simply copying the lip shapes of the input frames, thus effectively solving the problem of lip leakage.

[0191] The differentiated sampling strategy and its specific implementation method provided in this disclosure effectively solve the "lip leakage" problem commonly found in related technologies by adopting a sampling strategy that maximizes inter-frame lip shape differences for non-silent video samples. During the training phase, the model is forced to learn to modify the lip shape in the reference image based on the input audio, establishing a true mapping relationship between audio features and lip shape changes. This allows the model to accurately generate matching lip shape sequences based on the input audio during the inference phase, improving the accuracy of audio-visual synchronization. By adopting a random sampling strategy for silent video samples, the problem of introducing spurious change signals due to forcibly maximizing lip shape differences is avoided. During the training phase, the model learns the correct characteristic of maintaining stable lip shapes in silent states. During the inference phase, it can generate continuous and stable closed lip shapes when faced with silent input, without producing abnormal lip tremors. By mixing training batches of the two types of samples, the model simultaneously learns the ability to generate lip shapes in both speaking and silent states within a unified training framework. Compared to training two independent models separately or switching between different models during inference, hybrid training enables a unified feature representation within the model, achieving a more natural transition between speaking and silence scenarios and avoiding lip-shape abrupt changes caused by model switching. By calculating the difference values ​​of lip-shape key points between candidate frames from non-silent video samples and selecting the set of consecutive frames that maximize the sum of the difference values ​​as the training batch, the principle of "maximizing difference" is quantified into a computable and verifiable mathematical operation.

[0192] The differential sampling strategy, along with the independent encoding of silent segments during the inference phase and the training data construction method, together constitute the complete technical loop of this application, from "data construction—model training—inference optimization." Each of the three stages has its own focus and works in concert: data construction provides high-quality training materials, differential sampling ensures the correctness of model training, and independent encoding and smoothing during the inference phase ensure stability in practical applications. Together, they enable the digital human to produce natural and stable closed-lip shaped output in silent scenarios of any duration.

[0193] Based on and Figure 1 The method shown follows the same principle. Figure 9 A schematic diagram of the structure of an audio processing apparatus provided in an embodiment of this disclosure is shown, such as... Figure 9 As shown, the audio processing device 90 may include: The audio separation module 910 is used to receive input audio, perform speech activity detection on the input audio, and separate at least one silent segment and at least one audible audio segment. The feature extraction module 920 is used to independently extract audio features from at least one silent segment and at least one audio segment with sound, so as to obtain the silence feature corresponding to each silent segment and the audio feature corresponding to each audio segment with sound. The feature splicing module 930 is used to merge the silence features and the sound audio features according to the time order of each segment in the input audio, and use a pre-trained lip-shape generation model to generate the lip-shape video corresponding to the input audio based on the merged features.

[0194] In the audio processing apparatus provided in this embodiment, by performing speech activity detection on the input audio, the silent segments and the audio segments with sound are extracted independently, which effectively prevents the silent features from being contaminated by the features of adjacent audio segments with sound. When the input audio contains silent segments, it can ensure that the generated lip-sync video maintains a natural and stable closed mouth state during the silent period, while maintaining accurate audio-visual synchronization during the sound period. This effectively solves the problems of lip trembling, abnormal mouth opening, and visual abrupt changes when switching between silence and speaking in the related technology, and significantly improves the naturalness and spatiotemporal coherence of digital human videos.

[0195] In some possible implementations, the feature extraction module is specifically used to: in response to the existence of a silent segment in the silent segment whose duration exceeds a preset threshold, divide the silent segment into multiple sub-segments whose duration is less than the preset threshold; extract audio features from each sub-segment to obtain the features corresponding to each sub-segment; perform weighted smoothing on the portion of the adjacent end region in the features corresponding to two temporally adjacent sub-segments; and concatenate the features corresponding to each smoothed sub-segment according to the temporal order of each sub-segment in the silent segment to obtain the silent feature corresponding to the silent segment.

[0196] In some possible implementations, the feature extraction module is specifically used to: take the splicing boundary position of the corresponding features of the two sub-segments as the center, take feature frames of a preset length before and after the splicing boundary position to form a local smoothing window as the end region; calculate reference feature values ​​based on the feature frames outside the local smoothing window, and construct interpolated features within the local smoothing window; construct a Gaussian weight distribution centered on the splicing boundary position; and, based on the Gaussian weight distribution, perform weighted mixing of the original features and interpolated features within the local smoothing window to obtain the smoothed features.

[0197] In some possible implementations, the Gaussian weight distribution has a set upper limit on the mixing ratio, such that the weight of the interpolation feature within a preset range centered on the splicing boundary position does not exceed the upper limit.

[0198] In some possible implementations, the feature extraction module is specifically used to: respond to the existence of an audio segment in the audio clip whose duration exceeds a preset threshold, divide the audio clip into multiple sub-segments whose duration is less than the preset threshold; extract audio features from each sub-segment, wherein, when extracting audio features from a sub-segment, the information of the sub-segments that are temporally adjacent to the sub-segment is used as context information, and the features corresponding to the sub-segment are extracted based on the information of the sub-segment and the context information; and concatenate the features corresponding to each sub-segment according to the temporal order of each sub-segment in the audio clip to obtain the audio features corresponding to the audio clip.

[0199] In some possible implementations, the audio processing device also includes a preprocessing module for replacing the time-domain audio signal value corresponding to the silence segment with a zero value.

[0200] It is understood that the above-described modules of the audio processing apparatus in the embodiments of this disclosure have the ability to implement... Figure 1 The embodiments shown illustrate the functions of corresponding steps in the audio processing method. These functions can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the functions described above. These modules can be software and / or hardware, and each module can be implemented individually or multiple modules can be integrated. For a detailed description of the functions of each module in the audio processing device, please refer to [link to relevant documentation]. Figure 1 The corresponding descriptions of the audio processing methods in the embodiments shown are not repeated here.

[0201] Based on and Figure 6 The method shown follows the same principle. Figure 10 This diagram illustrates the structure of a training device for a lip shape generation model according to an embodiment of the present disclosure. Figure 10 As shown, the training device 10 for the lip shape generation model may include: The video sample acquisition module 1010 is used to acquire silent video samples and non-silent video samples. The model training module 1020 is used to construct a training dataset based on the acquired silent video samples and non-silent video samples; and to train the lip-shape generation model based on the training dataset.

[0202] In the training apparatus for the lip shape generation model provided in this disclosure, the lack of high-quality silent, closed-mouth samples in the training data is fundamentally solved by specifically acquiring and introducing silent video samples. During the training phase, the model has the opportunity to fully learn the mapping relationship between silent input and closed-mouth lip shape, thus reliably generating a natural closed-mouth state when faced with silent input during the inference phase, rather than remaining in a half-open or trembling state. By constructing a training dataset together with silent and non-silent video samples, the distribution of the training data is made closer to real-world business scenarios. In actual digital human applications, silence and speaking are two alternating basic states. The mixed training dataset enables the model to simultaneously master lip shape generation capabilities in both states, providing a foundation for subsequent differentiated processing. By training the lip-shape generation model on a hybrid training dataset, the model establishes a correct response pattern to silent input during the training phase. This, combined with the independent encoding of silent segments during the inference phase, forms a closed-loop "training-inference" technology. The training phase lays the foundation for the model's ability to correctly process silent features, while the inference phase ensures that silent features are not contaminated through VAD separation and independent encoding. The two work together to achieve stable closed-lip shape output of the digital human in silent scenarios of any duration.

[0203] In some possible implementations, the video sample acquisition module is used to: acquire videos containing spoken and silent segments; perform face detection on the video to determine face regions, perform facial key point detection within the face regions, and acquire lip key points; calculate the mouth aspect ratio based on the lip key points, and identify frames with a mouth aspect ratio lower than a preset closure threshold as closed frames; filter out silent segments from the video that consist of consecutive closed frames and whose duration exceeds a preset duration; and generate silent video samples based on the silent segments.

[0204] In some possible implementations, the video sample acquisition module is specifically used to: replace the time-domain audio signal value corresponding to the silent segment with a zero value to generate a silent video sample.

[0205] In some possible implementations, the model training module is specifically used to: sample from the training dataset using a differentiated sampling strategy: for non-silent video samples, select multiple frames as training batches in a way that maximizes the inter-frame lip shape difference; for silent video samples, select multiple frames as training batches using a random sampling method; mix the training batches corresponding to non-silent video samples and the training batches corresponding to silent video samples, and feed them into the lip shape generation model for training.

[0206] In some possible implementations, the model training module is specifically used to: calculate the lip-shape keypoint difference values ​​between candidate frames from non-silent video samples; and select a set of consecutive frames that maximizes the sum of the difference values ​​as the training batch.

[0207] It is understood that the above-described modules of the training device for the lip shape generation model in the embodiments of this disclosure have the ability to implement... Figure 1 The embodiments shown illustrate the functionality of the corresponding steps in the training method for the lip shape generation model. This functionality can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described functions. These modules can be software and / or hardware, and each module can be implemented individually or integrated from multiple modules. For a detailed description of the functions of each module in the above-described lip shape generation model training device, please refer to [link to relevant documentation]. Figure 1 The corresponding description of the training method for the lip shape generation model in the illustrated embodiments will not be repeated here.

[0208] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and there is no violation of public order and good morals.

[0209] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0210] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0211] The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the audio processing method and / or the training method of the lip-shape generation model provided in the embodiments of this disclosure.

[0212] Compared with existing technologies, this electronic device detects speech activity in the input audio and extracts audio features from silent and spoken audio segments independently. This effectively prevents the silence features from being contaminated by the features of adjacent spoken audio segments. When the input audio contains silent segments, it ensures that the generated lip-sync video maintains a natural and stable closed mouth state during silence and maintains accurate audio-visual synchronization during spoken segments. This effectively solves the problems of lip trembling, abnormal mouth opening, and visual abrupt changes when switching between silence and speaking in related technologies, significantly improving the naturalness and spatiotemporal coherence of digital human videos.

[0213] The readable storage medium is a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform the audio processing method and / or the training method of the lip-shape generation model provided in the embodiments of this disclosure.

[0214] Compared with existing technologies, this readable storage medium detects speech activity in the input audio and extracts audio features from silent segments and spoken audio segments independently. This effectively prevents the silence features from being contaminated by the features of adjacent spoken audio segments. When the input audio contains silent segments, it ensures that the generated lip-sync video maintains a natural and stable closed mouth state during silence and maintains accurate audio-visual synchronization during spoken segments. This effectively solves the problems of lip trembling, abnormal mouth opening, and visual abrupt changes when switching between silence and speaking in related technologies, significantly improving the naturalness and spatiotemporal coherence of digital human videos.

[0215] The computer program product includes a computer program that, when executed by a processor, implements the audio processing method and / or the training method for the lip-shape generation model provided in the embodiments of this disclosure.

[0216] Compared with existing technologies, this computer program product detects speech activity in the input audio and extracts audio features from silent and spoken audio segments independently. This effectively prevents the silence features from being contaminated by the features of adjacent spoken audio segments. When the input audio contains silent segments, it ensures that the generated lip-sync video maintains a natural and stable closed mouth state during silence and maintains accurate audio-visual synchronization during spoken segments. This effectively solves the problems of lip trembling, abnormal mouth opening, and visual abrupt changes when switching between silence and speaking in related technologies, significantly improving the naturalness and spatiotemporal coherence of digital human videos.

[0217] Figure 11 A schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0218] like Figure 11As shown, device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1102 or a computer program loaded into random access memory (RAM) 1103 from storage unit 1108. The RAM 1103 may also store various programs and data required for the operation of device 1100. The computing unit 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. Input / output (I / O) interface 1105 is also connected to bus 1104.

[0219] Multiple components in device 1100 are connected to I / O interface 1105, including: input unit 1106, such as keyboard, mouse, etc.; output unit 1107, such as various types of monitors, speakers, etc.; storage unit 1108, such as disk, optical disk, etc.; and communication unit 1109, such as network card, modem, wireless transceiver, etc. Communication unit 1109 allows device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0220] The computing unit 1101 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as audio processing methods and / or lip-sync model training methods. For example, in some embodiments, the audio processing methods and / or lip-sync model training methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by the computing unit 1101, one or more steps of the audio processing methods and / or lip-sync model training methods described above can be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured by any other suitable means (e.g., by means of firmware) to perform audio processing methods and / or training methods for lip-shape generation models.

[0221] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0222] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0223] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0224] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0225] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0226] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0227] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0228] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An audio processing method, comprising: Receive input audio, perform speech activity detection on the input audio, and separate at least one silent segment and at least one spoken audio segment; Audio features are extracted independently from the at least one silent segment and the at least one audio segment with sound, to obtain the silence feature corresponding to each silent segment and the audio feature corresponding to each audio segment with sound. According to the time sequence of each segment in the input audio, the silence feature and the sound audio feature are merged, and a pre-trained lip-shape generation model is used to generate a lip-shape video corresponding to the input audio based on the merged features.

2. The method according to claim 1, wherein, The step of extracting audio features from the at least one silent segment to obtain the silence features corresponding to each silent segment includes: In response to the existence of a silent segment whose duration exceeds a preset threshold, the silent segment is divided into multiple sub-segments whose duration is less than the preset threshold; Audio features are extracted from each of the sub-segments to obtain the features corresponding to each sub-segment; Weighted smoothing is applied to the portion of the adjacent end regions in the features corresponding to two temporally adjacent sub-segments. According to the time sequence of each sub-segment in the silent segment, the features corresponding to each smoothed sub-segment are spliced ​​together to obtain the silent feature corresponding to the silent segment.

3. The method according to claim 2, wherein, The weighted smoothing process for the portion of the adjacent end regions in the corresponding features of two temporally adjacent sub-segments includes: Centered on the splicing boundary position of the corresponding features of the two sub-segments, feature frames of a preset length are taken before and after the splicing boundary position to form a local smooth window, which serves as the end region. Reference feature values ​​are calculated based on the feature frames outside the local smoothing window, and interpolation features are constructed within the local smoothing window. Construct a Gaussian weighted distribution centered at the splicing boundary position; Based on the Gaussian weight distribution, the original features within the local smoothing window are weighted and mixed with the interpolated features to obtain the smoothed features.

4. The method according to claim 3, wherein, The Gaussian weight distribution has a set upper limit for the mixing ratio, such that the weight of the interpolation feature within a preset range centered on the splicing boundary position does not exceed the upper limit.

5. The method according to claim 1, wherein, The step of extracting audio features from the at least one audio segment to obtain the audio features corresponding to each audio segment includes: In response to the existence of an audio segment in the audio segment whose duration exceeds a preset threshold, the audio segment is divided into multiple sub-segments whose duration is less than the preset threshold; Audio features are extracted from each of the sub-segments. When extracting audio features from a sub-segment, the information of the sub-segments that are temporally adjacent to the sub-segment is used as context information. Based on the information of the sub-segment and the context information, the features corresponding to the sub-segment are extracted. According to the time sequence of each sub-segment in the audio segment, the features corresponding to each sub-segment are spliced ​​together to obtain the audio features corresponding to the audio segment.

6. The method according to claim 1, wherein, Before extracting audio features from the at least one silent segment, the method further includes: Replace the time-domain audio signal value corresponding to the silent segment with a zero value.

7. A training method for a lip shape generation model, comprising: Obtain silent video samples; Obtain non-silent video samples; Based on the acquired silent video samples and non-silent video samples, a training dataset is constructed; The lip shape generation model is trained based on the training dataset.

8. The method according to claim 7, wherein, The acquisition of silent video samples includes: Obtain videos that include both spoken and silent segments; Face detection is performed on the video to determine the face region, and facial key point detection is performed within the face region to obtain lip key points; The aspect ratio of the mouth is calculated based on the key points of the lips, and frames with an aspect ratio of the mouth that is lower than a preset closure threshold are identified as closed frames. Filter out silent segments from the video that consist of consecutive closed frames and last for a preset duration; The silent video sample is generated based on the silent segment.

9. The method according to claim 8, wherein, The process of generating the silent video sample based on the silent segment includes: The time-domain audio signal value corresponding to the silent segment is replaced with a zero value to generate the silent video sample.

10. The training method according to claim 7, wherein, The training of the lip shape generation model based on the training dataset includes: Sample from the training dataset using a differentiated sampling strategy: For the non-silent video samples, multiple frames are selected as training batches in a manner that maximizes the inter-frame lip shape difference. For the silent video samples, multiple frames are selected as training batches using random sampling. The training batches corresponding to the non-silent video samples and the training batches corresponding to the silent video samples are mixed and fed into the lip-shape generation model for training.

11. The training method according to claim 10, wherein, The step of selecting multiple frames as training batches in a manner that maximizes the inter-frame lip shape difference includes: Calculate the lip shape key point difference values ​​between candidate frames from the non-silent video samples; Select the set of consecutive frames that maximizes the sum of the difference values ​​as the training batch.

12. An audio processing apparatus, comprising: An audio separation module is used to receive input audio, perform speech activity detection on the input audio, and separate at least one silent segment and at least one audible audio segment. The feature extraction module is used to independently extract audio features from the at least one silent segment and the at least one audio segment, to obtain the silence feature corresponding to each silent segment and the audio feature corresponding to each audio segment. The feature splicing module is used to merge the silence feature and the sound audio feature according to the time order of each segment in the input audio, and use a pre-trained lip shape generation model to generate the lip shape video corresponding to the input audio based on the merged features.

13. A training device for a lip shape generation model, comprising: The video sample acquisition module is used to acquire silent video samples; And to obtain non-silent video samples; The model training module is used to construct a training dataset based on the acquired silent video samples and non-silent video samples; and to train the lip-shape generation model based on the training dataset.

14. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6 and / or any one of claims 7-11.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6 and / or any one of claims 7-11.

16. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6 and / or any one of claims 7-11.

Images