Digital population voice synchronization generation method and system based on time sequence decoupling

Abstract

The embodiment of the application provides a kind of based on timing decoupling's digital population voice synchronous generation method and system, belong to digital human technical field;The method includes face detection and cutting to original video sequence, obtain standard face image sequence, the feature extraction of target audio signal is carried out, and deep audio feature sequence is obtained;Mask processing is carried out to each frame, and the image of mouth area to be driven is obtained;It is constructed as multi-channel input tensor;The mouth image that pre-trained mouth generation network outputs is synchronized with target audio signal;After replacing the mouth image to the corresponding position of original video sequence, it is combined with target audio signal, and the digital human video of mouth voice synchronization is output.The deep audio feature extraction and audio-video accurate alignment of the application improve the synchronization accuracy of mouth and voice, the timing dependence is modeled when multi-channel input, the transition of mouth sequence is smooth, and the adaptive feature fusion mechanism can automatically adapt to different phonemes and speech rate.

Description

Technical Field

[0001] This invention relates to the field of digital human technology, specifically to a method and system for synchronously generating digital human lip readings and speech based on temporal decoupling. Background Technology

[0002] With the rapid development of the metaverse, virtual reality, online education, and digital media industries, digital human technology, characterized by highly anthropomorphic and natural interactive capabilities, has become a hot topic in research and application. Among these, voice synchronization technology—the generation of digital human speaking videos that are precisely synchronized with a given audio signal and exhibit natural and realistic expressions—is one of the core elements for achieving high-quality digital human interaction. This technology can be widely applied in numerous scenarios such as virtual anchors, intelligent customer service, movie dubbing, remote conferencing, and sign language interpretation, possessing immense commercial value and social significance.

[0003] Traditional speech synchronization methods primarily rely on rule-based approaches or large-scale end-to-end generative models. Rule-based methods typically require a predefined phoneme-to-lip-shape mapping table, using phoneme sequences in audio to drive 3D face models or 2D image sequences to generate corresponding lip movements. While these methods offer high controllability, they require tedious manual annotation, and the generated lip movements are often too mechanical, lacking coherence and naturalness, making them difficult to handle continuous and varied natural language. Summary of the Invention

[0004] The purpose of this invention is to provide a digital human video with lip-synced speech generation based on temporal decoupling. This method and system are used to generate lip-synced speech by processing the face of the original video, extracting audio features, constructing multi-channel input, adaptively fusing and generating lip-synced speech through a pre-trained network, and optimizing boundary fusion. The final output is a digital human video with lip-synced speech that is precisely synchronized with the target audio.

[0005] To achieve the above objectives, this invention provides a method for synchronously generating digital lip-sync speech based on temporal decoupling, comprising: acquiring an original video sequence containing a target face and a target audio signal; performing face detection and cropping on the original video sequence to obtain a standard face image sequence; extracting features from the target audio signal to obtain a deep audio feature sequence; performing masking processing on each frame of the standard face image sequence to obtain a lip-sync region image to be driven for each frame; concatenating the lip-sync region image to be driven in the current frame with the lip-sync image generated in the previous frame along the channel dimension to construct a multi-channel input tensor; extracting local audio features aligned with the current frame from the deep audio feature sequence and inputting them and the multi-channel input tensor into a pre-trained lip-sync generation network; the pre-trained lip-sync generation network outputting a lip-sync image synchronized with the target audio signal through an adaptive feature fusion mechanism; performing boundary fusion processing on the lip-sync image; replacing the lip-sync image with the corresponding position in the original video sequence; combining it with the target audio signal to output a lip-sync synchronized digital human video.

[0006] Optionally, face detection and cropping are performed on the original video sequence to obtain a standard face image sequence, including: performing frame-by-frame face detection on the original video sequence using a pre-trained face detection model to obtain the face bounding box coordinates of each frame; performing temporal smoothing on the face bounding box coordinates of multiple consecutive frames; and performing expansion processing on the smoothed face bounding box coordinates.

[0007] Based on the expanded face bounding box coordinates, the face region image is cropped from the corresponding frame of the original video sequence; the sequence composed of multiple frames of the face region image is scaled to a uniform preset size and pixel value normalization is performed to obtain a standard face image sequence.

[0008] Optionally, the step of extracting features from the target audio signal to obtain a deep audio feature sequence includes: preprocessing the target audio signal; inputting the preprocessed target audio signal into a speech representation model pre-trained on a multilingual corpus, and extracting the output of the last hidden layer of the speech representation model as a deep speech vector; and performing tensor rearrangement on the deep speech vector to obtain a deep audio feature sequence.

[0009] Optionally, the masking process for each frame in the standard face image sequence includes: masking the face region image of each frame constituting the standard face image sequence, wherein the masking process sets the pixel values ​​within a set area of ​​the face region image to zero, while completely preserving the pixel values ​​outside the set area.

[0010] Optionally, the step of concatenating the lip-shape region image to be driven in the current frame with the lip-shape image generated in the previous frame along the channel dimension to construct a multi-channel input tensor includes: using the lip-shape region image to be driven in the current frame as a first input component; if there is a lip-shape image output by the pre-trained lip-shape generation network in the previous frame, then using it as a second input component; otherwise, using a preset value tensor as the second input component; and concatenating the first input component and the second input component along the channel dimension to form a multi-channel input tensor.

[0011] Optionally, the step of extracting local audio features aligned with the current frame from the deep audio feature sequence and inputting them into the pre-trained lip-sync generation network using the multi-channel input tensor includes: calculating the mapping relationship between audio feature time steps and video frames based on the sampling rate of the target audio signal and the frame rate of the original video sequence; extracting a local feature block containing multiple time steps from the deep audio feature sequence based on the mapping relationship; and using the local feature block aligned with the center of the current video frame to be processed in the time dimension as the local audio feature.

[0012] Optionally, the step of extracting local audio features aligned with the current frame from the deep audio feature sequence and inputting them into the pre-trained lip-sync generation network using the multi-channel input tensor further includes: inputting the local audio features and the multi-channel input tensor into the pre-trained lip-sync generation network; the pre-trained lip-sync generation network includes a face encoder, an audio encoder, and an adaptive decoder; the adaptive decoder receives the multi-scale visual features output by the face encoder and the global audio feature vector output by the audio encoder, and uses modulation parameters generated based on the global audio feature vector to perform layer-by-layer adaptive modulation on the multi-scale visual features to generate a lip-sync image.

[0013] Optionally, the adaptive decoder performs layer-by-layer adaptive modulation through an adaptive feature fusion module; the adaptive feature fusion module generates scaling and offset parameters using global audio feature vectors, and performs adaptive instance normalization on the multi-scale visual features input to the adaptive feature fusion module.

[0014] Optionally, the boundary fusion processing of the lip-sync image includes: generating a gradient mask of a preset width in the junction boundary region between the lip-sync image and the original video sequence; and performing weighted fusion of the pixels of the lip-sync image in the junction boundary region with the pixels of the corresponding frame of the original video sequence based on the gradient mask to achieve a visually smooth transition.

[0015] On the other hand, the present invention provides a digital lip-sync speech generation system based on temporal decoupling, for implementing a digital lip-sync speech generation method based on temporal decoupling. The system includes a control module, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement the digital lip-sync speech generation method based on temporal decoupling.

[0016] The above technical solution improves the synchronization accuracy of lip movements and speech through deep audio feature extraction and precise audio-video alignment. It can adapt to complex speech content, and through multi-channel input modeling of temporal dependencies, it makes the transition of lip movement sequences smooth, effectively avoiding jitter and abrupt changes. Through the adaptive feature fusion mechanism, it can automatically adapt to different phonemes and speech rates without the need for manual rule design. Boundary fusion processing eliminates seams in the generated area, while preserving high-frequency details of the face, thus improving visual realism.

[0017] Other features and advantages of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the following detailed description to explain the invention, but do not constitute a limitation thereof. In the drawings:

[0019] Figure 1 This is a flowchart of a digital lip-type speech synchronization generation method based on temporal decoupling.

[0020] Figure 2 This is a flowchart of the lip shape image generation process. Detailed Implementation

[0021] The following is in conjunction with the appendix Figure 1 -Appendix Figure 2 The specific implementation methods of the embodiments of the present invention will be described in detail below. It should be understood that the specific implementation methods described herein are only for illustrating and explaining the embodiments of the present invention, and are not intended to limit the embodiments of the present invention.

[0022] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application all comply with the relevant provisions of national laws and regulations. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0023] In the process of realizing this invention, the inventors of this application discovered that the prior art has defects such as insufficient expression of audio features, poor temporal consistency, and insufficient preservation of multi-scale details.

[0024] Example 1

[0025] Reference Figures 1-2 This is the first embodiment of the present invention, which provides a digital lip-sync speech generation method based on temporal decoupling, comprising:

[0026] S100: Acquire the original video sequence containing the target face and the target audio signal, perform face detection and cropping on the original video sequence to obtain a standard face image sequence, and extract features from the target audio signal to obtain a deep audio feature sequence.

[0027] In the embodiments of this application, a pre-trained face detection model is used to perform frame-by-frame face detection on the original video sequence to obtain the face bounding box coordinates of each frame; temporal smoothing is performed on the face bounding box coordinates of multiple consecutive frames; the smoothed face bounding box coordinates are expanded; based on the expanded face bounding box coordinates, face region images are cropped from the corresponding frames of the original video sequence; the sequence composed of multiple frames of face region images is scaled to a uniform preset size and pixel value normalization is performed to obtain a standard face image sequence.

[0028] In the embodiments of this application, the target audio signal is preprocessed; the preprocessed target audio signal is input into a speech representation model pre-trained on a multilingual corpus, and the output of the last hidden layer of the speech representation model is extracted as a deep speech vector; the deep speech vector is tensor rearranged to obtain a deep audio feature sequence.

[0029] In a preferred embodiment of this application, an original video sequence (format limited to MP4 or AVI) containing the target face and the corresponding target audio signal are acquired. The original video sequence must meet the following requirements: a continuous frontal speech scene with no side profiles, obstructions, or significant posture changes; speech pauses of ≤2 seconds; and a real-world or green background. The target audio signal can be provided independently or embedded in the original video sequence. The acquired raw data undergoes parameter verification, including video parameter verification (resolution ≥1080P, frame rate ≥25fps, duration ≥3 minutes). If audio parameters are provided independently, they must be compatible with 16000Hz sampling rate conversion and free from significant noise or distortion interference. Raw data that does not meet the above requirements after verification must be preprocessed and adjusted before proceeding to subsequent processes to ensure the accuracy of subsequent feature extraction and model inference.

[0030] The original video sequence was standardized using FFmpeg with the following standard parameters: the resolution was uniformly converted to 2160×3840 (4K portrait) or 1920×1080 (landscape) to ensure consistent image pixel density; the frame rate was forcibly adjusted to 25fps to lay the foundation for accurate alignment of subsequent audio and video frames; if the target audio signal was embedded in the original video sequence, the audio track was extracted using FFmpeg and temporarily saved as an audio file to be processed. The initial parameters of this audio file were set to a 16000Hz sampling rate, mono, and 16-bit signed integer to avoid loss of audio information.

[0031] Furthermore, a custom pre-trained face detection model (model file identified as findFaceV2.h5) trained based on the Keras framework is loaded. This model has high-precision face localization capabilities and can adapt to face detection needs under different lighting and background scenarios. Batch processing mode (processing 16 video frames per batch) is adopted to improve detection efficiency. The video frames are uniformly adjusted to 384×384 pixels to match the input size requirements of the findFaceV2.h5 model. The pixel values ​​of the video frames are mapped from [0,255] to the [-1,1] range to eliminate the interference of inter-frame brightness differences on detection accuracy. The anchor-based bounding boxes output by the findFaceV2.h5 model are converted to absolute coordinates in the original video frame coordinate system as (x1, y1, x2, y2), where (x1, y1) are the coordinates of the top-left corner of the bounding box and (x2, y2) are the coordinates of the bottom-right corner of the bounding box. Non-maximum suppression (NMS) is performed on the bounding boxes detected in each frame, and the IOU (Intersection over Union) threshold is set to 0.3 to filter the single face bounding box with the highest confidence and exclude false detections of non-face regions.

[0032] Furthermore, the face bounding box coordinates of five consecutive frames are smoothed using a sliding window averaging algorithm to eliminate detection jitter and obtain a stable face trajectory. To fully preserve the lip shape and chin area, an expansion operation is performed on the smoothed bounding box: the lower boundary is expanded by 20px along the positive y-axis, and the left and right boundaries are expanded by 15px each along the negative and positive x-axis, respectively, and the expanded bounding box does not exceed the original video frame size range. Based on the expanded bounding box coordinates, face region images are cropped from the corresponding frames of the original video sequence. All cropped face region images are adjusted to a preset size of 256×256 pixels, and the pixel values ​​of the adjusted face region images are normalized to the [0,1] interval. According to the temporal order of the original video frames, the normalized face region images are used to construct a standard face image sequence.

[0033] Furthermore, if the target audio signal is embedded in the original video sequence, the audio track is extracted first, and then the target audio signal is uniformly converted to WAV format, with parameters set to a 16000Hz sampling rate, mono, and 16-bit signed integer. A speech representation model (model identifier: TencentGameMate / chinese-hubert-large) pre-trained on a multilingual corpus and its accompanying Wav2Vec2 feature extractor are loaded. The Wav2Vec2 feature extractor is used to convert the preprocessed target audio signal into a model-compatible input format. The converted input data is then input into the speech representation model, and the output of the last hidden layer of the model is extracted as a deep speech vector. The tensor shape of the deep speech vector is [1, time_steps, 1024], with a time resolution of 50Hz (i.e., 50 feature vectors are generated per second). Tensor rearrangement is performed on the deep speech vector, adjusting its tensor shape from [1, time_steps, 1024] to [1, 1024, time_steps], and then integrating them into a deep audio feature sequence according to the time step order.

[0034] The above scheme establishes high-quality feature representations with strict temporal correspondences for both face sequences and audio signals. This includes not only ensuring the stability and integrity of face region cropping through temporal smoothing and adaptive expansion, but more importantly, establishing a frame-accurate audio-video temporal alignment mapping in advance by using the integer multiple relationship between the video frame rate and the deep audio feature sampling rate, laying the foundation for subsequent frame-by-frame precise driving.

[0035] S200: Perform masking processing on each frame in the standard face image sequence to obtain the lip shape region image to be driven for each frame.

[0036] In the embodiments of this application, each frame of the face region image constituting the standard face image sequence is subjected to masking processing. The masking processing involves setting the pixel values ​​within a set area of ​​the face region image to zero, while completely preserving the pixel values ​​outside the set area.

[0037] In a preferred embodiment of this application, the lower half of each frame of a standard face image is designated as the masking target area, corresponding to an image row index range of 128-255 (since the image height is 256 pixels, the area below row 128 covers the mouth shape and chin, which are the mouth shape-related areas to be driven). The upper middle part of each frame of a standard face image is designated as the reserved area, corresponding to an image row index range of 0-127 (covering static face areas such as the forehead, eyebrows, and eyes that do not need to be driven). Following the temporal order of the standard face image sequence, single-frame standard face images are read frame by frame. The pixel values ​​of the masked area (row index 128-255, column index 0-255) of that frame are set to zero to eliminate the original mouth shape features in that area. The pixel values ​​of the reserved area (row index 0-127) remain unchanged, maintaining the original static features of the upper middle part of the face. Each frame of the masked image is marked as the "mouth shape region image to be driven," and arranged according to the temporal order of the original standard face image sequence to obtain the mouth shape region image sequence to be driven.

[0038] The above scheme, based on precise region division of fixed size, only masks the lower half of the region where the lip movements are located (below line 128), while fully preserving the upper and middle static regions. This clearly defines the model's learning objective, avoids interference from original lip movement features, and overcomes the inefficiency of traditional full-image or blurred masking. Through simple masking operations, complex tasks are decoupled. The tasks of "identity preservation" and "lip movement driving" are separated spatially. By preserving the upper half of the face's identity information and driving only the lower half's lip movement region, the model is forced to focus on learning the audio-lip movement mapping relationship. This fundamentally avoids modification of the person's identity features during the generation process, ensuring a high degree of consistency in identity within the resulting video.

[0039] For each frame in the image sequence of the lip shape region to be driven (from frame 1 to frame N in chronological order), perform the following iterative steps:

[0040] S300: The lip shape region image to be driven in the current frame is concatenated with the lip shape image generated in the previous frame along the channel dimension to construct a multi-channel input tensor.

[0041] In the embodiments of this application, the lip shape region image to be driven in the current frame is used as the first input component; if there is a lip shape image output by the pre-trained lip shape generation network in the previous frame, it is used as the second input component; otherwise, a preset value tensor is used as the second input component; the first input component and the second input component are concatenated along the channel dimension to form a multi-channel input tensor.

[0042] In a preferred embodiment of this application, the current frame's lip shape region image from the sequence of images to be driven is taken as the first input component of the multi-channel input tensor. This component has 3 channels (corresponding to RGB channels) and a size of 256×256 pixels. If there is a lip shape image output by the pre-trained lip shape generation network in the previous frame, this lip shape image is directly used as the second input component (3 channels, 256×256 pixels). If there is no lip shape image generated in the previous frame (such as the first frame of the processing sequence), a tensor of all 1s is used as the second input component. The tensor dimension is the same as that of the lip shape image generated in the previous frame (3 channels, 256×256 pixels, all pixel values ​​are 1). The first input component and the second input component are concatenated along the channel dimension to form a single-frame multi-channel input tensor. The concatenated tensor dimension is [6, 256, 256] (6 channels, corresponding to 3 channels of the current frame + 3 channels of the previous frame / preset tensor), and the pixel value range is maintained at [0, 1], adapting to the input format of the subsequent lip shape generation network. Following the temporal order of the image sequence of the lip shape region to be driven, frame by frame, a multi-channel input tensor sequence with the same number of frames as the image sequence of the lip shape region to be driven is constructed.

[0043] The above scheme uses the generation result of the previous frame as the explicit input of the current frame. This autoregressive design injects dynamic memory into the static generation model, ensuring that the generation of each frame is based on the state of the previous frame. This transforms independent frame-level predictions into coherent sequence generation, effectively guaranteeing the smoothness and naturalness of lip movements. Furthermore, the integrated input design of temporal information and visual features eliminates the need for an additional temporal modeling module, simplifying the process while improving the coherence of the lip movement sequence.

[0044] S400: Extract local audio features aligned with the current frame from the deep audio feature sequence and input them into the pre-trained lip-sync generation network along with the multi-channel input tensor. The pre-trained lip-sync generation network outputs a lip-sync image synchronized with the target audio signal through an adaptive feature fusion mechanism.

[0045] In the embodiments of this application, the mapping relationship between audio feature time steps and video frames is calculated based on the sampling rate of the target audio signal and the frame rate of the original video sequence; based on the mapping relationship, a local feature block containing multiple time steps is extracted from the deep audio feature sequence; the local feature block aligned with the center of the current video frame to be processed in the time dimension is taken as the local audio feature.

[0046] In the embodiments of this application, local audio features and multi-channel input tensors are input into a pre-trained lip-shape generation network; the pre-trained lip-shape generation network includes a face encoder, an audio encoder, and an adaptive decoder; the adaptive decoder receives multi-scale visual features output by the face encoder and a global audio feature vector output by the audio encoder, and uses modulation parameters generated based on the global audio feature vector to perform layer-by-layer adaptive modulation on the multi-scale visual features to generate a lip-shape image.

[0047] It should be noted that the adaptive decoder performs layer-by-layer adaptive modulation through the adaptive feature fusion module; the adaptive feature fusion module uses the global audio feature vector to generate scaling parameters and offset parameters, and performs adaptive instance normalization on the multi-scale visual features input to the adaptive feature fusion module.

[0048] In a preferred embodiment of this application, an alignment factor is determined based on the temporal resolution (50Hz) of the deep audio feature sequence and the frame rate (25fps) of the original video sequence. The alignment factor = 50.0 / 25.0 = 2, meaning that each video frame corresponds to 2 audio time steps. Ultimately, 10 consecutive audio time steps (covering the temporal information before and after the frame) need to be extracted from each frame. For the i-th frame to be processed, the starting index of the audio features is calculated. A feature block containing 10 consecutive time steps, starting from this index, is extracted from the deep audio feature sequence to form a local audio feature with a tensor of [1, 1024, 10]. The tensor dimension of the local audio feature is converted to [1, 1, 1024, 10] to adapt to the input format of the subsequent audio encoder.

[0049] Furthermore, the multi-channel input tensor sequence is input into the face encoder, and multi-scale visual features (including feature maps of different resolutions from 256×256 to 2×2) are extracted through 8 layers of downsampling operations. The features of each layer are retained for subsequent skip connections. The adjusted local audio features are input into the audio encoder, and the features are progressively compressed through 7 layers of convolution + residual modules, finally outputting a 1024-dimensional global audio feature vector. The adaptive decoder receives the multi-scale visual features and the global audio feature vector, and embeds AdaI in each layer through an 8-layer upsampling + skip connection structure. The N-adaptive feature fusion module includes: performing instance normalization on the input multi-scale visual features to obtain normalized features; generating scaling and offset parameters from the global audio feature vector through 1×1 convolution; performing adaptive modulation through a preset formula, and outputting the modulation features after activation by ReLU6 (a linear rectified function with an upper limit of 6); repeating the above modulation process layer by layer, and finally outputting a lip-sync image synchronized with the target audio (tensor shape [1,3,256,256], pixel value range [0,1]) through a two-dimensional convolutional layer (32→3) and Sigmoid activation. The generated lip-sync image is used as the output result of the current frame, and the "previous frame lip-sync image cache" is updated simultaneously. The current frame lip-sync image is stored in the cache to provide a second input component for the construction of multi-channel input tensors in subsequent frames (i+1 frames).

[0050] The above scheme introduces an adaptive fusion mechanism to dynamically generate scaling parameters (gamma) and offset parameters (beta) for audio features, modulating visual features layer by layer. It automatically matches lip-sync patterns corresponding to different phonemes and speech rates without requiring manual rule design. By extracting local features from 10 consecutive audio time steps per frame, it accurately covers intra-frame and inter-frame audio information, achieving a lip-sync synchronization error of ≤±20ms, breaking through the synchronization accuracy limitations of traditional single-time-step features.

[0051] The adaptive decoder uses global feature vectors extracted from audio to generate dynamic parameters to modulate the normalized statistics of visual features. This fine-grained, layer-by-layer feature modulation method can more accurately and flexibly modulate the temporal variation information of audio into visual features, thereby generating realistic lip movements that are highly synchronized with complex speech.

[0052] S500: Perform boundary fusion processing on the lip-sync image, replace the lip-sync image with the corresponding position in the original video sequence, and combine it with the target audio signal to output a lip-sync synchronized digital human video.

[0053] In the embodiments of this application, a gradient mask of a preset width is generated at the junction boundary region between the lip-sync image and the original video sequence; based on the gradient mask, the pixels of the lip-sync image in the junction boundary region are weighted and fused with the pixels of the corresponding frames of the original video sequence to achieve a visually smooth transition.

[0054] In a preferred embodiment of this application, the lip-shape image is converted into a NumPy array and denormalized (pixel values ​​are mapped from [0,1] to [0,255]). Then, based on the expanded face bounding box coordinates (x1,y1,x2,y2), the lip-shape image is adjusted to the original face region size (i.e., the width of x2-x1 and the height of y2-y1) to ensure matching with the face region size of the original video frame. The fusion boundary width is set to 20 pixels (to meet visual smoothing requirements and avoid seams). Fusion is only performed on the junction boundary region between the lip-shape image and the original video frame; non-boundary regions directly use the lip-shape image pixels. The bottom 20 rows of pixels of the lip-shape image (corresponding to the boundary near the chin of the original face region) are located, and linear gradient coefficients are generated. (Range 0-1, from the inside to the outside of the lip shape image) (incrementing from 0 to 1), according to the calculation formula: merged pixel = × lip shape image pixels + (1- The pixels corresponding to the original video frame are multiplied by 1, and a weighted fusion is performed on the bottom 20 rows of pixels. The left 20 columns of pixels in the lip-sync image (corresponding to the left cheek boundary of the original face region) are located, and linear gradient coefficients are generated. (Range 0-1, from the inside to the outside of the lip shape image) Decrease from 1 to 0), perform weighted fusion using the same formula as above; locate the right 20 columns of pixels in the lip shape image (corresponding to the right cheek boundary of the original face region), and generate linear gradient coefficients. (Range 0-1, from the inside to the outside of the lip shape image) (Incrementing from 0 to 1); perform weighted fusion using the same formula as above.

[0055] Furthermore, the background area of ​​the original video frame is detected by calculating the difference between the green channel value and the blue channel value of the pixels. If the difference is greater than 60, it is determined to be a green screen background area, and a binary mask is generated (the background area is marked as 1, and the face area is marked as 0). According to the formula: final pixel = lip image pixels × (1 - mask) + original video frame pixels × mask, the original pixels of the green screen background area are retained, and only the lip image pixels of the face foreground area are replaced.

[0056] Furthermore, based on the expanded face bounding box coordinates (x1, y1, x2, y2), the lip-sync image with completed boundary fusion is precisely pasted back to the starting position (x1, y1) of the corresponding frame in the original video sequence, covering the lower half of the face (lip-sync region) of the original frame, generating a "lip-sync updated video frame." This replacement operation is repeated frame by frame in the temporal order of the original video sequence to obtain a "lip-sync synchronized video sequence." Using FFmpeg, the "lip-sync synchronized video sequence" is synthesized with the "target audio signal." Both video and audio encoding remain consistent with the original video / audio (the parameter is set to "copy" to avoid quality loss due to re-encoding). The synthesized video is saved in a preset format (e.g., AVI) with the storage path set to . / result / res.avi. Temporary files generated during the process (such as intermediate frames and feature files) are cleaned up, and the final output is a lip-sync synchronized digital human video.

[0057] The above solution effectively eliminates harsh seams and color differences by designing directional gradient weights for pixel-level fusion at the bottom and side junctions of the lip-sync image and the original face region. This targeted fusion strategy ensures that the generated lip-sync region blends naturally into the original video background, a key step in achieving a high-quality final visual experience. Employing a 20-pixel gradient weighted fusion technique, differentiated gradient coefficients are designed for the bottom, left, and right boundaries to completely eliminate the seams between the lip-sync image and the original video, enhancing visual naturalness. A green screen background detection mechanism is introduced to accurately distinguish between the background and the foreground face, avoiding accidental background modifications during replacement. This adapts to various scenarios, including real-world scenes and green screen applications, overcoming the limitations of traditional single-scene fusion.

[0058] The present invention also provides a digital lip-sync speech generation system based on temporal decoupling, for implementing a digital lip-sync speech generation method based on temporal decoupling. The system includes a control module, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement the digital lip-sync speech generation method based on temporal decoupling.

[0059] This invention provides a storage medium storing a program that, when executed by a processor, implements the digital lip-sync speech generation method based on time-decoupling.

[0060] This invention provides a processor for running a program, wherein the program executes the digital lip-sync speech generation method based on temporal decoupling during runtime.

[0061] This invention provides a device including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements a digital lip-sync speech generation method based on time-decoupled processing. The device described herein can be a server, PC, tablet, mobile phone, etc.

[0062] This application also provides a computer program product that, when executed on a data processing device, is suitable for performing a time-decoupled digital lip-type speech synchronization generation method.

[0063] Those skilled in the art will understand that embodiments of this application can provide methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0064] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0065] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0066] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0067] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0068] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0069] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0070] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0071] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for synchronously generating digital lip-sync speech based on temporal decoupling, characterized in that, include: The original video sequence containing the target face and the target audio signal are acquired. Face detection and cropping are performed on the original video sequence to obtain a standard face image sequence. Feature extraction is performed on the target audio signal to obtain a deep audio feature sequence. Each frame in the standard face image sequence is masked to obtain the lip shape region image to be driven for each frame. The lip-sync region image to be driven in the current frame is concatenated with the lip-sync image generated in the previous frame along the channel dimension to construct a multi-channel input tensor, including: Use the image of the lip-sync region to be driven in the current frame as the first input component; If there exists a lip shape image output by a pre-trained lip shape generation network in the previous frame, then use it as the second input component; otherwise, use a preset value tensor as the second input component. The first input component and the second input component are concatenated along the channel dimension to form a multi-channel input tensor; Local audio features aligned with the current frame are extracted from the deep audio feature sequence and input to a pre-trained lip-sync generation network along with the multi-channel input tensor. The pre-trained lip-sync generation network outputs a lip-sync image synchronized with the target audio signal through an adaptive feature fusion mechanism. The lip-sync image is subjected to boundary fusion processing. After the lip-sync image is replaced with the corresponding position in the original video sequence, it is combined with the target audio signal to output a lip-sync synchronized digital human video.

2. The digital lip-sync speech generation method based on temporal decoupling according to claim 1, characterized in that, The original video sequence is subjected to face detection and cropping to obtain a standard face image sequence, including: The original video sequence is subjected to frame-by-frame face detection using a pre-trained face detection model to obtain the face bounding box coordinates for each frame. Temporal smoothing is performed on the face bounding box coordinates of multiple consecutive frames; Expand the coordinates of the smoothed face bounding box; Based on the expanded face bounding box coordinates, the face region image is cropped from the corresponding frame of the original video sequence; The sequence of multiple frames of the face region images is scaled to a uniform preset size and then normalized to obtain a standard face image sequence.

3. The digital lip-sync speech generation method based on temporal decoupling according to claim 1, characterized in that, The step of extracting features from the target audio signal to obtain a deep audio feature sequence includes: Preprocess the target audio signal; The preprocessed target audio signal is input into a speech representation model pre-trained on a multilingual corpus, and the output of the last hidden layer of the speech representation model is extracted as a deep speech vector. Tensor rearrangement is performed on the deep speech vectors to obtain the deep audio feature sequence.

4. The digital lip-sync speech generation method based on temporal decoupling according to claim 1, characterized in that, The masking process for each frame in the standard face image sequence includes: masking each frame of the face region image constituting the standard face image sequence, wherein the masking process sets the pixel values ​​within a set area of ​​the face region image to zero, while completely preserving the pixel values ​​outside the set area.

5. The digital lip-sync speech generation method based on temporal decoupling according to claim 1, characterized in that, The step of extracting local audio features aligned with the current frame from the deep audio feature sequence and inputting them into the pre-trained lip-sync generation network using the multi-channel input tensor includes: Based on the sampling rate of the target audio signal and the frame rate of the original video sequence, calculate the mapping relationship between the audio feature time steps and the video frames; Based on the mapping relationship, a local feature block containing multiple time steps is extracted from the deep audio feature sequence; The local feature block aligned with the center of the current video frame to be processed in the time dimension is used as the local audio feature.

6. The digital lip-sync speech generation method based on temporal decoupling according to claim 1, characterized in that, The step of extracting local audio features aligned with the current frame from the deep audio feature sequence and inputting them into the lip-sync generation network pre-trained with the multi-channel input tensor further includes: Local audio features are input into a pre-trained lip-sync generation network along with a multi-channel input tensor; The pre-trained lip-sync generation network includes a face encoder, an audio encoder, and an adaptive decoder; The adaptive decoder receives the multi-scale visual features output by the face encoder and the global audio feature vector output by the audio encoder, and uses the modulation parameters generated based on the global audio feature vector to perform layer-by-layer adaptive modulation on the multi-scale visual features to generate a lip-sync image.

7. The digital lip-sync speech generation method based on temporal decoupling according to claim 6, characterized in that, The adaptive decoder performs layer-by-layer adaptive modulation through the adaptive feature fusion module; the adaptive feature fusion module uses the global audio feature vector to generate scaling parameters and offset parameters, and performs adaptive instance normalization on the multi-scale visual features input to the adaptive feature fusion module.

8. The digital lip-sync speech generation method based on temporal decoupling according to claim 1, characterized in that, The boundary fusion processing of the lip-sync image includes: generating a gradient mask of a preset width in the junction boundary region between the lip-sync image and the original video sequence; and performing weighted fusion of the pixels of the lip-sync image in the junction boundary region with the pixels of the corresponding frame of the original video sequence based on the gradient mask to achieve a visually smooth transition.

9. A digital lip-sync speech generation system based on temporal decoupling, characterized in that, The system includes a control module, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement the digital lip-sync speech generation method based on temporal decoupling according to any one of claims 1-8.

Images