Digital population type voice synchronous generation method and system based on time sequence decoupling

Abstract

The embodiment of the invention provides a digital population type voice synchronous generation method and system based on time sequence decoupling, and belongs to the technical field of digital people. The method comprises the following steps: performing face detection and cutting on an original video sequence to obtain a standard face image sequence, and performing feature extraction on a target audio signal to obtain a deep audio feature sequence; performing mask processing on each frame to obtain a to-be-driven mouth shape area image; constructing a multi-channel input tensor; the pre-trained mouth shape generation network outputs a mouth shape image synchronized with the target audio signal; and after replacing the mouth shape image to a corresponding position of the original video sequence, combining the mouth shape image with the target audio signal, and outputting a digital human video with synchronous mouth shape voice. According to the invention, through deep audio feature extraction and audio and video accurate alignment, the synchronization accuracy of the mouth shape and the voice is improved, through multi-channel input modeling time sequence dependence, mouth shape sequence transition is smooth, and through an adaptive feature fusion mechanism, different phonemes and different speech speeds can be automatically adapted.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital human, in particular to a digital human speech synchronization generation method and system based on time sequence decoupling. BACKGROUND

[0002] With the rapid development of meta-universe, virtual reality, online education and digital media industry, digital human technology with high personification and natural interaction ability has become a research and application hotspot. Among them, the generation of digital human speaking video with accurate synchronization to the given audio signal and natural and realistic expression, i.e. speech synchronization technology, is one of the core links to realize high-quality digital human interaction. This technology can be widely used in virtual anchors, intelligent customer service, film dubbing, remote meetings and sign language translation and many other scenes, and has great commercial value and social significance.

[0003] Traditional speech synchronization methods mainly rely on rule-based methods or large-scale end-to-end generation models. Rule-based methods usually need to define a phoneme-to-mouth shape mapping table in advance, and generate the corresponding mouth shape by recognizing the phoneme sequence in the audio to drive the three-dimensional face model or two-dimensional image sequence. Although this kind of method has high controllability, it needs tedious manual annotation, and the generated mouth movement is often too mechanical, lacking coherence and naturalness, and it is difficult to handle continuous and variable natural language. SUMMARY

[0004] The purpose of the present application is to provide a digital human speech synchronization generation method and system based on time sequence decoupling, which can generate a digital human video with accurate synchronization of mouth shape and target audio by face processing of the original video, audio feature extraction, multi-channel input construction, pre-trained network adaptive fusion to generate mouth shape and boundary fusion optimization.

[0005] In order to achieve the above object, the application provides a digital lip-synch generation method based on time sequence decoupling, comprising: collecting an original video sequence containing a target face and a target audio signal, performing face detection and cropping on the original video sequence to obtain a standard face image sequence, and performing feature extraction on the target audio signal to obtain a deep audio feature sequence; performing mask processing on each frame in the standard face image sequence to obtain a corresponding to-be-driven lip region image of each frame; splicing the to-be-driven lip region image of the current frame and the lip image generated by the previous frame along the channel dimension to construct a multi-channel input tensor; extracting a local audio feature aligned with the current frame from the deep audio feature sequence, and inputting the local audio feature and the multi-channel input tensor into a pre-trained lip generation network, wherein the pre-trained lip generation network outputs a lip image synchronized with the target audio signal through an adaptive feature fusion mechanism; performing boundary fusion processing on the lip image, replacing the lip image to a corresponding position of the original video sequence, and combining the lip image with the target audio signal to output a digital human video with lip-synch.

[0006] Optionally, the face detection and cropping on the original video sequence to obtain a standard face image sequence comprises: performing frame-by-frame face detection on the original video sequence using a pre-trained face detection model to obtain face bounding box coordinates of each frame; performing time sequence smoothing processing on the face bounding box coordinates of consecutive multiple frames; performing extension processing on the face bounding box coordinates after smoothing processing; cropping a face region image from a corresponding frame of the original video sequence according to the face bounding box coordinates after extension; scaling a sequence composed of multiple face region images to a unified preset size and performing pixel value normalization processing to obtain a standard face image sequence.

[0007] Optionally, the feature extraction on the target audio signal to obtain a deep audio feature sequence comprises: pre-processing the target audio signal; inputting the pre-processed target audio signal into a speech representation model pre-trained on multi-language corpus, and extracting an output of a last hidden layer of the speech representation model as a deep speech vector; performing tensor rearrangement on the deep speech vector to obtain a deep audio feature sequence.

[0008] Optionally, the mask processing on each frame in the standard face image sequence comprises: performing mask processing on each face region image constituting the standard face image sequence, wherein the mask processing sets pixel values in a designated region of the face region image to zero while completely retaining pixel values outside the designated region.

[0009] Optionally, the step of splicing the to-be-driven mouth region image of the current frame and the mouth image generated from the previous frame along the channel dimension to construct a multi-channel input tensor comprises: taking the to-be-driven mouth region image of the current frame as a first input component; if there is a mouth image output by the pre-trained mouth generation network from the previous frame, taking the mouth image as a second input component; otherwise, using a preset value tensor as the second input component; and splicing the first input component and the second input component along the channel dimension to form a multi-channel input tensor.

[0010] Optionally, the step of extracting a local audio feature aligned with the current frame from the deep audio feature sequence and inputting the local audio feature and the multi-channel input tensor into the pre-trained mouth generation network comprises: calculating a mapping relationship between an audio feature time step and a video frame according to a sampling rate of the target audio signal and a frame rate of the original video sequence; extracting a local feature block containing multiple time steps from the deep audio feature sequence according to the mapping relationship; and taking the local feature block aligned with the center of the current video frame to be processed in the time dimension as the local audio feature.

[0011] Optionally, the step of extracting a local audio feature aligned with the current frame from the deep audio feature sequence and inputting the local audio feature and the multi-channel input tensor into the pre-trained mouth generation network further comprises: inputting the local audio feature and the multi-channel input tensor into the pre-trained mouth generation network; the pre-trained mouth generation network comprises a face encoder, an audio encoder, and an adaptive decoder; the adaptive decoder receives a multi-scale visual feature output by the face encoder and a global audio feature vector output by the audio encoder, and performs layer-by-layer adaptive modulation on the multi-scale visual feature by using modulation parameters generated based on the global audio feature vector to generate a mouth image.

[0012] Optionally, the adaptive decoder performs layer-by-layer adaptive modulation through an adaptive feature fusion module; the adaptive feature fusion module generates scaling parameters and offset parameters by using the global audio feature vector, and performs adaptive instance normalization on the multi-scale visual feature input into the adaptive feature fusion module.

[0013] Optionally, the boundary fusion processing of the mouth image comprises: generating a preset width of a gradient mask in a joint boundary region of the mouth image and the original video sequence; and based on the gradient mask, performing weighted fusion on pixels of the mouth image in the joint boundary region and pixels of a corresponding frame of the original video sequence to achieve a smooth transition in vision.

[0014] In another aspect, the present application provides a digital lip-synch generation system based on timing decoupling, for implementing a digital lip-synch generation method based on timing decoupling, comprising a control module, the control module comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the computer program to implement the digital lip-synch generation method based on timing decoupling.

[0015] The above technical solution improves the synchronization accuracy of lip shapes and voices by deep audio feature extraction and accurate audio-video alignment, can adapt to complex voice content, and through multi-channel input modeling timing dependence, makes the transition of the lip shape sequence smooth, effectively avoids jitter and mutation, through an adaptive feature fusion mechanism, can automatically adapt to different phonemes and speech rates, without the need for manual design rules, boundary fusion processing eliminates the joint of the generated area, while retaining high-frequency details of the human face, and improves visual realism.

[0016] Other features and advantages of the present application will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings are included to provide a further understanding of the present application, and constitute a part of the specification, and are used together with the following specific embodiments to explain the present application, but do not constitute a limitation on the present application. In the drawings: Figure 1 is a flowchart of a digital lip-synch generation method based on timing decoupling.

[0018] Figure 2 is a flowchart of lip shape image generation. DETAILED DESCRIPTION

[0019] The following will be described in detail in conjunction with the accompanying Figure 1 - the accompanying Figure 2 The specific embodiments of the embodiments of the present application are described in detail. It should be understood that the specific embodiments described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application.

[0020] It should be noted that the acquisition, transmission, storage, use, processing, etc. of data in the technical solution of the present application comply with the relevant provisions of national laws and regulations. In the embodiments of the present application, some existing industry solutions may be mentioned, such as software, components, models, etc., which should be considered as exemplary, and the purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0021] The present inventors found in the process of implementing the present application that the prior art has the defects of insufficient audio feature expression, poor timing consistency, and insufficient multi-scale detail retention.

[0022] Embodiment 1 Reference Figures 1-2 For a first embodiment of the present application, the embodiment provides a digital population voice synchronization generation method based on timing decoupling, comprising: S100: Collecting an original video sequence containing a target face and a target audio signal, performing face detection and cropping on the original video sequence to obtain a standard face image sequence, and performing feature extraction on the target audio signal to obtain a deep audio feature sequence.

[0023] In the embodiment of the present application, a pre-trained face detection model is used to perform frame-by-frame face detection on the original video sequence to obtain face bounding box coordinates of each frame; the face bounding box coordinates of consecutive multiple frames are subjected to timing smoothing processing; the face bounding box coordinates after smoothing processing are subjected to extension processing; the face region images are cropped from the corresponding frames of the original video sequence according to the face bounding box coordinates after extension; the sequence composed of multiple face region images is scaled to a unified preset size, and pixel value normalization processing is performed to obtain a standard face image sequence.

[0024] In the embodiment of the present application, the target audio signal is preprocessed; the preprocessed target audio signal is input into a speech representation model pre-trained on multi-language corpus, and the output of the last hidden layer of the speech representation model is extracted as a deep speech vector; the deep speech vector is subjected to tensor rearrangement to obtain a deep audio feature sequence.

[0025] In the preferred embodiment of the present application, an original video sequence containing a target face (format limited to MP4 or AVI) and a corresponding target audio signal are collected; wherein the original video sequence needs to meet the continuous front face speech broadcast scene, without side face, occlusion or large posture change, the speech pause duration is ≤2 seconds, and the background is real scene or green background; the target audio signal can be provided independently or embedded in the original video sequence. The collected original data is subjected to parameter verification, including video parameter verification (resolution ≥1080P, frame rate ≥25fps, time length ≥3 minutes); if the audio parameter is provided independently, it needs to be compatible with 16000Hz sampling rate conversion, and there is no obvious noise, distortion interference; the original data that does not meet the above requirements after verification needs to be preprocessed and adjusted before entering the subsequent process to ensure the accuracy of subsequent feature extraction and model inference.

[0026] The original video sequence is standardized by FFmpeg tool, and the following standard parameters are set: the resolution is uniformly converted to 2160x3840 (4K vertical screen) or 1920x1080 (horizontal screen), to ensure the consistency of image pixel density; the frame rate is forcibly adjusted to 25fps, to lay the foundation for accurate alignment of audio-video frames in the subsequent process; if the target audio signal is embedded in the original video sequence, the audio track is extracted by FFmpeg and temporarily stored as a to-be-processed audio file, with the initial parameters set as 16000Hz sampling rate, single channel, and 16-bit signed integer, to avoid loss of audio information.

[0027] Further, a custom pre-trained face detection model (model file identified as findFaceV2.h5) trained based on the Keras framework is loaded, which has high-precision face positioning capability and can adapt to face detection requirements in different lighting and background scenes. Batch processing mode (16 frames of video frames per batch) is used to improve detection efficiency, and the video frames are uniformly adjusted to 384x384 pixels to match the input size requirements of the findFaceV2.h5 model; the pixel values of the video frames are mapped from [0, 255] to [-1, 1] interval to eliminate the interference of inter-frame brightness difference on detection accuracy. The anchor-based format bounding box output by the findFaceV2.h5 model is converted to absolute coordinates in the original video frame coordinate system, marked as (x1, y1, x2, y2), where (x1, y1) is the left upper corner coordinate of the bounding box, and (x2, y2) is the right lower corner coordinate of the bounding box); non-maximum suppression (NMS) processing is performed on the bounding box detected in each frame, with an IOU (Intersection over Union) threshold of 0.3, to select the single face bounding box with the highest confidence and exclude non-face area false detection.

[0028] Further, the face bounding box coordinates of the continuous 5 frames are smoothed by sliding window average algorithm to eliminate detection jitter and obtain stable face trajectory. To completely retain the mouth shape and chin area, an expansion operation is performed on the smoothed bounding box: the lower boundary is expanded by 20px along the positive direction of y-axis, the left and right boundaries are respectively expanded by 15px along the negative direction and positive direction of x-axis, and the expanded bounding box does not exceed the original video frame size range. According to the expanded bounding box coordinates, the face region images are cropped from the corresponding frames of the original video sequence, all the cropped face region images are adjusted to a preset size of 256x256 pixels, the pixel values of the adjusted face region images are normalized to the interval [0, 1], and the normalized face region images are arranged in the original video frame time sequence to form a standard face image sequence.

[0029] Further, if the target audio signal is embedded in the original video sequence, the audio track is extracted first, and then the target audio signal is uniformly converted into WAV format, with the parameters set as 16000Hz sampling rate, single channel, and 16-bit signed integer. A pre-trained speech representation model (model identifier TencentGameMate / chinese-hubert-large) and a matching Wav2Vec2 feature extractor are loaded on a multi-language corpus. The pre-processed target audio signal is converted into a model-compatible input format using the Wav2Vec2 feature extractor, and the converted input data is input into the speech representation model. The output of the last hidden layer of the model is extracted as a deep speech vector. The tensor shape of the deep speech vector is [1, time_steps, 1024], and the time resolution is 50Hz (i.e., 50 feature vectors are generated per second). Tensor rearrangement is performed on the deep speech vector to adjust its tensor shape from [1, time_steps, 1024] to [1, 1024, time_steps], and then the deep audio feature sequence is integrated in time step order.

[0030] The above scheme establishes high-quality feature representations with strict time sequence correspondence for face sequences and audio signals. This not only includes ensuring the stability and integrity of face region cropping through time sequence smoothing and adaptive expansion, but more importantly, through the integer multiple relationship between video frame rate and deep audio feature sampling rate, an accurate audio-video time alignment mapping is established to the frame level, laying a cornerstone for subsequent frame-by-frame accurate driving.

[0031] S200: Mask processing is performed on each frame of the standard face image sequence to obtain a corresponding to-be-driven mouth shape region image for each frame.

[0032] In an embodiment of the present application, mask processing is performed on each frame of the standard face image sequence, and the mask processing is to set the pixel values within the set region of the face region image to zero while completely retaining the pixel values outside the set region.

[0033] In the preferred embodiment of the present application, the lower half of each frame of the standard face image is set as the mask target region, and the corresponding image line index range is 128-255 (since the image height is 256 pixels, the area below 128 lines covers the mouth shape and chin, which is the mouth shape related area to be driven). The upper middle part of each frame of the standard face image is set as the reserved area, and the corresponding image line index range is 0-127 (covering the forehead, eyebrows, eyes, and other static face areas that do not need to be driven). In the time sequence order of the standard face image sequence, single-frame standard face images are read frame by frame. The pixel value zero operation is performed on the mask area (line index 128-255, column index 0-255) of the frame image, eliminating the original mouth shape features in this area. The pixel values of the reserved area (line index 0-127) remain unchanged, maintaining the original static features of the upper middle part of the face. Each frame of image after completion of the mask processing is marked as "mouth shape area image to be driven", and is arranged in the time sequence order of the original standard face image sequence to obtain a sequence of mouth shape area images to be driven.

[0034] The above scheme is based on precise region division of fixed size, only masks the lower half area (below 128 lines) where the mouth shape is located, and completely reserves the upper middle static area, clearly defines the model learning goal, avoids interference from original mouth shape features, and breaks through the low efficiency problem of traditional full image or fuzzy mask. Through simple mask operation, complex task decoupling is achieved. The "identity preservation" and "mouth shape driving" tasks are separated in the spatial dimension by preserving the identity information of the upper half face and driving only the mouth shape area of the lower half face. The model is forced to focus on learning the audio-mouth shape mapping relationship, fundamentally avoiding modification of the character identity features in the generation process, and ensuring high consistency of the identity in the result video.

[0035] For each frame (from the 1st frame to the Nth frame in time sequence) in the sequence of mouth shape area images to be driven, the following iterative steps are performed: S300: Concatenate the mouth shape area image to be driven of the current frame and the mouth shape image generated by the previous frame along the channel dimension to construct a multi-channel input tensor.

[0036] In the embodiment of the present application, the mouth shape area image to be driven of the current frame is taken as the first input component; if there is a mouth shape image output by the pre-trained mouth shape generation network from the previous frame, it is taken as the second input component; otherwise, a preset value tensor is used as the second input component; the first input component and the second input component are concatenated along the channel dimension to form a multi-channel input tensor.

[0037] In the preferred embodiment of the present application, the current frame of the to-be-driven mouth shape region image sequence is taken as the first input component of the multi-channel input tensor, which is 3 channels (corresponding to RGB channels) and keeps the size of 256x256 pixels. If there is a mouth shape image output by the pre-trained mouth shape generation network in the previous frame, the mouth shape image is directly taken as the second input component (3 channels, 256x256 pixels); if there is no mouth shape image generated in the previous frame (such as the first frame of the sequence), a full-1 tensor is used as the second input component, which has the same dimension as the mouth shape image generated in the previous frame (3 channels, 256x256 pixels, and the pixel value is 1). The first input component and the second input component are spliced along the channel dimension to form a single-frame multi-channel input tensor; the spliced tensor has a dimension of [6, 256, 256] (6 channels, corresponding to 3 channels of the current frame + 3 channels of the previous frame / preset tensor), and the pixel value range keeps [0, 1], which is suitable for the input format of the subsequent mouth shape generation network. According to the time sequence order of the to-be-driven mouth shape region image sequence, the above process is repeated frame by frame to construct a multi-channel input tensor sequence consistent with the number of frames of the to-be-driven mouth shape region image sequence.

[0038] The above scheme takes the generation result of the previous frame as the explicit input of the current frame. This self-recurrent design injects dynamic memory into the static generation model, so that the generation of each frame is based on the state of the previous frame, thereby converting independent frame-level prediction into coherent sequence generation, effectively ensuring the smoothness and naturalness of the mouth shape movement. The integrated input of time sequence information and visual features eliminates the need for an additional time sequence modeling module, simplifying the process while improving the coherence of the mouth shape sequence.

[0039] S400: A local audio feature aligned with the current frame is cut from the deep audio feature sequence and input into the pre-trained mouth shape generation network together with the multi-channel input tensor, and the pre-trained mouth shape generation network outputs a mouth shape image synchronized with the target audio signal through an adaptive feature fusion mechanism.

[0040] In the embodiments of the present application, the mapping relationship between the audio feature time steps and the video frames is calculated according to the sampling rate of the target audio signal and the frame rate of the original video sequence; a local feature block containing multiple time steps is extracted from the deep audio feature sequence according to the mapping relationship; and the local feature block aligned with the center of the current to-be-processed video frame in the time dimension is taken as the local audio feature.

[0041] In the embodiments of the present application, the local audio features are input into a pre-trained lip generation network with a multi-channel input tensor; the pre-trained lip generation network comprises a face encoder, an audio encoder and an adaptive decoder; the adaptive decoder receives multi-scale visual features output by the face encoder and global audio feature vectors output by the audio encoder, and performs layer-by-layer adaptive modulation on the multi-scale visual features by using modulation parameters generated based on the global audio feature vectors to generate a lip image.

[0042] It should be noted that the adaptive decoder performs layer-by-layer adaptive modulation through an adaptive feature fusion module; the adaptive feature fusion module generates scaling parameters and offset parameters by using global audio feature vectors, and performs adaptive instance normalization operation on multi-scale visual features input into the adaptive feature fusion module.

[0043] In the preferred embodiments of the present application, according to the time resolution (50Hz) of the deep audio feature sequence and the frame rate (25fps) of the original video sequence, an alignment factor is determined, alignment factor = 50.0 / 25.0 = 2, that is, 2 audio time steps correspond to 1 frame of video, and finally 10 continuous audio time steps (covering the time sequence information before and after the frame) need to be extracted for each frame. For the i-th frame to be processed, the audio feature starting index is calculated, and a feature block with the starting index as the starting point and containing 10 continuous time steps is cut from the deep audio feature sequence to form local audio features, and the tensor of which is [1, 1024, 10]. The tensor dimension of the local audio features is converted to [1, 1, 1024, 10], which is adapted to the input format of the subsequent audio encoder.

[0044] Further, the multi-channel input tensor sequence is input into a face encoder, multi-scale visual features (including different resolution feature maps from 256x256 to 2x2) are extracted through 8 layers of down-sampling operations, and the features of each layer are reserved for subsequent jump connection; the local audio feature after adjusting the dimension is input into an audio encoder, the feature is gradually compressed through 7 layers of convolution + residual module, and finally a 1024-dimensional global audio feature vector is output; the adaptive decoder receives the multi-scale visual features and the global audio feature vector, and through 8 layers of up-sampling + jump connection structure, an AdaIN adaptive feature fusion module is embedded at each layer, including: performing instance normalization on the input multi-scale visual features to obtain normalized features; the global audio feature vector generates scaling parameters and offset parameters through 1x1 convolution; the adaptive modulation is performed through a preset formula, and the modulation features are output after ReLU6 (linear rectifier function with an upper limit of 6) activation; the above modulation process is repeated layer by layer, and finally a two-dimensional convolution layer (32→3) and Sigmoid activation are used to output the lip image synchronized with the target audio (tensor shape [1,3,256,256], pixel value range [0,1]). The generated lip image is taken as the output result of the current frame, and the "previous frame lip image cache" is updated, and the current frame lip image is stored in the cache to provide a second input component for the multi-channel input tensor of the subsequent frame (i+1 frame).

[0045] The above scheme introduces an adaptive fusion mechanism to dynamically generate scaling parameters (gamma) and offset parameters (beta) for audio features, and modulates visual features layer by layer without manual design rules, automatically matching different phonemes and speech rates to corresponding lip movement patterns. By extracting 10 continuous local audio features per frame, the intra-frame and inter-frame audio information is accurately covered, and the lip-speech synchronization error is ≤±20ms, breaking through the synchronization accuracy limitation of traditional single time step features.

[0046] The adaptive decoder uses the global feature vector extracted from the audio to generate dynamic parameters to modulate the normalized statistics of the visual features. This fine-grained and layer-by-layer feature modulation method can more accurately and flexibly modulate the timing change information of the audio into the visual features, thereby generating realistic lip movements highly synchronized with complex speech.

[0047] S500: Perform boundary fusion processing on the lip image, replace the lip image to the corresponding position of the original video sequence, and combine with the target audio signal to output a digital human video with synchronized lip speech.

[0048] In the embodiments of the present application, a preset width of gradient mask is generated at the joint boundary region of the mouth shape image and the original video sequence; based on the gradient mask, the pixels of the mouth shape image and the pixels of the corresponding frame of the original video sequence in the joint boundary region are weighted and fused to realize visual smooth transition.

[0049] In the preferred embodiments of the present application, the mouth shape image is converted into a NumPy array, de-normalization (pixel value is mapped from [0, 1] to [0, 255]) is performed, and then the mouth shape image is adjusted to the size of the original face region (i.e. the width of x2-x1 and the height of y2-y1) according to the extended face bounding box coordinates (x1, y1, x2, y2), to ensure matching with the face region size of the original video frame. The fusion boundary width is set to 20 pixels (adapted to the visual smoothness requirement to avoid seams), and only the joint boundary region of the mouth shape image and the original video frame is fused, and the non-boundary region directly uses the pixels of the mouth shape image. The bottom 20 rows of pixels of the mouth shape image (corresponding to the boundary near the chin of the original face region) are located, and a linear gradient coefficient (range 0-1, from the inside to the outside of the mouth shape image, increasing from 0 to 1) is generated, and weighted fusion is performed on the bottom 20 rows of pixels according to the calculation formula: fused pixel = a x mouth shape image pixel + (1-a) x corresponding pixel of the original video frame. The left 20 columns of pixels of the mouth shape image (corresponding to the left cheek boundary of the original face region) are located, a linear gradient coefficient (range 0-1, from the inside to the outside of the mouth shape image, decreasing from 1 to 0) is generated, and weighted fusion is performed according to the same formula as above; the right 20 columns of pixels of the mouth shape image (corresponding to the right cheek boundary of the original face region) are located, a linear gradient coefficient (range 0-1, from the inside to the outside of the mouth shape image, increasing from 0 to 1) is generated, and weighted fusion is performed according to the same formula as above.

[0050] Further, the background region of the original video frame is detected, and by calculating the "green channel value - blue channel value" of the pixel, if the difference is > 60, it is determined as a green screen background region, and a binary mask is generated (the background region is marked as 1, and the face region is marked as 0). According to the final pixel = mouth shape image pixel x (1-mask) + original video frame pixel x mask, the original pixels of the green screen background region are retained, and only the mouth shape pixels of the face foreground region are replaced.

[0051] ​Further, according to the extended face boundary box coordinates (x1, y1, x2, y2), the mouth shape image after boundary fusion is accurately pasted back to the starting position (x1, y1) of the corresponding frame of the original video sequence, covering the lower half of the face (mouth shape area) of the original frame to generate a "mouth shape updated video frame". The above replacement operation is repeated frame by frame in the time sequence order of the original video sequence to obtain a "mouth shape synchronized video sequence". Using the FFmpeg tool, the "mouth shape synchronized video sequence" is synthesized with the "target audio signal"; wherein the video encoding and audio encoding are consistent with the original video / audio (the parameter is set to "copy" to avoid quality loss caused by re-encoding). The synthesized video is saved in a preset format (such as AVI), and the storage path is set to. / result / res.avi. Temporary files (such as intermediate frames and feature files) generated during cleaning are finally output as a digital human video with synchronized mouth shape and voice.

[0052] The above scheme effectively eliminates the harsh seams and color differences by designing directional gradient weights for the bottom and side joints of the mouth shape image and the original face area for pixel-level fusion. This targeted fusion strategy ensures that the generated mouth shape area can naturally blend into the original video background, which is a key step to achieve high-quality final visual effect. The 20-pixel gradient weighting fusion technology is used to design differentiated gradient coefficients for the bottom, left and right boundaries, completely eliminating the splicing seams between the mouth shape image and the original video, and improving the visual naturalness. The green screen background detection mechanism is introduced to accurately distinguish the background and the face foreground, avoid modifying the background during replacement, adapt to real scene and green screen multi-scene applications, and break through the limitations of traditional single scene fusion.

[0053] The application further provides a digital human mouth shape voice synchronization generation system based on time sequence decoupling, which is used for realizing the digital human mouth shape voice synchronization generation method based on time sequence decoupling.

[0054] The application embodiment provides a storage medium having a program stored thereon, and the program is executed by a processor to realize the digital human mouth shape voice synchronization generation method based on time sequence decoupling.

[0055] The application embodiment provides a processor for running a program, wherein the program is executed to perform the digital human mouth shape voice synchronization generation method based on time sequence decoupling.

[0056] The embodiment of the present application provides a device, the device comprises a processor, a memory and a program stored in the memory and executable on the processor, and the processor implements a digital population type voice synchronization generation method based on timing decoupling when executing the program. The device herein can be a server, a PC, a PAD, a mobile phone or the like.

[0057] The present application also provides a computer program product suitable for executing the digital population type voice synchronization generation method based on timing decoupling when executed on a data processing device.

[0058] Those skilled in the art should understand that the embodiments of the present application can provide a method, a system or a computer program product. Therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can adopt a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0059] The present application is described with reference to flowcharts and / or block diagrams according to the method, device (system) and computer program product of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams and the combination of the flows and / or blocks can be implemented by computer program instructions. These computer program instructions can be provided to a general purpose computer, a special purpose computer, an embedded processor or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices produce a device implemented in the flowcharts and / or block diagrams. Figure 1 The function of one flow or multiple flows and / or blocks Figure 1 The function of one block or multiple blocks.

[0060] These computer program instructions can also be stored in a computer readable storage medium capable of guiding the computer or other programmable data processing devices to work in a specific way, so that the instructions stored in the computer readable storage medium produce a product including instruction devices, which implement the functions specified in the flowcharts and / or block diagrams. Figure 1 The function of one flow or multiple flows and / or blocks Figure 1 The function of one block or multiple blocks.

[0061] These computer program instructions can also be loaded into a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer implemented process, so that the instructions executed on the computer or other programmable device provide a process for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The function of one flow or multiple flows and / or blocks Figure 1steps of a function specified in one or more blocks.

[0062] In one typical arrangement, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0063] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) about which the computer stores information about an operating system, application software, and / or the like. Memory is an example of computer readable media.

[0064] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0065] It should also be noted that the terms "comprising", "containing", or any other similar term are intended to encompass non-exclusive inclusions such that a process, method, article, or apparatus that comprises a list of elements does not include those elements solely, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0066] The above merely provides an embodiment of the present application and is not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A method for synchronously generating digital lip-sync speech based on temporal decoupling, characterized in that, include: The original video sequence containing the target face and the target audio signal are acquired. Face detection and cropping are performed on the original video sequence to obtain a standard face image sequence. Feature extraction is performed on the target audio signal to obtain a deep audio feature sequence. Each frame in the standard face image sequence is masked to obtain the lip shape region image to be driven for each frame. The lip shape region image to be driven in the current frame is concatenated with the lip shape image generated in the previous frame along the channel dimension to construct a multi-channel input tensor. Local audio features aligned with the current frame are extracted from the deep audio feature sequence and input to a pre-trained lip-sync generation network along with the multi-channel input tensor. The pre-trained lip-sync generation network outputs a lip-sync image synchronized with the target audio signal through an adaptive feature fusion mechanism. The lip-sync image is subjected to boundary fusion processing. After the lip-sync image is replaced with the corresponding position in the original video sequence, it is combined with the target audio signal to output a lip-sync synchronized digital human video.

2. The digital lip-sync speech generation method based on temporal decoupling according to claim 1, characterized in that, The original video sequence is subjected to face detection and cropping to obtain a standard face image sequence, including: The original video sequence is subjected to frame-by-frame face detection using a pre-trained face detection model to obtain the face bounding box coordinates for each frame. Temporal smoothing is performed on the face bounding box coordinates of multiple consecutive frames; Expand the coordinates of the smoothed face bounding box; Based on the expanded face bounding box coordinates, the face region image is cropped from the corresponding frame of the original video sequence; The sequence of multiple frames of the face region images is scaled to a uniform preset size and then normalized to obtain a standard face image sequence.

3. The digital lip-sync speech generation method based on temporal decoupling according to claim 1, characterized in that, The step of extracting features from the target audio signal to obtain a deep audio feature sequence includes: Preprocess the target audio signal; The preprocessed target audio signal is input into a speech representation model pre-trained on a multilingual corpus, and the output of the last hidden layer of the speech representation model is extracted as a deep speech vector. Tensor rearrangement is performed on the deep speech vectors to obtain the deep audio feature sequence.

4. The digital lip-sync speech generation method based on temporal decoupling according to claim 1, characterized in that, The masking process for each frame in the standard face image sequence includes: masking each frame of the face region image constituting the standard face image sequence, wherein the masking process sets the pixel values ​​within a set area of ​​the face region image to zero, while completely preserving the pixel values ​​outside the set area.

5. The digital lip-sync speech generation method based on temporal decoupling according to claim 1, characterized in that, The step of concatenating the lip-shape region image to be driven in the current frame with the lip-shape image generated in the previous frame along the channel dimension to construct a multi-channel input tensor includes: Use the image of the lip-sync region to be driven in the current frame as the first input component; If there exists a lip shape image output by the pre-trained lip shape generation network in the previous frame, then use it as the second input component; otherwise, use a preset value tensor as the second input component. The first input component and the second input component are concatenated along the channel dimension to form a multi-channel input tensor.

6. The digital lip-sync speech generation method based on temporal decoupling according to claim 1, characterized in that, The step of extracting local audio features aligned with the current frame from the deep audio feature sequence and inputting them into the pre-trained lip-sync generation network using the multi-channel input tensor includes: Based on the sampling rate of the target audio signal and the frame rate of the original video sequence, calculate the mapping relationship between the audio feature time steps and the video frames; Based on the mapping relationship, a local feature block containing multiple time steps is extracted from the deep audio feature sequence; The local feature block aligned with the center of the current video frame to be processed in the time dimension is used as the local audio feature.

7. The digital lip-sync speech generation method based on temporal decoupling according to claim 1, characterized in that, The step of extracting local audio features aligned with the current frame from the deep audio feature sequence and inputting them into the lip-sync generation network pre-trained with the multi-channel input tensor further includes: Local audio features are input into a pre-trained lip-sync generation network along with a multi-channel input tensor; The pre-trained lip-sync generation network includes a face encoder, an audio encoder, and an adaptive decoder; The adaptive decoder receives the multi-scale visual features output by the face encoder and the global audio feature vector output by the audio encoder, and uses the modulation parameters generated based on the global audio feature vector to perform layer-by-layer adaptive modulation on the multi-scale visual features to generate a lip-sync image.

8. The digital lip-sync speech generation method based on temporal decoupling according to claim 7, characterized in that, The adaptive decoder performs layer-by-layer adaptive modulation through the adaptive feature fusion module; the adaptive feature fusion module uses the global audio feature vector to generate scaling parameters and offset parameters, and performs adaptive instance normalization on the multi-scale visual features input to the adaptive feature fusion module.

9. The digital lip-sync speech generation method based on temporal decoupling according to claim 1, characterized in that, The boundary fusion processing of the lip-sync image includes: generating a gradient mask of a preset width in the junction boundary region between the lip-sync image and the original video sequence; and performing weighted fusion of the pixels of the lip-sync image in the junction boundary region with the pixels of the corresponding frame of the original video sequence based on the gradient mask to achieve a visually smooth transition.

10. A digital lip-sync speech generation system based on temporal decoupling, characterized in that, The system includes a control module, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement the digital lip-sync speech generation method based on temporal decoupling according to any one of claims 1-9.

Images