Video generation method and device, computer equipment and storage medium

Abstract

The embodiment of the invention relates to a video generation method and device, computer equipment and a storage medium, and the method comprises the steps: carrying out the filling processing of each frame of image of an initial video, and obtaining a first image; obtaining a second image from the initial video, and obtaining a target audio; extracting audio features from the target audio, extracting first image features from the first image, and extracting second image features from the second image; performing alignment operation on the first image feature and the second image feature; performing spatial deformation on the second image feature according to the audio feature and the aligned second image feature; generating a mouth shape image according to the deformed second image feature and the first image feature; and generating a target video according to each mouth shape image and the target audio. Therefore, the target video with highly synchronous mouth shape and voice content can be generated while the character identity features are kept, and the naturalness and visual reality of mouth shape and voice matching generation are improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of video processing technology, and in particular to a video generation method, apparatus, computer equipment, and storage medium. Background Art

[0002] With the widespread application of voice-driven virtual character technology in scenarios such as digital humans, video conferencing, virtual anchors, and film and television production, how to generate synchronized and natural lip animations based on input voice has become one of the key technologies. Although the existing technology has proposed a variety of methods based on rules, deep learning, or 3D modeling, among them, the rule-based method is to design rules based on the mapping relationship between phonemes and lip shapes, but the performance is relatively mechanical and lacks natural transitions. The deep learning-based method is to generate lip images directly from audio, but it often lacks details and personalized features. The 3D model-based method is to generate lip shapes by controlling the parameters of the 3D model, but the calculation is complex and relies on an accurate 3D model.

[0003] In practical applications, the problem of lip-syncing and speech content being out of sync is common, especially when speaking at a fast rate or with unusual pronunciations. This severely impacts the realism of the visual presentation and the interactive experience. Therefore, a technical solution is urgently needed to improve the accuracy of the temporal alignment between speech and lip-syncing, addressing the issue of lip-syncing mismatch caused by inaccurate cross-modal mapping. Summary of the Invention

[0004] In view of this, in order to solve the above technical problems or part of the technical problems, the embodiments of the present invention provide a video generation method, apparatus, computer equipment and storage medium.

[0005] In a first aspect, an embodiment of the present invention provides a video generation method, comprising:

[0006] Filling the mouth area of the character in each frame of the initial video to obtain a first image corresponding to each processed frame;

[0007] Acquire a preset number of second images from the initial video, and acquire target audio corresponding to the initial video;

[0008] extracting audio features from the target audio, extracting first image features from each of the first images, and extracting second image features from each of the second images;

[0009] performing an alignment operation on the first image feature and the second image feature to obtain an aligned second image feature;

[0010] performing spatial deformation processing on the second image feature according to the audio feature and the aligned second image feature to obtain a deformed second image feature;

[0011] generating a lip shape image corresponding to each first image according to the deformed second image feature and the first image feature;

[0012] A target video is generated according to each of the lip-sync images and the target audio.

[0013] In a possible implementation, extracting audio features from the target audio, extracting first image features from each of the first images, and extracting second image features from each of the second images includes:

[0014] Extracting the target audio context window through a trained audio feature extraction network to obtain initial audio features of a preset duration corresponding to each first image;

[0015] The audio feature is extracted for each of the initial audio features by a convolutional neural network, the first image feature is extracted for each of the first images by a convolutional neural network, and the second image feature is extracted for each of the second images by a convolutional neural network.

[0016] In one possible implementation, aligning the first image feature and the second image feature to obtain the aligned second image feature includes:

[0017] Calculating a similarity matrix between the plurality of first image features and the plurality of second image features through an attention mechanism, and performing normalization processing to obtain a plurality of attention weights;

[0018] Perform a weighted summation on each of the attention weights and the corresponding second image feature to generate the aligned second image feature that is aligned with the feature space of the first image feature.

[0019] In a possible implementation, performing spatial deformation processing on the second image feature according to the audio feature and the aligned second image feature to obtain the deformed second image feature includes:

[0020] Copying and expanding the audio feature to the same spatial dimension as the aligned second image feature to obtain an expanded audio feature;

[0021] splicing the aligned second image features and the expanded audio features in the channel dimension, and predicting the offset of the variable convolution through a convolution layer;

[0022] A variable convolution operation is performed on the second image feature using the offset to obtain the deformed second image feature.

[0023] In a possible implementation, generating a lip image corresponding to each first image based on the deformed second image feature and the first image feature includes:

[0024] splicing the deformed second image feature and the first image feature in a channel dimension;

[0025] The convolutional network is transposed and upsampled step by step. After upsampling at each layer, the features of the corresponding layer of the encoder are introduced through a jump connection. The last layer uses a target activation function to output an image in the target range to obtain the lip shape image.

[0026] In one possible implementation, obtaining a preset number of second images from the initial video includes:

[0027] Identifying a target person contained in each frame of the initial video;

[0028] Recognize the mouth shape of the target person in each frame of the image;

[0029] A preset number of images of the target person with different lip shapes are obtained from the initial video to obtain the second image.

[0030] In a possible implementation, after obtaining a preset number of second images from the initial video, the method further includes:

[0031] performing face detection and alignment processing on each of the first images and each of the second images;

[0032] The processed first image and the second image are normalized into images of a preset size.

[0033] In a second aspect, an embodiment of the present invention provides a video generation device, including:

[0034] A first processing module is used to perform filling processing on the mouth area of the character in each frame image of the initial video to obtain a first image corresponding to each processed frame image;

[0035] an acquisition module, configured to acquire a preset number of second images from the initial video, and acquire target audio corresponding to the initial video;

[0036] an extraction module, configured to extract audio features from the target audio, extract first image features from each of the first images, and extract second image features from each of the second images;

[0037] a second processing module, configured to perform an alignment operation on the first image feature and the second image feature to obtain an aligned second image feature;

[0038] a third processing module, configured to perform spatial deformation processing on the second image feature according to the audio feature and the aligned second image feature to obtain a deformed second image feature;

[0039] A first generating module, configured to generate a lip shape image corresponding to each first image according to the deformed second image feature and the first image feature;

[0040] The second generating module is used to generate a target video according to each of the lip-shaped images and the target audio.

[0041] In a third aspect, an embodiment of the present invention provides a computer device, comprising: a processor and a memory, wherein the processor is configured to execute a program stored in the memory to implement the video generation method described in any one of the first aspects above.

[0042] In a fourth aspect, an embodiment of the present invention provides a storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the video generation method described in any one of the first aspects above.

[0043] The video generation scheme provided by the embodiment of the present invention performs a filling process on the mouth area of a person in each frame of an initial video to obtain a first image corresponding to each processed frame; obtains a preset number of second images from the initial video, and obtains a target audio corresponding to the initial video; extracts audio features from the target audio, extracts first image features from each first image, and extracts second image features from each second image; aligns the first image features and the second image features to obtain aligned second image features; spatially deforms the second image features based on the audio features and the aligned second image features to obtain deformed second image features; generates a lip shape image corresponding to each first image based on the deformed second image features and the first image features; and generates a target video based on each lip shape image and the target audio. In this way, the mouth area in each frame of the initial video can be filled, and the audio features and image features can be combined for alignment and spatial deformation, so that the target video generation can be achieved while maintaining the identity of the person and achieving a high degree of synchronization between the lip shape and the voice content, thereby improving the naturalness and visual realism of the lip shape and voice matching generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of a flow chart of a video generation method provided by an embodiment of the present invention;

[0045] Figure 2 A schematic structural diagram of a video generation system provided by an embodiment of the present invention;

[0046] Figure 3 A schematic structural diagram of a video generation device provided by an embodiment of the present invention;

[0047] Figure 4 A schematic structural diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0049] To facilitate understanding of the embodiments of the present invention, specific embodiments will be further explained below with reference to the accompanying drawings. The embodiments do not limit the embodiments of the present invention.

[0050] Figure 1 A flow chart of a video generation method provided by an embodiment of the present invention is shown as follows: Figure 1 As shown, the method specifically includes:

[0051] S11 , performing filling processing on the mouth area of the character in each frame image of the initial video to obtain a first image corresponding to each processed frame image.

[0052] The video generation method provided in embodiments of the present invention is applied to computer devices, including but not limited to servers, desktop computers, and tablet computers, and is used in scenarios where a video of a person is generated that matches their speech, with the person's lip movements in the video matching the speech. Specifically, by filling the mouth area in each frame of the initial video and combining audio and image features for alignment and spatial deformation, the method can achieve target video generation with highly synchronized lip movements and speech content while preserving the person's identity.

[0053] In this embodiment, the initial video is a video that needs to lip-sync to the target audio. The initial video can be a video of the same person speaking in the picture, and the facial image of the person in the video is clear and unobstructed. The target audio is the voice dubbing of the initial video, and the lip sync of the person in the initial video needs to be matched with the voice content in the target audio.

[0054] Specifically, face detection is performed frame by frame on the initial video, and the mouth region of the person in each frame is located (for example, using a facial landmark detection algorithm such as MediaPipe or OpenCV's 68-point face annotation). Filling processing is performed within the mouth region, for example, filling the mouth region with black so that new mouth shapes can be generated in this region later. Each processed frame is output as the first image set.

[0055] S12: Acquire a preset number of second images from the initial video, and acquire target audio corresponding to the initial video.

[0056] In this embodiment, obtaining a preset number of second images from the initial video includes: identifying the target person contained in each frame of the initial video; identifying the lip shape of the target person in each frame of the initial video; and obtaining a preset number of images of the target person with different lip shapes from the initial video to obtain the second image.

[0057] Specifically, several frames of images are extracted from the initial video at certain time intervals or random sampling as reference images (second images). Each second image contains the same target person, and the target person's lip shape is different in each second image. The second images are used to learn the static lip shape features of the person.

[0058] At the same time, the target audio corresponding to the initial video is obtained for subsequent generation of lip movements synchronized with the voice. At this time, the lip movements of the characters in the initial video are not synchronized with the target audio.

[0059] S13. Extract audio features from the target audio, extract first image features from each first image, and extract second image features from each second image.

[0060] In this embodiment, the target audio is input into a feature extraction network (such as using a pre-trained audio feature extraction model wav2vec, HuBERT, etc.) to extract audio feature vectors reflecting information such as phonemes and pronunciation rhythm.

[0061] A convolutional neural network, an image coding network (such as ResNet, VisionTransformer, etc.) is used for each first image to extract visual features to obtain first image features. Similarly, visual features are extracted for each second image to obtain second image features.

[0062] In one possible embodiment, extracting audio features from the target audio includes: extracting the target audio context window through a trained audio feature extraction network to obtain initial audio features of a preset duration corresponding to each first image; and extracting audio features for each initial audio feature through a convolutional neural network.

[0063] Specifically, the audio signal is first converted into a Mel-spectrogram. The original audio signal is preprocessed, such as resampling (typically to 16kHz) and pre-emphasis. The spectrum is extracted using a short-time Fourier transform (STFT) and then converted to a Mel-spectrogram using a Mel filter bank. This reflects the time-frequency structure of the audio and is suitable for human auditory perception.

[0064] Furthermore, a pre-trained audio feature model (such as HuBERT / wav2vec) is used to extract contextual features. The mel-spectrogram or raw waveform is input into a pre-trained audio feature model (such as wav2vec 2.0 or HuBERT). An audio context window is extracted within a range of 2 seconds (preset duration) before and after the time point corresponding to the current first image to ensure that the lip shape generation has a complete speech dynamic background. Thus, the initial audio features are obtained and the extracted initial audio features are normalized using mean-variance normalization or batch normalization to eliminate amplitude differences between different samples and enhance network robustness.

[0065] The normalized initial audio features are input into the audio encoding module, and one-dimensional convolution is used to extract local time features. A one-dimensional convolutional neural network (1D-CNN) is constructed to model the local features of the audio sequence: convolution kernel size: 3×1 (sliding only in the time dimension); step size: 1; Batch Normalization and ReLU activation functions are added after each layer. The channel dimension is expanded layer by layer: 32→64→128→256 to increase the feature expression capability of the model. The feature sequence output after multiple layers of convolution is used as the audio feature vector corresponding to the current image time point, which is used for lip-shaping driving by the subsequent spatial deformation module. Thus, the audio feature (F audio ).

[0066] In one possible embodiment, before extracting features from the first image and the second image, each first image and each second image needs to be preprocessed, including: facial detection and alignment processing; normalizing the processed first image and second image to images of a preset size (for example, 256 (width) × 416 (height) pixels).

[0067] Facial detection can be performed using a pre-trained face detection model (such as MTCNN, RetinaFace, or Face Alignment Network) on each input image (first image) and each reference image (second image). The detection output includes: a facial bounding box; and the position coordinates of key points (such as the eyes, nose tip, and mouth corners). Facial alignment can be performed by performing an affine transformation based on the detected key points (such as the eyes, nose, and mouth) to rotate, scale, and translate the face to a standard position. The alignment standard is usually to fix the eye distance and mouth position. This ensures that all faces are in a unified reference coordinate system, reducing the impact of expression and angle changes on subsequent feature extraction.

[0068] In one possible embodiment, after preprocessing the first image and the second image, first image features are extracted from each first image, and second image features are extracted from each second image, including: extracting the first image features for each first image through a convolutional neural network, and extracting the second image features for each second image through a convolutional neural network.

[0069] Specifically, the first image (input image) feature extraction includes: inputting the first image after single-frame alignment and normalization into the "input image encoding module", the size of which can be B×3×H×W, and extracting image features layer by layer through a 5-layer convolutional neural network (CNN); after each layer of convolution, sequentially performing: convolution operation (convolution kernel size 4×4, step size 2) for spatial downsampling; batch normalization (BatchNorm) to stabilize feature distribution; LeakyReLU activation (usually with a negative slope of 0.2) to increase nonlinear expression capabilities. Feature channel changes: 64→128→256→512→512; output: obtain the deep feature representation corresponding to the input image (recorded as multiple first image features F input ).

[0070] The feature extraction of the second image (reference image) includes: inputting a preset number (4) of aligned and normalized second images into the "reference image encoding module", each with a size of B×3×H×W; concatenating the 4 images in the channel dimension to obtain a feature map of size B×12×H×W, which is input into a specially designed reference image encoding network; also using a 5-layer convolutional neural network, consistent with the first image feature extraction process: the convolution kernel size is 4×4, the stride is 2; each convolution layer is followed by batch normalization and LeakyReLU activation. Note: The number of input channels of the first convolution layer is 12 (due to the splicing of 4 RGB images); the feature channel changes: 64→128→256→512→512; output: the deep feature representation of the reference image is obtained (denoted as multiple second image features Frefer ).

[0071] In this way, the deep visual features of the input image and the reference image can be effectively extracted, providing high-quality input for subsequent feature alignment and lip shape generation; multiple reference images are cascaded to integrate information from multiple angles and different expressions of the character, thereby enhancing the diversity and accuracy of lip shape generation; downsampling layer by layer can extract semantic features with a larger receptive field, thereby improving the model's sensitivity to local lip shape changes.

[0072] S14: Perform an alignment operation on the first image feature and the second image feature to obtain an aligned second image feature.

[0073] In this embodiment, the second image features are mapped to a feature space that is consistent with the first image feature space through feature matching, feature normalization, or a small Transformer alignment network. The aligned features can be better used for subsequent fusion and deformation.

[0074] In one possible embodiment, a similarity matrix between multiple first image features and multiple second image features is calculated through an attention mechanism, and after normalization, multiple attention weights are obtained; each attention weight is weighted and summed with the corresponding second image feature to generate an aligned second image feature that is aligned with the feature space of the first image feature.

[0075] Specifically, the first image feature F input and the second image feature F refer Input the "feature alignment module" to perform a dot product operation or cross-channel matching on the first image feature and the second image feature to obtain a similarity matrix; this similarity matrix represents the degree of match between each input image feature position and the reference image feature position.

[0076] The similarity matrix is normalized along the dimension of the reference feature through the Softmax function; the normalized matrix is the attention weight, which represents the importance distribution of the reference feature (second image feature) to the input feature (first image feature) at each position.

[0077] Use the obtained attention weights to perform weighted summation on the second image features; generate a new feature representation so that the second image features are aligned with the first image features in terms of spatial distribution. Obtain the aligned second image features F aligned , used for subsequent spatial deformation and fusion processing.

[0078] Therefore, flexible alignment can be performed through the attention mechanism, so that the reference image features can accurately adapt to the spatial position of the input image features; make up for the inaccuracy of traditional hard alignment (such as affine transformation) under complex expression changes; and help maintain high-quality details and identity consistency when subsequently generating lip shapes.

[0079] S15. Perform spatial deformation processing on the second image feature according to the audio feature and the aligned second image feature to obtain a deformed second image feature.

[0080] In this embodiment, a spatial warping network (e.g., using learnable deformation convolution or flow-based warping) is designed to dynamically adjust the spatial structure of the second image feature based on the audio features of the current frame. This warping operation can simulate movements such as mouth opening and closing and lip stretching, allowing the feature to change naturally with pronunciation. The resulting warped second image feature is obtained.

[0081] In one possible implementation, the audio feature is copied and expanded to the same spatial dimension as the aligned second image feature to obtain the expanded audio feature; the aligned second image feature and the expanded audio feature are spliced in the channel dimension, and the offset of the variable convolution is predicted through the convolution layer; the offset is used to perform a variable convolution operation on the second image feature to obtain the deformed second image feature.

[0082] Specifically, the aligned second image feature F aligned and audio feature F audio Input the "deformation module (variable convolution)", copy and expand the audio features, and adjust them to the same spatial dimension and spatial size (height H, width W) as the image features; the expansion can be performed using nearest neighbor interpolation or bilinear interpolation. The expanded audio features are spliced with the aligned second image features in the channel dimension; the spliced features contain visual information (second image) and voice-driven information (audio). The spliced features are input into the convolution layer (usually using 3×3 convolution); the convolution layer outputs the offset corresponding to each position for subsequent deformable convolution operations; the offset controls the position offset of the sampling point in the local area. Using the predicted offset, the aligned second image features are deformed and sampled based on deformable convolution; this allows the second image features to dynamically adjust their local structure according to changes in the audio to simulate changes in lip shape. Obtain the deformed second image feature F deformed , which has integrated the audio information of the current moment and adjusted the local deformation according to the audio content.

[0083] Therefore, by introducing spatial deformation guided by audio features, local expression adjustments can be achieved according to dynamic changes in pronunciation; the unnatural lip shape or stiffness of the static second image under pronunciation changes can be effectively avoided; and the natural fluency and synchronization accuracy of the generated lip shape image can be greatly improved.

[0084] S16: Generate a lip shape image corresponding to each first image according to the deformed second image features and the first image features.

[0085] In this embodiment, the deformed second image features are fused with the corresponding first image features (which can be done by concatenation, weighted summation, or a cross-attention mechanism). The fused features are then input into an image generation module (such as a UNet, GAN, or a diffusion-based decoder) to generate high-quality images (lip-sync images) with correct lip movements in each frame.

[0086] In one possible implementation, the deformed second image features and the first image features are concatenated in the channel dimension; upsampling is performed step by step through a transposed convolutional network, and after upsampling at each layer, the features of the corresponding layer of the encoder are introduced through a jump connection. The last layer uses a target activation function to output an image in the target range to obtain a lip shape image.

[0087] Specifically, the first image feature F input and the deformed second image feature F deformed The image is then input into the "image decoding module," where the deformed second image features are concatenated with the first image features along the channel dimension, combining the deformed features with the original appearance information. Five layers of transposed convolution (or fractionally-strided convolution) are used, with progressive upsampling. Each layer uses a transposed convolution kernel of 4×4 with a stride of 2. Batch normalization (BatchNorm) and ReLU activation functions are added after each layer, reducing the number of feature channels to 512 → 256 → 128 → 64 → 3 (the final RGB image). Skip connections are then made. After each upsampling layer, features from the corresponding encoder layer (i.e., intermediate features from the input image encoder) are introduced via skip connections. This approach helps preserve more local details and low-level texture information, improving image clarity and realism.

[0088] After the final transposed convolution layer, the Tanh activation function is used to normalize the output pixel values to the range [-1, 1]. This generates the final lip-sync image (RGB format). The synthesized lip-sync image corresponding to each frame is output; all frames can then be combined into a coherent video sequence.

[0089] Therefore, by fusing the original image features with the deformed reference features and combining jump connections, we can ensure that the lip shape changes are natural and realistic, and maintain the identity characteristics of the character; the spatial resolution is gradually restored during the upsampling process to ensure that the generated image is rich in details and has smooth edges. The final output lip shape image is highly synchronized with the input speech content, and the visual effect is realistic and smooth.

[0090] S17. Generate a target video according to each lip image and the target audio.

[0091] In this embodiment, all generated lip images are arranged into a new video frame sequence in the order of the corresponding first images, and the target audio is synchronously synthesized with the generated video sequence to form a final target video.

[0092] In one possible implementation, a training loss function is processed, and a combination of multi-task loss functions used in model training includes: 1. Reconstruction Loss, form: using the L1 loss function (mean absolute error), function: measuring the difference between the generated image and the real image at the pixel level, and encouraging the generated result to restore the appearance of the original image as much as possible. Advantage: L1 can better preserve edges and details than L2, and is not prone to blurring. 2. Perceptual Loss, form: input the generated image and the real image into the pre-trained VGG network, extract multiple levels of feature representations; calculate the L2 distance in these feature spaces. Function: capture high-level information such as image structure, texture, and semantics, improve the overall perceptual quality of the image, and make the result more natural and realistic. 3. Adversarial Loss, form: adopting the GAN architecture, introducing the discriminator network, using RaLSGAN or standard adversarial loss (such as BCE) to measure the distinguishability of the generated image and the real image. Purpose: Enhances the realism of generated images, making them more visually deceptive. Training method: The discriminator determines the authenticity of the image, while the generator attempts to "fool" the discriminator. 4. Lip Region Weighted Loss: Form: Based on the reconstruction loss, a higher loss weight is assigned to the lip region (obtained through facial landmark detection). Purpose: Strengthens learning of details in the mouth region, improving the accuracy and clarity of lip shape generation. Method: Separate calculations and re-integration can be performed through mask weighting or region cropping. 5. Synchronization Loss: Form: Introduces a synchronization discriminant network to determine whether the audio and lip-sync image are synchronized; the generated image is calculated to measure the synchronization loss with the audio. Purpose: Strengthens the temporal alignment between lip shape and speech, preventing lip shape drift or desynchronization. The training objective of the final loss function is to minimize the total loss: Ltotal = λ1Lrecon + λ2Lperc + λ3Ladv + λ4Llip + λ5Lsync. The weights λi of each loss can be adjusted based on actual training results.

[0093] The video generation method provided by an embodiment of the present invention combines reference images and a deformable convolutional network to achieve high-quality lip-sync generation driven by audio. The speaker's facial features are retained, and only the lip-sync area is deformed, resulting in more natural and personalized lip-sync generation. Through a multi-reference image mechanism, the system receives not only input images with occluded mouth areas, but also multiple reference images of the same speaker with different lip shapes. This provides rich visual information for subsequent lip-sync generation. A deformable convolutional network is used as the core deformation module. Unlike traditional convolution, deformable convolution can adaptively adjust the sampling position of the convolution kernel based on the input content, thereby achieving more flexible feature extraction and deformation capabilities. The offset of the deformable convolution is determined by both audio and image features, allowing lip-sync deformation to accurately follow the speech content. Furthermore, to address the temporal incoherence problem in lip-sync generation, the present invention introduces an audio context window mechanism. The system considers not only the audio features at the current moment, but also audio information over a period of time, thereby capturing the dynamic changes in speech and generating smoother and more coherent lip movements. Secondly, a specialized alignment encoding network was designed to effectively integrate audio and image features. Through this alignment mechanism, the system can accurately map speech content to corresponding lip movements, improving the accuracy and naturalness of the generated results.

[0094] like Figure 2 As shown in FIG, a schematic diagram of the structure of a video generation system provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, the system specifically includes: a speech encoding module, an input image encoding module, a reference image encoding module, a feature alignment module, a deformation module and an image decoding module. The first image is used as the input image and the second image is used as the reference image for description.

[0095] First, obtain the input data, including: a single person speaking video of more than 5 minutes. Split the video into training data triplets. A single training data triplet includes:

[0096] Input image: The speaker's face image with the mouth area filled in black.

[0097] Reference images: Multiple facial images of the same speaker, collected from the same video clip, with different lip states

[0098] Audio data: speech signal corresponding to the target mouth shape

[0099] Furthermore, face detection and alignment are performed on the input image and the reference image, and the images are normalized to a uniform size (e.g., 256 (width) × 416 (height) pixels).

[0100] 1. Feature extraction stage

[0101] Input data: includes reference image group (multiple reference images), input image (current frame image in the target video) and audio input (corresponding target audio).

[0102] Reference image encoding module: used to cascade multiple reference images on the channel and extract the deep features of the reference image through a convolutional neural network.

[0103] Input image encoding module: used to extract features of the input image and obtain input image features.

[0104] Speech encoding module: used to convert audio input into Mel-spectrogram and then extract audio features using a pre-trained audio feature extraction network.

[0105] 2. Feature fusion and deformation stage

[0106] Feature alignment module: used to perform similarity matching between input image features and reference image features, and to achieve spatial alignment between reference features and input image features through the attention mechanism.

[0107] Deformation module: used to fuse the aligned reference image features with the audio features, perform variable convolution on the reference features by predicting the spatial deformation offset, and adjust the feature shape to match the speech-driven lip changes.

[0108] 3. Image generation stage

[0109] Image decoding module: used to fuse the features of the deformed reference image with the input image features, splice them in the channel dimension, gradually upsample them through transposed convolution, and use jump connections to restore details, and finally generate a resolution that is the same as the original. Figure 1 A precise RGB lip image.

[0110] Output: Generates lip-sync images synchronized with the current audio clip for subsequent synthesis of continuous video frames.

[0111] Therefore, we can introduce variable convolution to perform spatial deformation on the feature map of the reference image so that its mouth shape is synchronized with the driving audio. Compared with the traditional direct generation method, variable convolution can more accurately control the changes in mouth shape, retain the speaker's personal characteristics, generate more natural and realistic mouth movements, design the audio context window, and consider the longer audio information before and after the current image. It effectively solves the problems of temporal incoherence and jitter in mouth shape generation, making the generated mouth movements smoother and more natural.

[0112] The video generation system provided in this embodiment can be as follows Figure 2 The system shown in , can execute Figure 1-2 All steps of the video generation method in Figure 1-2For technical effects of the video generation method shown, please refer to Figure 1-2 For the sake of brevity, the relevant description will not be repeated here.

[0113] Figure 3 A schematic diagram of the structure of a video generation device provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, the device specifically includes:

[0114] A first processing module 31 is configured to perform filling processing on the mouth area of the character in each frame image of the initial video to obtain a first image corresponding to each processed frame image;

[0115] an acquisition module 32, configured to acquire a preset number of second images from the initial video, and acquire target audio corresponding to the initial video;

[0116] an extraction module 33, configured to extract audio features from the target audio, extract first image features from each of the first images, and extract second image features from each of the second images;

[0117] A second processing module 34 is configured to align the first image feature and the second image feature to obtain an aligned second image feature;

[0118] a third processing module 35, configured to perform spatial deformation processing on the second image feature according to the audio feature and the aligned second image feature to obtain a deformed second image feature;

[0119] A first generating module 36 is configured to generate a lip image corresponding to each first image based on the deformed second image features and the first image features;

[0120] The second generating module 37 is configured to generate a target video according to each of the lip-shaped images and the target audio.

[0121] In one possible implementation, the extraction module is specifically configured to extract the target audio context window using a trained audio feature extraction network to obtain initial audio features of a preset duration corresponding to each first image;

[0122] The audio feature is extracted for each of the initial audio features by a convolutional neural network, the first image feature is extracted for each of the first images by a convolutional neural network, and the second image feature is extracted for each of the second images by a convolutional neural network.

[0123] In one possible implementation, the second processing module is specifically configured to calculate a similarity matrix between the plurality of first image features and the plurality of second image features through an attention mechanism, and obtain a plurality of attention weights after performing normalization processing;

[0124] Perform a weighted summation on each of the attention weights and the corresponding second image feature to generate the aligned second image feature that is aligned with the feature space of the first image feature.

[0125] In a possible implementation, the third processing module is specifically configured to copy and expand the audio feature to the same spatial dimension as the aligned second image feature to obtain an expanded audio feature;

[0126] splicing the aligned second image features and the expanded audio features in the channel dimension, and predicting the offset of the variable convolution through a convolution layer;

[0127] A variable convolution operation is performed on the second image feature using the offset to obtain the deformed second image feature.

[0128] In a possible implementation, the first generating module is specifically configured to splice the deformed second image feature and the first image feature in a channel dimension;

[0129] The convolutional network is transposed and upsampled step by step. After upsampling at each layer, the features of the corresponding layer of the encoder are introduced through a jump connection. The last layer uses a target activation function to output an image in the target range to obtain the lip shape image.

[0130] In a possible implementation, the acquisition module is specifically configured to identify a target person contained in each frame of the initial video;

[0131] Recognize the mouth shape of the target person in each frame of the image;

[0132] A preset number of images of the target person with different lip shapes are obtained from the initial video to obtain the second image.

[0133] In a possible implementation, the fourth processing module 38 is configured to perform face detection and alignment processing on each of the first images and each of the second images;

[0134] The processed first image and the second image are normalized into images of a preset size.

[0135] The video generation device provided in this embodiment can be as follows Figure 3 The device shown in , can perform the following Figure 1-2All steps of the video generation method in Figure 1-2 For technical effects of the video generation method shown, please refer to Figure 1-2 For the sake of brevity, the relevant description will not be repeated here.

[0136] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention is provided. Figure 4 The computer device 400 shown includes: at least one processor 401, memory 402, at least one network interface 404 and other user interfaces 403. The various components in the computer device 400 are coupled together via a bus system 405. It is understood that the bus system 405 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 405 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 405 is not described in detail. Figure 4 Various buses are labeled as bus system 405 .

[0137] The user interface 403 may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, a touchpad, or a touch screen).

[0138] It is understood that the memory 402 in the embodiment of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDRSDRAM), enhanced synchronous DRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 402 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0139] In some embodiments, the memory 402 stores the following elements, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system 4021 and application programs 4022 .

[0140] The operating system 4021 includes various system programs, such as a framework layer, a core library layer, and a driver layer, for implementing various basic services and handling hardware-based tasks. Application programs 4022 include various application programs, such as a media player and a browser, for implementing various application services. Programs implementing the methods of the embodiments of the present invention may be included in application programs 4022.

[0141] In an embodiment of the present invention, by calling a program or instruction stored in the memory 402, specifically, a program or instruction stored in the application 4022, the processor 401 is configured to execute the method steps provided in each method embodiment, for example, including:

[0142] Filling the mouth area of the character in each frame of the initial video to obtain a first image corresponding to each processed frame;

[0143] Acquire a preset number of second images from the initial video, and acquire target audio corresponding to the initial video;

[0144] extracting audio features from the target audio, extracting first image features from each of the first images, and extracting second image features from each of the second images;

[0145] performing an alignment operation on the first image feature and the second image feature to obtain an aligned second image feature;

[0146] performing spatial deformation processing on the second image feature according to the audio feature and the aligned second image feature to obtain a deformed second image feature;

[0147] generating a lip shape image corresponding to each first image according to the deformed second image feature and the first image feature;

[0148] A target video is generated according to each of the lip-sync images and the target audio.

[0149] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by processor 401. Processor 401 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in processor 401 or by software instructions. The above processor 401 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present invention can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software units in the decoding processor. The software units can be located in storage media well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 402 , and the processor 401 reads the information in the memory 402 and completes the steps of the above method in combination with its hardware.

[0150] It is understood that the embodiments described herein may be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSP devices, DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or a combination thereof.

[0151] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0152] The computer device provided in this embodiment may be Figure 4 The device shown in , can perform Figure 1-2 All steps of the video generation method in Figure 1-2 For technical effects of the video generation method shown, please refer to Figure 1-2 For the sake of brevity, the relevant description will not be repeated here.

[0153] An embodiment of the present invention further provides a storage medium (computer-readable storage medium). The storage medium stores one or more programs. The storage medium may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; and the memory may also include a combination of the aforementioned types of memory.

[0154] When one or more programs in the storage medium can be executed by one or more processors, the video generation method executed on the device side can be implemented.

[0155] The processor is configured to execute a video generation program stored in the memory to implement the following steps of a video generation method executed on the device side:

[0156] Filling the mouth area of the character in each frame of the initial video to obtain a first image corresponding to each processed frame;

[0157] Acquire a preset number of second images from the initial video, and acquire target audio corresponding to the initial video;

[0158] extracting audio features from the target audio, extracting first image features from each of the first images, and extracting second image features from each of the second images;

[0159] performing an alignment operation on the first image feature and the second image feature to obtain an aligned second image feature;

[0160] performing spatial deformation processing on the second image feature according to the audio feature and the aligned second image feature to obtain a deformed second image feature;

[0161] generating a lip shape image corresponding to each first image according to the deformed second image feature and the first image feature;

[0162] A target video is generated according to each of the lip-sync images and the target audio.

[0163] Professionals should also be further aware that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0164] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0165] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A video generation method, characterized in that: include: Filling the mouth area of the character in each frame of the initial video to obtain a first image corresponding to each processed frame; Acquire a preset number of second images from the initial video, and acquire target audio corresponding to the initial video; extracting audio features from the target audio, extracting first image features from each of the first images, and extracting second image features from each of the second images; performing an alignment operation on the first image feature and the second image feature to obtain an aligned second image feature; performing spatial deformation processing on the second image feature according to the audio feature and the aligned second image feature to obtain a deformed second image feature; generating a lip shape image corresponding to each first image according to the deformed second image feature and the first image feature; A target video is generated according to each of the lip-sync images and the target audio.

2. The method according to claim 1, characterized in that The extracting audio features from the target audio, extracting first image features from each of the first images, and extracting second image features from each of the second images includes: Extracting the target audio context window through a trained audio feature extraction network to obtain initial audio features of a preset duration corresponding to each first image; The audio feature is extracted for each of the initial audio features by a convolutional neural network, the first image feature is extracted for each of the first images by a convolutional neural network, and the second image feature is extracted for each of the second images by a convolutional neural network.

3. The method according to claim 1, characterized in that The aligning the first image feature and the second image feature to obtain the aligned second image feature includes: Calculating a similarity matrix between the plurality of first image features and the plurality of second image features through an attention mechanism, and performing normalization processing to obtain a plurality of attention weights; Perform a weighted summation on each of the attention weights and the corresponding second image feature to generate the aligned second image feature that is aligned with the feature space of the first image feature.

4. The method according to claim 1, wherein The performing spatial deformation processing on the second image feature according to the audio feature and the aligned second image feature to obtain the deformed second image feature includes: Copying and expanding the audio feature to the same spatial dimension as the aligned second image feature to obtain an expanded audio feature; splicing the aligned second image features and the expanded audio features in the channel dimension, and predicting the offset of the variable convolution through a convolution layer; A variable convolution operation is performed on the second image feature using the offset to obtain the deformed second image feature.

5. The method according to claim 1, wherein Generating a lip shape image corresponding to each first image according to the deformed second image feature and the first image feature includes: splicing the deformed second image feature and the first image feature in a channel dimension; The convolutional network is transposed and upsampled step by step. After upsampling at each layer, the features of the corresponding layer of the encoder are introduced through a jump connection. The last layer uses a target activation function to output an image in the target range to obtain the lip shape image.

6. The method according to claim 1, characterized in that The acquiring a preset number of second images from the initial video includes: Identifying a target person contained in each frame of the initial video; Recognize the mouth shape of the target person in each frame of the image; A preset number of images of the target person with different lip shapes are obtained from the initial video to obtain the second image.

7. The method according to claim 1, characterized in that After obtaining a preset number of second images from the initial video, the method further includes: performing face detection and alignment processing on each of the first images and each of the second images; The processed first image and the second image are normalized into images of a preset size.

8. A video generating device, characterized in that: include: A first processing module is used to perform filling processing on the mouth area of the character in each frame image of the initial video to obtain a first image corresponding to each processed frame image; an acquisition module, configured to acquire a preset number of second images from the initial video, and acquire target audio corresponding to the initial video; an extraction module, configured to extract audio features from the target audio, extract first image features from each of the first images, and extract second image features from each of the second images; a second processing module, configured to perform an alignment operation on the first image feature and the second image feature to obtain an aligned second image feature; a third processing module, configured to perform spatial deformation processing on the second image feature according to the audio feature and the aligned second image feature to obtain a deformed second image feature; A first generating module, configured to generate a lip shape image corresponding to each first image according to the deformed second image feature and the first image feature; The second generating module is used to generate a target video according to each of the lip-shaped images and the target audio.

9. A computer device, characterized in that: include: A processor and a memory, wherein the processor is configured to execute a program stored in the memory to implement the video generation method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the video generation method according to any one of claims 1 to 7.

Images