Video generation method and apparatus, and electronic device and computer program product

Abstract

The present disclosure belongs to the technical field of image processing. Provided are a video generation method and apparatus, and an electronic device and a computer program product. The video generation method comprises: extracting a facial identity feature of a target virtual human by using a facial recognition model; extracting an audio feature vector and an emotion prompt word from target audio by using an audio classifier; and using the audio feature vector and the emotion prompt word as control features, merging the control features with a diffusion model, using the facial identity feature to control the output of the diffusion model, and using the diffusion model to generate speech video data of the target virtual human, which speech video data matches the target audio. The technical solution of the present disclosure can generate a speech video stream of a virtual human with natural facial expressions.

Description

Video generation method and device, electronic device, and computer program product

[0001] Cross-reference to Related Applications

[0002] This application claims priority to Chinese Patent Application No. 202410985397.8, filed on July 22, 2024, the contents of which are incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present disclosure relates to the technical field of image processing, and particularly refers to a video generation method and device, an electronic device, and a computer program product. BACKGROUND

[0004] With the development of virtual human technology, existing artificial intelligence (AI) algorithms can generate a virtual human speaking video with consistent mouth and voice according to a given audio. However, most of the existing algorithms can only modify the face of the virtual human according to the mouth shape, and the virtual human lacks naturalness of expression. SUMMARY

[0005] The technical problem to be solved by the present disclosure is to provide a video generation method and device, an electronic device, and a computer program product, which can generate a virtual human speaking video stream with natural expression.

[0006] To solve the above technical problem, embodiments of the present disclosure provide the following technical solutions:

[0007] In one aspect, a video generation method is provided, comprising:

[0008] extracting a face identity feature of a target virtual human using a face recognition model;

[0009] extracting an audio feature vector and an emotion prompt word of a target audio using an audio classifier;

[0010] integrating the audio feature vector and the emotion prompt word as control features into a diffusion model, and controlling an output of the diffusion model using the face identity feature, and generating a target virtual human speaking video data matching the target audio using the diffusion model.

[0011] In some embodiments, the extracting a face identity feature of a target virtual human using a face recognition model comprises:

[0012] inputting a face image of the target virtual human into the face recognition model;

[0013] outputting an intermediate layer feature of the face recognition model as the face identity feature.

[0014] In some embodiments, the extracting an emotion prompt word of a target audio using an audio classifier comprises:

[0015] obtaining an emotion classification result of the target audio by using the audio classifier;

[0016] combining the emotion classification result with a preset emotion feature prompt word to obtain the emotion prompt word.

[0017] In some embodiments, the diffusion model includes a self-encoder and a denoising model, the self-encoder includes a decoder, the audio feature vector and the emotion prompt word are integrated into the diffusion model as control features, and the output of the diffusion model is controlled by using the face identity features, and the diffusion model is used to generate target virtual human speech video data matched with the target audio, which includes:

[0018] obtaining noise features subject to a Gaussian distribution;

[0019] integrating the audio feature vector and the emotion prompt word into a first denoising model of the diffusion model through a cross-attention operation to obtain a second denoising model, and denoising the noise features subject to the Gaussian distribution by using the second denoising model to obtain a denoising result;

[0020] controlling the intermediate inference result of the noise of the first denoising model by using the face identity features, and decoding the denoising result by using the decoder to obtain a target virtual human speech image.

[0021] In some embodiments, the self-encoder further includes an encoder, and the obtaining of the noise features subject to the Gaussian distribution includes:

[0022] encoding a target virtual human image into a first feature by using the encoder, and performing a forward noise adding operation on the first feature to obtain noise features subject to the Gaussian distribution; or

[0023] randomly generating noise features subject to the Gaussian distribution.

[0024] In some embodiments, the encoder and the decoder each include a 3D convolution layer, the second denoising model includes a 3D convolution and a time attention layer, and the method specifically includes:

[0025] simultaneously encoding N target virtual human images into N first features by using the encoder, N being an integer greater than 1;

[0026] simultaneously performing a forward noise adding operation on the N first features to obtain N noise features;

[0027] simultaneously denoising the N noise features by using the second denoising model to obtain N denoising results;

[0028] Decode N denoising results simultaneously using the decoder to obtain N target virtual human speech images.

[0029] In some embodiments, the audio feature vector and the emotion prompt word are integrated into a first denoising model of the diffusion model through a cross-attention operation to obtain a second denoising model, and the noise features subject to Gaussian distribution are denoised using the second denoising model to obtain a denoising result, including:

[0030] The audio feature vector is integrated into an intermediate layer of the first denoising model through a cross-attention layer mapping to obtain a second denoising model, and the noise features subject to Gaussian distribution are denoised using the second denoising model to obtain denoising features F1;

[0031] The emotion prompt word is integrated into an intermediate layer of the first denoising model through a cross-attention layer mapping to obtain another second denoising model, and the noise features subject to Gaussian distribution are denoised using the second denoising model to obtain denoising features F2;

[0032] The denoising features F1 and F2 are added as the denoising result.

[0033] In some embodiments, the use of the face identity feature to control the output of the diffusion model includes:

[0034] The face identity feature is used as an input of a controlNet;

[0035] The intermediate inference result of the noise of the diffusion model is controlled using the controlNet.

[0036] In some embodiments, the method further includes:

[0037] The parameters of the controlNet are trained using a first loss function loss:

[0038] wherein z0 represents a first feature without noise, z t represents a first feature with noise at the t-th time step, ∈ θ represents a denoising model, c t represents an audio feature vector, and ε represents a training target.

[0039] In some embodiments, the method further includes:

[0040] The autoencoder is trained using a second loss function Diff Loss:

[0041] wherein, and are reconstruction results of the autoencoder on the i-th input image and on the i+1-th input image, and are ground truths corresponding to the i-th input image and the i+1-th input image.

[0042] In some embodiments, the method further comprises:

[0043] training the autoencoder using a third loss function Optical Loss:

[0044] wherein, is an optical flow from to extracted using a recurrent full affine transformation (RAFT) network, is an optical flow from to extracted using the RAFT network.

[0045] The embodiments of the present disclosure further provide a video generation apparatus, comprising:

[0046] a first extraction module configured to extract a facial identity feature of a target virtual human using a facial recognition model;

[0047] a second extraction module configured to extract an audio feature vector and an emotional cue of a target audio using an audio classifier;

[0048] a generation module configured to integrate the audio feature vector and the emotional cue as a control feature into a diffusion model, to control an output of the diffusion model using the facial identity feature, and to generate a target virtual human speech video data matching the target audio using the diffusion model.

[0049] The embodiments of the present disclosure further provide an electronic device, comprising a processor, a memory, and a program stored in the memory and executable in the processor, wherein the program, when executed by the processor, implements the steps of the video generation method as described above.

[0050] The embodiments of the present disclosure further provide a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program, when executed by a processor, implements the steps of the video generation method as described above.

[0051] The embodiments of the present disclosure further provide a computer program product, comprising computer instructions, and the computer instructions, when executed by a processor, implement the steps of the video generation method as described above.

[0052] Embodiments of the present disclosure have the following beneficial effects:

[0053] In the above scheme, the face identity feature of the target virtual person is extracted by using the face recognition model, and the output of the diffusion model is controlled by using the face identity feature; and the audio feature vector and the emotional prompt word of the target audio are extracted by using the audio classifier, the audio feature vector and the emotional prompt word are integrated into the diffusion model as control features, and the diffusion model is used to generate the target virtual person speaking video data matched with the target audio. By applying the audio feature vector and the emotional prompt word to the diffusion model, the expression and the lip shape of the generated face can be controlled, and a more natural virtual person speaking video stream can be generated. BRIEF DESCRIPTION OF DRAWINGS

[0054] FIG. 1 is a flowchart of a video generation method according to an embodiment of the present disclosure;

[0055] FIG. 2 is a schematic diagram of generating a target virtual person speaking image according to an embodiment of the present disclosure;

[0056] FIG. 3 is a structural schematic diagram of a video generation device according to an embodiment of the present disclosure;

[0057] FIG. 4 is a structural schematic diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0058] In order to make the technical problems, technical solutions and advantages of the embodiments of the present disclosure clearer, the following will be described in detail with reference to the drawings and specific embodiments.

[0059] The embodiments of the present disclosure provide a video generation method and device, an electronic device and a computer program product, which can generate a virtual person speaking video stream with natural expression.

[0060] The embodiments of the present disclosure provide a video generation method, as shown in FIG. 1, which includes:

[0061] Step 101: Extracting the face identity feature of the target virtual person by using a face recognition model;

[0062] In this embodiment, the face recognition model can be trained in advance using multiple sets of training data, each set of training data of the face recognition model including input data and output data, the input data being a face image, and the output data being a face identity feature. The network structure of the face recognition model can be selected from resnet50, resnet101, etc. or an attention module such as a Squeeze-and-Excitation (SE) attention module or a Convolutional Block Attention Module (CBAM) can be inserted into the resnet50, resnet101, etc. network as needed, which can help improve the accuracy of feature extraction by the face recognition model.

[0063] During training of the face recognition model, the loss function of the face recognition model can be selected as ArcFace Loss. The main feature of ArcFace Loss is to introduce an angular interval in the feature space to enhance the discriminability of the features. Specifically, ArcFace Loss inputs the product of the feature vector and the weight matrix into a Softmax function, and then calculates the similarity between the feature vector and the weight matrix through cosine similarity to obtain a normalized representation of the feature vector in the feature space. In this process, ArcFace Loss makes the angle between feature vectors of the same class smaller and the angle between feature vectors of different classes larger, thereby increasing the boundary between classes and improving the robustness and generalization ability of the model.

[0064] After the training of the face recognition model is completed, the intermediate layer features of the face recognition model are output as face identity features. Specifically, the face recognition model can extract a 64*64*4 size face identity feature from a 256*256*3 size face image. The face identity feature can be used as an input of the controlNet, and the controlNet can control the diffusion model to generate an image closer to the user's demand.

[0065] Step 102: extracting an audio feature vector and an emotional prompt word of the target audio using an audio classifier;

[0066] In this embodiment, the audio classifier can be trained in advance using multiple sets of training data, each set of training data including input data and output data, the input data being audio data, and the output data being an audio feature vector of the audio data and an emotion classification result. After the training of the audio classifier is completed, the target audio used to generate the virtual human speech video stream is input into the audio classifier, the audio feature vector of the target audio is obtained through a feature extraction backbone network stacked with multiple residual blocks (res blocks), and then the emotion classification result of the target audio is obtained through a classification head composed of multiple fully connected layers. The audio feature vector is output by the shallow feature layer of the audio classifier, and is used to express the content-related information in the speech signal, and to control the mouth shape of the generated face, such as the word "ah" should be open-mouthed and the word "close" should be closed-mouthed.

[0067] The obtained emotion classification result is then combined with a preset emotion feature prompt word format to obtain the emotion prompt word. For example, the preset emotion feature prompt word format is a "emotion" face, and the emotion classification result of the target audio is happy, and the corresponding emotion prompt word is "a happy face".

[0068] The target audio can correspond to multiple emotion classification results and audio feature vectors. The target audio can be divided into multiple audio segments of the same length, and each audio segment is input into the audio classifier to obtain the corresponding audio feature vector and emotion classification result. The length of the audio segment is determined by the number of frames of the face image processed each time. For example, 5 consecutive frames of face image are processed each time, and the video processing rate is 25 frames per second, and the length of the audio segment is 0.2s.

[0069] Step 103: The audio feature vector and the emotion prompt word are integrated as control features into the diffusion model, and the face identity features are used to control the output of the diffusion model, and the diffusion model is used to generate target virtual human speech video data matching the target audio.

[0070] In this embodiment, a latent diffusion model (Latent Diffusion Model) is used as the diffusion model. The Latent Diffusion Model generates images by iteratively "denoising" data in a latent representation space, and then decodes the representation result into a complete image. The diffusion model includes an autoencoder and a denoising model (which can be a unet denoising model), the autoencoder includes an encoder and a decoder, the encoder can be used to compress the image, then perform diffusion operation on the latent representation space, and finally use the decoder to restore to the original pixel space, this process is called perceptual compression.

[0071] The diffusion model can be explained as an equally weighted sequence of denoising autoencoders ∈ θ (x t , t) ; t = 1…T, whose goal is to predict a corresponding denoised variant from an input x t , or in other words, to predict the noise, where x t is a noisy version of the input x.

[0072] In the latent diffusion model, a pre-trained perceptual compression model is introduced, which includes an encoder and a decoder , so that z t can be obtained by using the encoder during training, so that the model can learn in the latent representation space.

[0073] In addition to unconditional image generation, diffusion models can be used for conditional image generation, mainly by extending a conditional denoising autoencoder ∈ θ (z t , t, y) to achieve this, which can control the image synthesis process through y. Specifically, ∈ θ (z t , t, y) is achieved by adding a cross-attention mechanism to the unet backbone network. In order to be able to preprocess y from multiple different modalities, a domain specific encoder τ θ is used to map y to an intermediate representation , so that various forms of conditions (text, category, layout, etc.) can be easily introduced. The final model can then map the control information into the intermediate layer of the unet through a cross-attention layer, which is implemented as follows:

[0074] where is an intermediate representation of the unet, Q is the intermediate feature of the unet (i.e. the intermediate result obtained by reasoning from the noisy image) obtained by a fully connected layer, K and V are semantic features extracted by the clip model, which are obtained by a fully connected layer, and the control generation is realized through the above formula.

[0075] In this embodiment, the target virtual human image can be encoded into a first feature by the encoder, a forward noise adding operation is performed on the first feature to obtain a noise feature subject to a Gaussian distribution, the noise feature subject to the Gaussian distribution is denoised by using the denoising model to obtain a denoising result, and the denoising result is decoded by using the decoder to obtain a target virtual human speech image. The target virtual human speech image and the target virtual human image are located in the same pixel space. Alternatively, for a trained diffusion model, the encoder can be skipped, and a noise feature subject to a Gaussian distribution can be randomly generated. The randomly generated noise feature is denoised by using the denoising model to obtain a denoising result. The denoising result is decoded by using the decoder to obtain a target virtual human speech image. Specifically, the first feature can be a latent feature.

[0076] The target virtual human speech images obtained through multiple processing are combined together to obtain target virtual human speech video data. For example, N target virtual human speech images are obtained through the above steps 101-103 each time, and N*M target virtual human speech images are obtained after M times of processing. The N*M target virtual human speech images can be used to obtain target virtual human speech video data. N and M are positive integers.

[0077] In a specific example, an input image with a size of 512*512*3 is taken as an example. The encoder encodes it into a latent feature with a size of 64*64*4, and obtains a noise feature subject to a Gaussian distribution through a forward noise adding operation. The unet denoising model denoises the noise feature, and obtains a generated image with the same distribution as the input image through the decoder. For a trained diffusion model, the encoder can be skipped, and a noise feature subject to a Gaussian distribution with a size of 64*64*4 can be directly generated for the denoising and decoding process.

[0078] During the denoising operation of the unet denoising model, the control feature (including the audio feature vector and the emotion prompt word) can be fused into the unet denoising model through a cross attention operation to realize conditional image generation. In addition, the output of the diffusion model is controlled by the controlNet. Specifically, the face identity feature can be taken as the input of the controlNet, the intermediate reasoning result of the noise of the diffusion model is controlled by using the controlNet, and an image conforming to the face identity feature is obtained. The controlNet fixes the parameters of the original unet denoising model, and copies a trainable parameter copy. The control condition is applied to the copy to obtain the final output by adding the result after applying the control to the result of the original model.

[0079] The face identity feature is applied to the denoising model through the controlNet. Specifically, the initialization of the controlNet is completed by copying the part of the denoising model in the original diffusion model, so the structure is completely consistent with the denoising model. The size of the face identity feature is also completely consistent with the size of the noise input to the denoising model, so the face identity feature is input into the controlNet, and the size of the intermediate result of the model inference should be the same as the size of the intermediate result of the noise image in the face identity feature. The intermediate inference result of the face identity feature is added to the intermediate inference result of the noise, that is, the control effect is realized. The specific implementation result is that the face feature of a person is used as the input of the controlNet, and the final generated speaker face is the person.

[0080] The original latent diffusion model takes the first feature of a single image as input and finally outputs a single image; in order to improve the processing efficiency, the first features of multiple images can be input into the diffusion model at a time in the embodiment. For example, the first features of 5 adjacent target virtual human images can be input into the diffusion model to generate 5 target virtual human speaking images at the same time, and the time features between the adjacent frame target virtual human images also need to be used. In the embodiment, 3D convolution and temporal attention layer can be added to the unet denoising model of the diffusion model to describe the time features. When training the diffusion model, the original parameters of the unet denoising model are fixed, and the newly added 3D convolution and temporal attention layer and the controlNet parameters are adjusted.

[0081] Specifically, in the feature extraction module for extracting time features, in the 2D convolution part, the features of the network are arranged in the form of (b*t)*c*h*w, where b is the batch size, t is the time window size (5 in this embodiment), the features are reshaped to the size of b*c*t*h*w, and then input to the 3D convolution for time feature extraction; after obtaining the features with the size of b*c*t*h*w, the features are reshaped back to the size of (b*t)*c*h*w, and weighted superposition is performed with the intermediate layer features before inputting to the 3D convolution (i.e., the classic residual operation). Then, the temporal attention layer is further used to extract the time sequence features, at this time, the features with the size of (b*t)*c*h*w are reshaped to the size of (b*h*w)*t*c, i.e., the self-attention calculation can be performed, and then the features are reshaped back to the size of (b*t)*c*h*w to perform the residual operation, i.e., the full operation process of one feature extraction module is completed. In this embodiment, by setting the time window size t, multiple images can be processed at the same time, for example, when t is 5, 5 images can be processed at the same time.

[0082] In a specific embodiment, taking the processing of 5 images at the same time as an example, as shown in FIG. 2, the video generation method of this embodiment includes the following steps:

[0083] (1) The face recognition model trained in step 101 is used to encode 5 images of the same face (which can be the face image of the target virtual person) into a face feature map with the size of 5*64*64*4, as the input of the controlNet;

[0084] (2) A Gaussian noise with the size of 5*64*64*4 is randomly generated as the input of the unet denoising model in the diffusion model;

[0085] (3) In each denoising time step, the audio feature vector of the target audio (the sliced mel spectrum in FIG. 2) is obtained by using the audio classifier trained in step 102 to perform audio feature extraction on the target audio, and the audio feature vector is superimposed into the unet denoising model including the 3D convolution and the temporal attention layer through the cross attention operation to calculate the denoising feature F1; the emotion prompt word corresponding to the target audio is obtained by using the emotion classification head, the semantic feature vector of the emotion prompt word can be extracted by using the clip model, and the semantic feature vector is superimposed into the unet denoising model through the cross attention operation to calculate the denoising feature F2. F1 and F2 are added as the result of the denoising operation at a certain time step, and then the denoising operation at the next time is continued.

[0086] Taking a diffusion model as an example, the diffusion model includes a self-encoder, a CLIP model, and a UNet denoising model. The CLIP model includes a text encoder and an image encoder, wherein the text encoder can convert an input sentence into a text feature, and inputting an emotional prompt word into the text encoder can obtain a semantic feature vector of the emotional prompt word.

[0087] In this embodiment, a 3D convolution layer needs to be added in the decoder, and if a Gaussian noise is generated by using the encoder, a 3D convolution layer also needs to be added in the encoder, so that the encoder can simultaneously encode multiple target virtual human images into multiple first features; the UNet denoising model including the 3D convolution and the temporal attention layer can simultaneously perform a forward noise adding operation on the multiple first features to obtain multiple noise features; and the decoder including the 3D convolution layer can simultaneously decode the multiple denoising results to obtain multiple target virtual human speech images.

[0088] The original self-encoder is composed of 2D convolution, and for an input image of b*3*h*w, the i-th layer of the network extracts a feature map of b*ci*hi*wi; in this embodiment, all the 2D convolution of the self-encoder is replaced by 3D convolution, and for an input image of b*t*3*h*w, the i-th layer of the network extracts a feature map of b*ti*ci*hi*wi, and finally outputs a target decoding result of b*3*h*w.

[0089] In the original diffusion model, the UNet denoising model includes 12 sub-blocks, of which the first 6 sub-blocks are down-sampling blocks, and the last 6 sub-blocks are up-sampling blocks. In this embodiment, a 3D convolution layer is added from the second down-sampling sub-block, and for the 3D convolution layer, the number of feature channels is consistent with that of the 2D convolution in the original model, and the size of the convolution kernel is designed as 3*3*3. Taking the second sub-block as an example, since the number of 2D convolution channels of the sub-block is 320, the size of the added 3D convolution layer is set as 320*3*3*3, and the convolution method used is padding convolution, that is, the t, h, and w dimensions of the feature map are not changed before and after convolution. The 2D convolution layer can only process a single image, the 3D convolution layer can extract temporal features, and multiple images can be processed simultaneously by using the 3D convolution layer, so that the temporal continuity of the multiple target virtual human speech images is better.

[0090] In this embodiment, the face recognition model is used to extract the face identity features of the target virtual person, and the face identity features are used to control the output of the diffusion model; and the audio classifier is used to extract the audio feature vector and emotional cue of the target audio, and the audio feature vector and emotional cue are used as control features to be integrated into the diffusion model, and the diffusion model is used to generate the target virtual person speaking video data matched with the target audio. By applying the audio feature vector and emotional cue to the diffusion model, the expression and mouth shape of the generated face can be controlled, and a more natural virtual person speaking video stream can be generated.

[0091] Since the unet denoising model outputs the prediction of the noise at the time in the denoising operation at the t-th time step, the parameters of the controlNet need to be trained using the first loss function loss to improve the prediction accuracy of the unet denoising model:

[0092] wherein z0 represents the first feature without noise, z t represents the first feature with noise at the t-th time step, ε θ represents the unet denoising model, c t represents the audio feature vector, and ε represents the training target, and t is a positive integer.

[0093] In this embodiment, the autoencoder of the diffusion model can also be optimized to improve the inter-frame consistency of the synthesized video stream. The function of the autoencoder is to reconstruct the input image into the same image. The training loss of the autoencoder can be composed of GAN Loss and MSE Loss.

[0094] GAN Loss refers to the loss function in the generative adversarial network (GAN), which is used for the adversarial training between the generator and the discriminator. In a standard GAN framework, there are two main parts: the generator and the discriminator.

[0095] The task of the generator (Generator) is to generate fake data samples similar to the real data. The task of the discriminator (Discriminator) is to distinguish between real data and fake data generated by the generator.

[0096] GAN Loss is composed of two parts:

[0097] 1. Generator Loss: The goal of the generator is to generate fake data that is realistic enough to fool the discriminator. Generator loss is the degree to which the fake data generated by the generator is misclassified by the discriminator.

[0098] 2. Discriminator Loss: The goal of the discriminator is to correctly distinguish between real data and fake data generated by the generator. The discriminator loss is the degree of difference between the correct classification of real data and generated fake data by the discriminator.

[0099] Through adversarial training, the generator and discriminator compete with each other during the training process, so that the generator continuously improves the realism of the generated fake data, while the discriminator also continuously improves the ability to distinguish between real data and fake data. The goal of GAN Loss is to make the generator generate realistic fake data, while making the discriminator unable to accurately distinguish between real data and fake data, so as to achieve the purpose of generating high-quality data.

[0100] MSE Loss represents Mean Squared Error Loss, which is a common loss function used to measure the difference between predicted values and actual values. In machine learning and deep learning, MSE Loss is commonly used in regression problems, where the output of the model is a continuous value.

[0101] The calculation method of MSE Loss is to square the difference between the predicted value and the actual value of each sample, and then average the squared differences of all samples. The specific formula is as follows: MSE Loss = 1 / n*Σ(y_pred-y_true)^2

[0102] Where n is the number of samples, y_pred is the predicted value of the model, and y_true is the actual target value. The smaller the MSE Loss, the smaller the difference between the predicted value and the actual value of the model, and the better the performance of the model. In the training process, the optimization algorithm (such as gradient descent) is used to minimize the MSE Loss to adjust the parameters of the model, so that the model can better fit the data and improve the accuracy of prediction.

[0103] In order to improve the inter-frame consistency of the synthesized video stream, the second loss function Diff Loss is also used to train the autoencoder:

[0104] Where, and are the reconstruction results of the i-th frame input image and the i+1-th frame input image by the autoencoder, and The ground truth (GT) corresponding to the i-th frame input image and the i+1-th frame input image, in the case of an autoencoder, the GT can refer to the input data itself, because the goal of an autoencoder is to learn how to reconstruct the input data. By taking the input data as the target output of the autoencoder, the model is required to reconstruct the input data as accurately as possible to minimize the reconstruction error.

[0105] Further, in order to improve the inter-frame consistency of the synthesized video stream, the third loss function Optical Loss can also be used to train the autoencoder:

[0106] wherein, the optical flow from to extracted by a Recurrent All-Pairs Field Transforms (RAFT) network, the optical flow from to extracted by the RAFT network.

[0107] Optical flow is the instantaneous velocity of the pixel movement of a spatial moving object on the observation imaging plane, is a method of finding the corresponding relationship between the previous frame and the current frame by using the change of the pixel in the time domain and the correlation between the adjacent frames, and thus calculating the motion information of the object between the adjacent frames. Generally speaking, optical flow is caused by the movement of the foreground target in the scene itself, the movement of the camera, or the combined movement of the two. Optical flow represents the movement speed and direction of each pixel in the adjacent two frames of images. Optical flow often appears in video or image sequence (multiple images) and is used to depict the instantaneous motion state (motion direction and motion offset) of the moving object (camera or observed object). The representation of optical flow is digital, which generally uses a three-dimensional array ([height, width, 2]) to represent, where height represents the height of the image, that is, the number of rows in the array, width represents the width of the image, that is, the number of columns in the array, and 2 represents x and y directions. In the third dimension of the optical flow array, the first channel (i.e., [height, width, 0]) represents the offset direction and size of the image in the x direction. The x direction here is the horizontal direction, that is, the row vector direction in the image array; the second channel (i.e., [height, width, 1]) represents the offset direction and size of the image in the y direction. The y direction here is the vertical direction, that is, the column vector direction in the image array. The size of the offset is reflected by the size of the value in the optical flow array, and the direction of the offset is reflected by the positive and negative in the optical flow array. In the x direction, a positive value indicates that the object moves to the left, and a negative value indicates that the object moves to the right; in the y direction, a positive value indicates that the object moves upward, and a negative value indicates that the object moves downward. In order to extract optical flow, generally adjacent two frames in the video or adjacent two images in the image sequence are needed, and then the optical flow is extracted through the algorithm.

[0108] In the above formula, "warp" is a computational symbol commonly used in computer vision and image processing. In image processing, warping means spatial transformation of an image, usually achieved by applying a certain transformation matrix or function. In computer vision, warping can be used for image registration, image deformation, optical flow estimation, etc. Through warp operation, the pixels in one image can be mapped to the positions in another image, thereby realizing the alignment or transformation between images. Mathematically, warping can be represented as a function that maps the pixel coordinates in the input image to the new positions in the output image. Common warp operations include affine transformation, perspective transformation, bilinear interpolation, etc., which can be used to realize rotation, scaling, translation, etc. of the image.

[0109] The present embodiment uses the RAFT network to extract the feature map from to Optical flow record as Extract from arrive Optical flow record as Known Can Convert to The estimate, in this embodiment, is constrained to be consistent with... Similar; known Can Convert to The estimate, in this embodiment, is constrained to be consistent with... similar.

[0110] RAFT (Recurrent All-Pairs Field Transforms) is a neural network architecture for optical flow estimation. Optical flow estimation is an important task in computer vision, aiming to predict pixel-level motion between adjacent frames in an image sequence. The RAFT network employs a recurrent neural network (RNN) structure, improving the accuracy of optical flow estimation by modeling the motion between all pixel pairs. The RAFT network leverages a combination of convolutional neural networks (CNNs) and recurrent neural networks (RNNs) to achieve high-quality optical flow estimation by modeling the relationships between image patches.

[0111] In this embodiment, the autoencoder is trained using the four loss functions mentioned above: Diff Loss, Optical Loss, GAN Loss, and MSE Loss. The trained diffusion model can then be applied to the subsequent denoising process. Specifically, training can be stopped when the value of the aforementioned loss function is less than a preset threshold.

[0112] This disclosure also provides a video generation apparatus, as shown in FIG3, comprising:

[0113] The first extraction module 21 is used to extract the facial identity features of the target virtual human using a facial recognition model;

[0114] The second extraction module 22 is used to extract the audio feature vector and emotion cue words of the target audio using an audio classifier;

[0115] The generation module 23 is used to incorporate the audio feature vector and emotion cue words as control features into the diffusion model, and use the facial identity features to control the output of the diffusion model, and use the diffusion model to generate target virtual human speaking video data that matches the target audio.

[0116] In this embodiment, the face identity feature of the target virtual human is extracted by using the face recognition model, and the output of the diffusion model is controlled by using the face identity feature; and the audio feature vector and the emotional prompt word of the target audio are extracted by using the audio classifier, the audio feature vector and the emotional prompt word are integrated into the diffusion model as control features, and the diffusion model is used to generate the target virtual human speaking video data matched with the target audio. By applying the audio feature vector and the emotional prompt word to the diffusion model, the expression and mouth shape of the generated face can be controlled, and a more natural virtual human speaking video stream can be generated.

[0117] In some embodiments, the first extraction module 21 is specifically configured to input the face image of the target virtual human into the face recognition model; and output the intermediate layer features of the face recognition model as the face identity feature.

[0118] In some embodiments, the second extraction module 22 is specifically configured to obtain the emotional classification result of the target audio by using the audio classifier; and combine the emotional classification result with a preset emotional feature prompt word to obtain the emotional prompt word.

[0119] In some embodiments, the diffusion model includes a self-encoder and a denoising model, the self-encoder includes a decoder, and the generation module 23 is specifically configured to obtain noise features subject to a Gaussian distribution; integrate the audio feature vector and the emotional prompt word into a first denoising model of the diffusion model through a cross-attention operation to obtain a second denoising model, denoise the noise features subject to the Gaussian distribution by using the second denoising model to obtain a denoising result; control the intermediate inference result of the noise of the first denoising model by using the face identity feature, and decode the denoising result by using the decoder to obtain a target virtual human speaking image.

[0120] In some embodiments, the self-encoder further includes an encoder, and the generation module 23 is specifically configured to encode a target virtual human image into a first feature by using the encoder, and perform a forward noise adding operation on the first feature to obtain noise features subject to a Gaussian distribution; or, randomly generate noise features subject to a Gaussian distribution.

[0121] In some embodiments, the encoder and the decoder each include a 3D convolution layer, the second denoising model includes a 3D convolution and a time attention layer, and the generation module 23 is specifically configured to simultaneously encode N target virtual human images into N first features by using the encoder, N is an integer greater than 1; simultaneously perform a forward noise adding operation on N first features to obtain N noise features; simultaneously denoise N noise features by using the second denoising model to obtain N denoising results; and simultaneously decode N denoising results by using the decoder to obtain N target virtual human speaking images.

[0122] In some embodiments, the generation module 23 is specifically configured to map the audio feature vector into an intermediate layer of the first denoising model through a cross-attention layer, to obtain a second denoising model, use the second denoising model to denoise the noise feature subject to Gaussian distribution, and obtain a denoised feature F1; map the emotional prompt word into the intermediate layer of the first denoising model through a cross-attention layer, to obtain another second denoising model, use the second denoising model to denoise the noise feature subject to Gaussian distribution, and obtain a denoised feature F2; and add the denoised features F1 and F2 as the denoising result.

[0123] In some embodiments, the generation module 23 is specifically configured to use the face identity feature as an input of a controlNet; and use the controlNet to control an intermediate inference result of the noise of the diffusion model.

[0124] In some embodiments, to improve the prediction accuracy of the denoising model, the device further includes:

[0125] a first training module configured to train parameters of the controlNet using a first loss function loss:

[0126] wherein z0 represents the first feature without noise, z t represents the first feature with noise at the tth time step, ε θ represents a denoising model, c t represents an audio feature vector, and represents a training target.

[0127] In some embodiments, to improve the inter-frame consistency of the synthesized video stream, the device further includes:

[0128] a second training module configured to train the autoencoder using a second loss function Diff Loss:

[0129] wherein, and are reconstruction results of the autoencoder on the i th input image and the i+1 th input image, and are reference real values corresponding to the i th input image and the i+1 th input image.

[0130] In some embodiments, to improve the inter-frame consistency of the synthesized video stream, the device further includes:

[0131] A third training module is configured to train the auto-encoder using a third loss function Optical Loss.

[0132] wherein, is the optical flow from to extracted using the recursive full affine transformation RAFT network, is the optical flow from to extracted using the RAFT network.

[0133] Referring to FIG. 4, the embodiment of the present application further provides an electronic device 30, which comprises a processor 31, a memory 32, and a computer program stored in the memory 32 and capable of running on the processor 31. When the computer program is executed by the processor 31, each process of the above-mentioned video generation method embodiment is implemented, and the same technical effects can be achieved. To avoid repetition, details are not described herein.

[0134] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program. When the computer program is executed by a processor, each process of the above-mentioned video generation method embodiment is implemented, and the same technical effects can be achieved. To avoid repetition, details are not described herein. The computer readable storage medium includes permanent and non-permanent, removable and non-removable media, which can be realized by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of the computer readable storage medium include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage, or any other non-transmission medium that can be used to store information that can be accessed by a computer, and can be used to store information that can be accessed by a computer. According to the definition in this paper, the computer readable storage medium does not include transitory computer readable media, such as modulated data signals and carriers.

[0135] The embodiment of the present application further provides a computer program product, which comprises computer instructions. When the computer instructions are executed by a processor, each process of the above-mentioned method embodiment shown in FIG. 1 is implemented, and the same technical effects can be achieved. To avoid repetition, details are not described herein.

[0136] It should be noted that the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a... " does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0137] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and necessary general hardware platform, of course, also can be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of software product, and the computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a plurality of instructions for making a terminal (which can be a mobile phone, computer, server, air conditioner, or network equipment, etc.) execute the method described in each embodiment of the present application.

[0138] In the method embodiments of the present disclosure, the serial numbers of the steps cannot be used to limit the sequence of the steps, and for those skilled in the art, the sequence of the steps can be changed without creative labor, which is within the protection scope of the present disclosure.

[0139] It should be noted that each of the embodiments in the present disclosure is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment mainly describes the difference from other embodiments. Especially, for the embodiment, since it is basically similar to the product embodiment, the description is relatively simple, and the related parts can be referred to the part of the description of the product embodiment.

[0140] The above is the preferred embodiment of the present disclosure, and it should be pointed out that for those skilled in the art, without departing from the principles of the present disclosure, some improvements and refinements can be made, which should also be regarded as the protection scope of the present disclosure.

Claims

1. A video generation method, characterized in that, include: Use a facial recognition model to extract facial identity features of a target virtual human; Audio feature vectors and sentiment cue words of the target audio are extracted using an audio classifier; The audio feature vector and emotion cue words are incorporated into the diffusion model as control features, and the facial identity features are used to control the output of the diffusion model. The diffusion model is then used to generate video data of a target virtual human speaking that matches the target audio.

2. The video generation method according to claim 1, characterized in that, The extraction of facial identity features of the target virtual human using a facial recognition model includes: The facial image of the target virtual human is input into the facial recognition model; The intermediate layer feature output of the face recognition model is used as the face identity feature.

3. The video generation method according to claim 1, characterized in that, The extraction of sentiment cue words from the target audio using an audio classifier includes: The emotion classification result of the target audio is obtained using the audio classifier; The emotion classification results are combined with preset emotion feature prompts to obtain the emotion prompts.

4. The video generation method according to claim 1, characterized in that, The diffusion model includes an autoencoder and a denoising model. The autoencoder includes a decoder. The process of incorporating the audio feature vector and emotion cue words as control features into the diffusion model, and using the facial identity features to control the output of the diffusion model, and generating target virtual human speaking video data matching the target audio using the diffusion model, includes: Obtain noise features that follow a Gaussian distribution; The audio feature vector and the emotion cue words are integrated into the first denoising model of the diffusion model through cross-attention operation to obtain the second denoising model. The second denoising model is then used to denoise the noise features that follow a Gaussian distribution to obtain the denoising result. The intermediate inference results of the noise in the first denoising model are controlled by the facial identity features, and the denoising results are decoded by the decoder to obtain the target virtual human speaking image.

5. The video generation method according to claim 4, characterized in that, The autoencoder further includes an encoder, and the acquisition of noise features following a Gaussian distribution includes: The encoder is used to encode the target virtual human image into a first feature, and a forward noise addition operation is performed on the first feature to obtain noise features that follow a Gaussian distribution; or Noise features that follow a Gaussian distribution are randomly generated.

6. The video generation method according to claim 5, characterized in that, Both the encoder and the decoder include 3D convolutional layers, the second denoising model includes 3D convolution and a temporal attention layer, and the method specifically includes: The encoder is used to simultaneously encode N images of the target virtual human into N first features, where N is an integer greater than 1; Perform forward noise addition on N of the first features simultaneously to obtain N noise features; The second denoising model is used to denoise the N noise features simultaneously, resulting in N denoising results; The decoder is used to simultaneously decode the N denoising results to obtain N images of the target virtual human speaking.

7. The video generation method according to claim 4, characterized in that, The first denoising model, which integrates the audio feature vector and the emotion cue word into the diffusion model through a cross-attention operation, yields a second denoising model. This second denoising model is then used to denoise the Gaussian-distributed noise features, resulting in the following denoising results: The audio feature vector is incorporated into the intermediate layer of the first denoising model by mapping through a cross-attention layer to obtain a second denoising model. The second denoising model is then used to denoise the noise features that follow a Gaussian distribution to obtain the denoised feature F1. The emotion prompts are incorporated into the intermediate layer of the first denoising model by mapping through a cross-attention layer to obtain another second denoising model. The second denoising model is then used to denoise the noise features that follow a Gaussian distribution to obtain denoised features F2. The denoising feature F1 and the denoising feature F2 are added together to obtain the denoising result.

8. The video generation method according to claim 1, characterized in that, The method of controlling the output of the diffusion model using the facial identity features includes: The facial identity features are used as input to controlNet; The controlNet is used to control the intermediate inference results of the noise in the diffusion model.

9. The video generation method according to claim 8, characterized in that, The method further includes: The parameters of the controlNet are trained using the first loss function: Where z0 represents the first feature without noise, z t ε represents the first noisy feature at time step t. θ Represents the denoising model, c t Let represent the audio feature vector, and ε represent the training target.

10. The video generation method according to claim 4, characterized in that, The method further includes: The autoencoder is trained using the second loss function, Diff Loss: in, and The reconstruction results of the autoencoder for the input image of frame i and the input image of frame i+1 are given. and These are the baseline true values ​​corresponding to the input image of frame i and the input image of frame i+1.

11. The video generation method according to claim 10, characterized in that, The method further includes: The autoencoder is trained using the third loss function, Optical Loss: in, To extract from the recursive all-domain transform RAFT network arrive Optical flow, To extract from RAFT network arrive The optical flow.

12. A video generation apparatus, characterized in that, include: The first extraction module is used to extract the facial identity features of the target virtual human using a facial recognition model; The second extraction module is used to extract the audio feature vector and emotion cue words of the target audio using an audio classifier; The generation module is used to incorporate the audio feature vector and emotion cue words as control features into the diffusion model, and to use the facial identity features to control the output of the diffusion model, thereby generating target virtual human speaking video data that matches the target audio.

13. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the video generation method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the video generation method as described in any one of claims 1 to 11.

15. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the video generation method as described in any one of claims 1 to 11.

Images