Video generation model acquisition method and device, computer equipment, storage medium and program product

Abstract

The invention relates to a video generation model acquisition method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring a training sample set for a target face action; executing the following training process by using the training sample set to train a time sequence feature extraction network: inputting a plurality of preorder video frames corresponding to the video samples into the time sequence feature extraction network to obtain time sequence features corresponding to the plurality of preorder video frames; inputting the time sequence features corresponding to the plurality of preorder video frames into a pre-trained image generator to obtain a predicted video frame corresponding to the target video frame; training a time sequence feature extraction network according to the difference between the target video frame and the predicted video frame; and obtaining a video generation model according to the trained time sequence feature extraction network and a pre-trained image generator. By adopting the method, a video with relatively high quality can be generated.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computer equipment, storage medium and computer program product for obtaining a video generation model. Background Art

[0002] Deepfake is a method of content forgery using deep learning technology. It mainly generates content for multimedia content such as images and videos through artificial intelligence algorithms, especially deep neural network models such as generative adversarial networks (GANs).

[0003] Most existing deep fake technologies use generative adversarial networks (GANs) alone to generate facial images or videos. This generation method mainly focuses on learning the spatial features of facial images, such as the shape of facial features, skin texture, etc. However, the facial movements in the videos generated by this generation method are incoherent and contain artifacts, resulting in poor quality of the generated videos.

[0004] Therefore, there is a problem in the conventional technology that the quality of the generated video is poor. Summary of the invention

[0005] Based on this, it is necessary to provide a method, device, computer equipment, computer-readable storage medium and computer program product for obtaining a video generation model that can improve the quality of video generation in response to the above technical problems.

[0006] A method for obtaining a video generation model, comprising:

[0007] Obtain a training sample set for a target facial action; the training sample set includes video samples recording the target facial action; any video sample includes a target video frame and a plurality of preceding video frames arranged before the target video frame;

[0008] Use the training sample set to perform the following training process to train the time series feature extraction network:

[0009] Inputting a plurality of preceding video frames corresponding to the video sample into a time series feature extraction network to obtain time series features corresponding to the plurality of preceding video frames;

[0010] Inputting time series features corresponding to a plurality of preceding video frames into a pre-trained image generator to obtain a predicted video frame corresponding to a target video frame;

[0011] Based on the difference between the target video frame and the predicted video frame, a time series feature extraction network is trained;

[0012] A video generation model is obtained based on the trained time series feature extraction network and the pre-trained image generator; the video generation model is used to generate a predicted video that records the target facial movements; the coherence of the target facial movements recorded in the predicted video meets preset requirements.

[0013] In one embodiment, training a time series feature extraction network based on the difference between the target video frame and the predicted video frame includes:

[0014] Determining loss information for a time series feature extraction network based on a difference between a target video frame and a predicted video frame;

[0015] The optimizer is used to update the network parameters of the time series feature extraction network based on the loss information.

[0016] In one embodiment, obtaining a training sample set for a target facial action includes:

[0017] Obtaining video samples recording target facial movements;

[0018] Convert the video format of each video sample into a preset video format to obtain a converted video corresponding to each video sample;

[0019] According to the motion features of the target facial motion, the video screen of each converted video is cropped to obtain the cropped video corresponding to each video sample;

[0020] A training sample set is generated based on the cropped video corresponding to each video sample.

[0021] In one embodiment, a plurality of preceding video frames corresponding to a video sample are input into a time series feature extraction network to obtain time series features corresponding to the plurality of preceding video frames, including:

[0022] Grayscale processing is performed on a plurality of preceding video frames corresponding to the video sample to obtain a grayscale image sequence corresponding to the plurality of preceding video frames;

[0023] The grayscale image sequences corresponding to the multiple preceding video frames are input into the time series feature extraction network to be trained to obtain the time series features corresponding to the multiple preceding video frames.

[0024] A video generation method, comprising:

[0025] Obtain the original video recording the target person's facial movements;

[0026] The video frame sequence corresponding to the original video is input into the pre-trained video generation model to obtain the predicted video corresponding to the original video; the pre-trained video generation model is generated by a pre-trained time series feature extraction network and a pre-trained image generator; the pre-trained video generation model is obtained by using the video generation model acquisition method as described above; the coherence of the target facial movements recorded in the predicted video meets the preset requirements.

[0027] In one embodiment, a video frame sequence corresponding to an original video is input into a pre-trained video generation model to obtain a predicted video corresponding to the original video, including:

[0028] The video frame sequence corresponding to the original video is input into the pre-trained video generation model, and the time series features of the video frame sequence are extracted through the time series feature extraction network in the pre-trained video generation model;

[0029] The image generator in the pre-trained video generation model generates predicted video frames of the video frame sequence based on the guidance of time series features; the predicted video frames conform to the changing logic of the target face action;

[0030] Add the predicted video frame to the video frame sequence, update the video frame sequence, and return to the step of extracting the time series features of the video frame sequence through the time series feature extraction network in the pre-trained video generation model until the number of frames in the video frame sequence is equal to a preset threshold;

[0031] Based on each predicted video frame in the video frame sequence, a predicted video corresponding to the original video is generated.

[0032] A device for acquiring a video generation model, comprising:

[0033] An acquisition module is used to acquire a training sample set for a target facial action; the training sample set includes video samples recording the target facial action; any video sample includes a target video frame and a plurality of preceding video frames arranged before the target video frame;

[0034] The training module is used to perform the following training process using the training sample set to train the time series feature extraction network:

[0035] Inputting a plurality of preceding video frames corresponding to the video sample into a time series feature extraction network to obtain time series features corresponding to the plurality of preceding video frames;

[0036] Inputting time series features corresponding to a plurality of preceding video frames into a pre-trained image generator to obtain a predicted video frame corresponding to a target video frame;

[0037] Based on the difference between the target video frame and the predicted video frame, a time series feature extraction network is trained;

[0038] A determination module is used to obtain a video generation model based on a trained time series feature extraction network and a pre-trained image generator; the video generation model is used to generate a predicted video recording a target facial action; the coherence of the target facial action recorded in the predicted video meets preset requirements.

[0039] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0040] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0041] A computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0042] The acquisition method, device, computer equipment, storage medium and computer program product of the above-mentioned video generation model obtain a training sample set for the target facial action; the training sample set includes each video sample recording the target facial action; any video sample includes a target video frame, and multiple preceding video frames arranged before the target video frame; the training sample set is used to perform the following training process to train the time series feature extraction network: multiple preceding video frames corresponding to the video sample are input into the time series feature extraction network to obtain the time series features corresponding to the multiple preceding video frames; the time series features corresponding to the multiple preceding video frames are input into the pre-trained image generator to obtain the predicted video frame corresponding to the target video frame; the time series feature extraction network is trained according to the difference between the target video frame and the predicted video frame; according to the trained time series features The video generation model is obtained by extracting the network and the pre-trained image generator; the video generation model is used to generate a predicted video that records the target facial movements; the coherence of the target facial movements recorded in the predicted video meets the preset requirements; in this way, the time series feature extraction network and the image generator can cooperate with each other in the training process, realizing the fusion training of the time series feature extraction network and the image generator. During the training process, since the time series features output by the time series feature extraction network can be directly used as the input of the image generator to guide the image generation process of the image generator, the time series information between video frames is fully utilized. Therefore, the final video generation model can closely combine the time series information and spatial generation capabilities. Different from the simple splicing of the traditional model, the generated video expression is more coherent and has no artifacts, thereby improving the quality of the generated video. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A diagram of an application environment of a method for acquiring a video generation model in an embodiment;

[0045] Figure 2 A schematic diagram of a flow chart of a method for acquiring a video generation model in an embodiment;

[0046] Figure 3 is a schematic diagram of a generative adversarial network in one embodiment;

[0047] Figure 4 A schematic diagram of a video generation model training process in one embodiment;

[0048] Figure 5 A schematic diagram of a flow chart of a video generation method in an embodiment;

[0049] Figure 6 A schematic diagram of a data processing flow of a video generation model in a prediction phase in an embodiment;

[0050] Figure 7 A schematic diagram of a process from a training phase of a video generation model to a prediction phase of a video generation model in one embodiment;

[0051] Figure 8 A schematic flow chart of a method for acquiring a video generation model in another embodiment;

[0052] Fig. 9 It is a structural block diagram of a device for acquiring a video generation model in one embodiment;

[0053] Fig.10 is a structural block diagram of a video generating device in one embodiment;

[0054] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] In the existing field of deep fake technology, there are mainly the following common methods: such as image generation based on traditional GAN, using generative adversarial networks (GAN) alone to generate facial images or videos. This method mainly focuses on learning the spatial features of facial images, such as the shape of facial features, skin texture, etc., but when generating videos, the time series information between frames is not fully utilized, which easily leads to incoherent generated movements and expressions. In addition, deep fake videos can also be generated by simply splicing or combining models. For example, first train a model for processing time series (such as a recurrent neural network RNN) and a generation model (such as GAN) separately, and then combine them together by simple splicing. This simple combination method results in insufficient fusion between the two models, and has limited effect on improving problems such as artifacts and incoherent movements and expressions in deep fake images. When generating complex face action videos, obvious unnatural phenomena still occur.

[0057] In order to solve the above problems, the present application uses a fixed facial action training sample set to train the Transformer network (a deep learning model based on the self-attention mechanism) during the training phase, thereby achieving more accurate prediction capabilities. When training the Transformer network, the output of the image generator is used to determine the loss information of the Transformer network during training. Compared with training the Transformer network and the GAN model separately, it can have better image authenticity performance. At the same time, when training the Transformer network, fixing the parameters of the GAN can avoid the influence of the parameters of the GAN on the training results of the Transformer network during training. In the prediction phase, the present application will combine the Transformer-GAN model to serially generate deep pseudo images (predicted video frames) based on the powerful generation ability of GAN and the global sequence modeling ability of the Transformer network. The Transformer network can better capture the global feature functions in the sequence. From the perspective of model architecture, this application achieves a deep fusion of the Transformer network and GAN. The Transformer network and GAN collaborate with each other during the training process to form a whole. The output of the Transformer network can be directly used as an important input of the GAN generator to guide the generation process. Different from the simple splicing method, it achieves a close combination of time series information and spatial generation capabilities.

[0058] The method for obtaining the video generation model provided in the embodiment of the present application can be applied to Figure 1In the application environment shown, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers. The server 104 obtains a training sample set for the target facial action; the training sample set includes each video sample recording the target facial action; any video sample includes a target video frame, and multiple preceding video frames arranged before the target video frame; the server 104 uses the training sample set to perform the following training process to train the time series feature extraction network: the server 104 inputs multiple preceding video frames corresponding to the video sample into the time series feature extraction network to obtain time series features corresponding to the multiple preceding video frames; the server 104 inputs the time series features corresponding to the multiple preceding video frames into a pre-trained image generator to obtain a predicted video frame corresponding to the target video frame; the server 104 trains the time series feature extraction network according to the difference between the target video frame and the predicted video frame; the server 104 obtains a video generation model according to the trained time series feature extraction network and the pre-trained image generator; the video generation model is used to generate a predicted video recording the target facial action; the coherence of the target facial action recorded in the predicted video meets the preset requirements. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0059] In an exemplary embodiment, Figure 2 As shown, a method for obtaining a video generation model is provided, and the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate the method, which includes the following steps S210 to S230. Among them:

[0060] Step S210, obtaining a training sample set for a target facial action; the training sample set includes video samples recording the target facial action; any video sample includes a target video frame and a plurality of preceding video frames arranged before the target video frame.

[0061] The target facial motion may be a motion corresponding to any expression, for example, a motion corresponding to a blinking expression or a motion corresponding to an opening mouth expression.

[0062] The training sample set includes video samples that record the target facial movements. In practical applications, the original video data used to generate deep fake videos will be collected in advance, and the focus of the collection is on high-quality positive expression data of the task. Let the training set be D, D = {D1, D2, ..., Dn}, where n is the number of categories of different expressions, and each subset contains a single character expression. For example, D1 is a training sample set generated by blinking expression data, and D2 is a training sample set generated by open mouth expression data. If you need to generate a video generation model that can simulate blinking expressions, you need to use the training sample set D1 with high-quality blinking expression video samples for training. If you need to generate a video generation model that can simulate open mouth expressions, you need to use the training sample set D2 with high-quality open mouth expression video samples for training.

[0063] The video sample may be a video with K video frames, the target video frame of the video sample may refer to the n+1th video frame of the video sample, and the preceding video frame refers to the 1st to nth video frames, n+1 <K。

[0064] Optionally, after determining the target facial motion to be generated, the server obtains a high-quality video sample set recording the target facial motion.

[0065] Step S220, using the training sample set to perform the following training process to train the time series feature extraction network:

[0066] Step S222: inputting a plurality of preceding video frames corresponding to the video sample into a time series feature extraction network to obtain time series features corresponding to the plurality of preceding video frames.

[0067] Among them, the time series feature extraction network can refer to a Transformer (a deep learning model based on a self-attention mechanism) network, which can effectively capture long-distance dependencies in sequence data.

[0068] The time series feature may refer to a global feature in a sequence consisting of multiple preceding video frames captured by a time series feature extraction network.

[0069] Optionally, refer to Figure 4 , the server generates a sequence of preceding video frames {V1, V2, …, V n}, the previous video frame sequence {V1, V2, …, V n} Input to the time series feature extraction network Transformer, extract the time series features corresponding to multiple previous video frames, the time series features can be expressed as h Transformer =f Transformer (V1,V2,…,Vn ).

[0070] Step S224, inputting the time series features corresponding to the plurality of preceding video frames into a pre-trained image generator to obtain a predicted video frame corresponding to the target video frame.

[0071] The pre-trained image generator may refer to the generator G in the pre-trained GAN (Generative Adversarial Network) generative adversarial network. Figure 3 As mentioned above, GAN is a deep learning model architecture, which mainly consists of two parts: the generator and the discriminator. The generator takes random noise as input and converts it into output similar to the training data. The discriminator is used to determine whether the input data is real data or generated data generated by the generator. During training, the generator and the discriminator compete with each other. The generator must generate data that can deceive the discriminator, and the discriminator must accurately distinguish between true and false. The generated data generated by the generator and the real data are input into the discriminator together. After the discriminator updates its parameters, the generator updates its own parameters according to the judgment of the discriminator. This is repeated, so that the quality of the data generated by the generator is continuously improved.

[0072] It should be noted that in each embodiment of the present application, the generative adversarial network needs to be pre-trained first, and a suitable network architecture needs to be selected as the generator G, for example, a deep convolutional generator, which can gradually build an image based on a transposed convolutional layer. It is also necessary to build a discriminator, which can adopt a convolutional neural network architecture and increase the number of channels to capture the texture, shape and other features of the image. After completing the model selection of the generator and the discriminator, the processed training sample set is used for training, and the training goal is to minimize the generator loss L G and the discriminator loss L D . Among them, the generator loss is expressed as: The discriminator loss is expressed as: where p z (z) is the probability distribution of noise, p data (x) is the probability distribution of the real data. After the training is completed, the pre-trained generator G is separated from the generative adversarial network, and then the generator G in the pre-trained generative adversarial network is fused with the time series feature extraction network to obtain the video generation model.

[0073] Among them, the predicted video frame corresponding to the target video frame can be a video frame predicted based on multiple previous video frames. For example, the predicted video frame is the n+1th video frame predicted based on the 1st to nth previous video frames. The predicted video frame is a generated video frame, and the target video frame corresponds to the n+1th video frame of the video sample. The target video frame is a real video frame.

[0074] Optionally, refer to Figure 4The server inputs the time series features corresponding to the multiple preceding video frames output by the time series feature extraction network Transformer into the pre-trained image generator G, and the image generator G outputs G(h Transformer ), thereby obtaining the target video frame V n+1 The corresponding predicted video frame

[0075] Step S226, training the time series feature extraction network according to the difference between the target video frame and the predicted video frame.

[0076] Optionally, the server may n+1 With predicted video frame The difference between them is used to determine the loss information, and based on the loss information, the time series feature extraction network is trained.

[0077] Step S230, obtaining a video generation model based on the trained time series feature extraction network and the pre-trained image generator; the video generation model is used to generate a predicted video recording the target facial action; the coherence of the target facial action recorded in the predicted video meets the preset requirements.

[0078] Among them, the video generation model can be a model for generating predicted videos based on a time series feature extraction network and an image generator.

[0079] The predicted video may be a video that accurately simulates the target person's facial movements.

[0080] Optionally, the server generates a video generation model based on a trained time series feature extraction network and a pre-trained image generator. The predicted video generated by the video generation model and recording the target facial movements has high coherence, and the predicted video has higher generation quality than videos generated by traditional methods.

[0081] The video generation model of the present application can give full play to the advantage of Transformer in processing sequence data, regard the face video as a sequence of images with time series characteristics, and through the self-attention mechanism of Transformer, deeply explore the global information association between frames when the face moves, and capture global information such as the transition of expression changes and the coherence of head movements. From the traditional simple image generation to the entire generation process as a sequence modeling problem, the Transformer and GAN can be used to jointly predict the content of the next one or more frames based on the existing frame information. Compared with the traditional method based only on GAN, it can make better use of time series information. For example, when generating a face video where a smile gradually turns into a laugh, the Transformer can model the trend characteristics of expression changes, and the trend characteristics can be subsequently passed to the GAN image generator, making the generated video more natural, reducing artifacts and incoherence, and improving the generation quality.

[0082] In the method for obtaining the above-mentioned video generation model, a training sample set for the target facial action is obtained; the training sample set includes video samples that record the target facial action; any video sample includes a target video frame, and multiple preceding video frames that are arranged before the target video frame; the training sample set is used to perform the following training process to train the time series feature extraction network: multiple preceding video frames corresponding to the video sample are input into the time series feature extraction network to obtain time series features corresponding to the multiple preceding video frames; the time series features corresponding to the multiple preceding video frames are input into a pre-trained image generator to obtain a predicted video frame corresponding to the target video frame; the time series feature extraction network is trained according to the difference between the target video frame and the predicted video frame; and the time series feature extraction network is compared with the pre-trained image generator to obtain the predicted video frame corresponding to the target video frame. The image generator is used to obtain a video generation model; the video generation model is used to generate a predicted video that records the target facial movements; the coherence of the target facial movements recorded in the predicted video meets the preset requirements; in this way, the time series feature extraction network and the image generator can cooperate with each other during the training process, realizing the fusion training of the time series feature extraction network and the image generator. During the training process, since the time series features output by the time series feature extraction network can be directly used as the input of the image generator to guide the image generation process of the image generator, the time series information between video frames is fully utilized. Therefore, the final video generation model can closely combine the time series information and spatial generation capabilities, which is different from the simple splicing of the traditional model. The generated video expression is more coherent and has no artifacts, thereby improving the quality of the generated video.

[0083] In an exemplary embodiment, a time series feature extraction network is trained based on the difference between a target video frame and a predicted video frame, including: determining loss information for the time series feature extraction network based on the difference between the target video frame and the predicted video frame; and using an optimizer to update network parameters of the time series feature extraction network based on the loss information.

[0084] The loss information can be a loss value determined based on the target video frame and the predicted video frame. In practical applications, the loss function of Transformer can be used to calculate the loss value based on the output of the image generator G(h Transformer ) and the real target video frame V n+1 Calculate the loss value.

[0085] The optimizer may be any optimizer that can be used by the time series feature extraction network.

[0086] Optionally, the server may calculate a loss value according to the difference between the target video frame and the predicted video frame, and update the network parameters of the time series feature extraction network based on the loss information according to the selected optimizer.

[0087] In the actual training process, the training strategy can be adjusted as needed, such as adjusting the learning rate, adding regularization terms, etc., to prevent overfitting and improve model performance.

[0088] In this embodiment, the loss information of the time series feature extraction network is determined based on the difference between the target video frame and the predicted video frame; the optimizer is used to update the network parameters of the time series feature extraction network based on the loss information; in this way, the loss information of the time series feature extraction network can be calculated based on the predicted video frame output by the image generator, which is conducive to promoting the close coordination between the time series feature extraction network and the image generator, thereby promoting the fusion of the time series feature extraction network and the image generator.

[0089] In an exemplary embodiment, a training sample set for a target facial action is obtained, including: obtaining each video sample recording the target facial action; converting the video format of each video sample into a preset video format to obtain a converted video corresponding to each video sample; cropping the video screen of each converted video according to the action features of the target facial action to obtain a cropped video corresponding to each video sample; and generating a training sample set based on the cropped video corresponding to each video sample.

[0090] The preset video format may be any common video format. In practical applications, different video samples may have different formats, so all video samples need to be converted into the same video format.

[0091] The motion feature of the target facial motion may refer to the motion position of the target facial motion.

[0092] Optionally, the server obtains video samples recording the target facial movements, and uniformly converts the video formats of the video samples into a preset video format to obtain converted videos corresponding to the video samples, and then determines the key areas of the video screen according to the action features of the target facial movements. In order to determine that the position and proportion of the key areas in the screen are relatively stable, it is necessary to crop the video screen of each converted video to make it have a uniform size, so as to obtain the cropped videos corresponding to the video samples, and then generate a training sample set based on the cropped videos corresponding to the video samples.

[0093] In practical applications, after cropping the video, pixel values ​​need to be normalized to facilitate model training.

[0094] In this embodiment, video samples recording the target facial movements are obtained; the video format of each video sample is converted into a preset video format to obtain a converted video corresponding to each video sample; the video screen of each converted video is cropped according to the action features of the target facial movements to obtain a cropped video corresponding to each video sample; a training sample set is generated based on the cropped video corresponding to each video sample; in this way, each video sample can be preprocessed in a targeted manner, which is beneficial to improving model training efficiency and accuracy.

[0095] In an exemplary embodiment, multiple preceding video frames corresponding to a video sample are input into a time series feature extraction network to obtain time series features corresponding to the multiple preceding video frames, including: grayscale processing is performed on the multiple preceding video frames corresponding to the video sample to obtain a grayscale image sequence corresponding to the multiple preceding video frames; the grayscale image sequence corresponding to the multiple preceding video frames is input into a time series feature extraction network to be trained to obtain time series features corresponding to the multiple preceding video frames.

[0096] The grayscale image sequence corresponding to the plurality of preceding video frames may be a time sequence formed by the grayscale images corresponding to the preceding video frames.

[0097] Optionally, the server performs grayscale processing on multiple preceding video frames corresponding to the video sample to obtain a grayscale image sequence corresponding to the multiple preceding video frames, and then inputs the grayscale image sequence corresponding to the multiple preceding video frames into a time series feature extraction network to be trained to obtain time series features corresponding to the multiple preceding video frames.

[0098] In this embodiment, by graying multiple preceding video frames corresponding to the video sample, a grayscale image sequence corresponding to the multiple preceding video frames is obtained; the grayscale image sequence corresponding to the multiple preceding video frames is input into the time series feature extraction network to be trained to obtain the time series features corresponding to the multiple preceding video frames; in this way, by graying multiple preceding video frames, the amount of calculation can be reduced, which is beneficial to improving the feature extraction efficiency of the time series feature extraction network.

[0099] In an exemplary embodiment, Figure 5 As shown, a video generation method is provided, which is applied to Figure 1 The server 104 in the example is used for explanation, and the steps include the following steps S502 to S504.

[0100] in:

[0101] Step S502, obtaining the original video recording the target facial action.

[0102] Optionally, the server obtains an original video recording the target person's facial movements.

[0103] Step S504, input the video frame sequence corresponding to the original video into the pre-trained video generation model to obtain the predicted video corresponding to the original video; the pre-trained video generation model is generated by a pre-trained time series feature extraction network and a pre-trained image generator; the pre-trained video generation model is obtained by using the video generation model acquisition method as described above; the coherence of the target facial movements recorded in the predicted video meets the preset requirements.

[0104] Optionally, the server inputs a video frame sequence corresponding to the original video into a pre-trained video generation model to obtain a predicted video corresponding to the original video.

[0105] The above-mentioned video generation method obtains an original video recording the target facial movement; inputs the video frame sequence corresponding to the original video into a pre-trained video generation model to obtain a predicted video corresponding to the original video; the pre-trained video generation model is generated by a pre-trained time series feature extraction network and a pre-trained image generator; the pre-trained video generation model is obtained by adopting the video generation model acquisition method as described above; the coherence of the target facial movement recorded in the predicted video meets the preset requirements; in this way, the sequence features of the target facial movement in the original video can be captured through the time series feature extraction network, thereby guiding the image generator to generate a logically consistent predicted video, ensuring the temporal and spatial coherence of the predicted video, and effectively solving the problem of incoherent facial movement in the prior art.

[0106] In an exemplary embodiment, a video frame sequence corresponding to an original video is input into a pre-trained video generation model to obtain a predicted video corresponding to the original video, including: inputting the video frame sequence corresponding to the original video into the pre-trained video generation model, extracting time series features of the video frame sequence through a time series feature extraction network in the pre-trained video generation model; generating predicted video frames of the video frame sequence based on the guidance of the time series features by an image generator in the pre-trained video generation model; the predicted video frames conform to the changing logic of the target facial movements; adding the predicted video frames to the video frame sequence, updating the video frame sequence, and returning to the step of extracting time series features of the video frame sequence through the time series feature extraction network in the pre-trained video generation model until the number of frames in the video frame sequence is equal to a preset threshold; generating a predicted video corresponding to the original video based on each predicted video frame in the video frame sequence.

[0107] Among them, the change logic of the target facial movement can refer to the change pattern of the target facial movement over time. The predicted video conforms to the change logic of the target facial movement, which can be understood as the target facial movement recorded in the predicted video is similar to the real target facial movement, and there will be no unreasonable facial movement.

[0108] The preset threshold may be set to m+j, where m is the number of frames of the original video, m+j is the number of frames of the predicted video, and j is the total number of frames of all predicted video frames.

[0109] Optionally, refer to Figure 6 , the server sends the video frame sequence {U1,U2,…,U m} Input to the pre-trained video generation model, and extract the time series features h of the video frame sequence through the time series feature extraction network Tramsformer in the pre-trained video generation model Transformer =f Transformer (U1,U2,…,U m ), the image generator G in the pre-trained video generation model is guided by the time series features to generate the predicted video frames of the video frame sequence The server will predict the video frame Add to {U1,U2,…,U m}In the video frame sequence, a new video frame sequence is obtained And the new video frame sequence Input to the time series feature extraction network Tramsformer to extract the video frame sequence The image generator G in the pre-trained video generation model generates the predicted video frames of the video frame sequence based on the time series features. Repeat the above generation steps until the generation Assemble in chronological order Get the predicted video corresponding to the original video.

[0110] In this embodiment, the video frame sequence corresponding to the original video is input into a pre-trained video generation model, and the time series features of the video frame sequence are extracted through the time series feature extraction network in the pre-trained video generation model; the image generator in the pre-trained video generation model generates predicted video frames of the video frame sequence based on the guidance of the time series features; the predicted video frames conform to the changing logic of the target facial movements; the predicted video frames are added to the video frame sequence, the video frame sequence is updated, and the step of extracting the time series features of the video frame sequence through the time series feature extraction network in the pre-trained video generation model is returned until the number of frames in the video frame sequence is equal to a preset threshold; based on each predicted video frame in the video frame sequence, a predicted video corresponding to the original video is generated; in this way, the known preceding video frame can be input into the time series feature extraction network to extract the time series features and pass them to the image generator, so that the image generator can generate more coherent video frames according to the time series features, which is conducive to generating a higher quality predicted video.

[0111] In order to facilitate the understanding of those skilled in the art, Figure 7 The whole process from the training stage of the video generation model to the prediction stage of the video generation model is provided, taking the target facial action as the blinking action as an example: in the training stage, first, collect facial videos of blinking actions, then preprocess the collected facial videos to form a training sample set D1, and then use the training sample set D1 to train the generative adversarial network and the Transformer network to obtain a trained video generation model; in the prediction stage, first, input the preprocessed video frame sequence corresponding to the original video into the Transformer network, and use the output of the Transformer network as the input of the image generator, and then the image generator generates a predicted video frame according to the output of the Transformer network, and the generated predicted video frames are combined in time order to form a complete predicted video. After completing the generation of the predicted video for the blinking action, by collecting facial videos of other actions and repeating the above steps, the predicted videos of other actions can be generated.

[0112] In another embodiment, Figure 8 As shown, a method for obtaining a video generation model is provided, and the method is applied to Figure 1 Taking the server 104 in the example as an example, the following steps are included:

[0113] Step S810, obtaining video samples recording target facial movements; the video samples include a target video frame and a plurality of preceding video frames arranged before the target video frame.

[0114] Step S820: convert the video format of each video sample into a preset video format to obtain a converted video corresponding to each video sample.

[0115] Step S830 , cropping the video frames of each converted video according to the motion features of the target facial motion, to obtain cropped videos corresponding to each video sample.

[0116] Step S840, generating a training sample set based on the cropped video corresponding to each video sample;

[0117] Step S850, using the training sample set to perform the following training process to train the time series feature extraction network:

[0118] Step S852, inputting a plurality of preceding video frames corresponding to the video sample into a time series feature extraction network to obtain time series features corresponding to the plurality of preceding video frames;

[0119] Step S854, inputting the time series features corresponding to the plurality of preceding video frames into a pre-trained image generator to obtain a predicted video frame corresponding to the target video frame;

[0120] Step S856, training the time series feature extraction network according to the difference between the target video frame and the predicted video frame;

[0121] Step S860, obtaining a video generation model based on the trained time series feature extraction network and the pre-trained image generator; the video generation model is used to generate a predicted video recording the target facial movement; the coherence of the target facial movement recorded in the predicted video meets the preset requirements.

[0122] It should be noted that the specific limitations of the above steps can refer to the specific limitations of the method for obtaining a video generation model mentioned above.

[0123] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0124] Based on the same inventive concept, the embodiment of the present application also provides a device for acquiring a video generation model for implementing the method for acquiring a video generation model involved above, and a video generation device for implementing the method for acquiring a video generation model involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the device for acquiring one or more video generation models provided below can refer to the limitations of the method for acquiring a video generation model in the above text, and the specific limitations in the embodiments of the device for acquiring one or more video generation models provided below can refer to the limitations of the video generation method in the above text, which will not be repeated here.

[0125] In an exemplary embodiment, Fig. 9 As shown, a device for obtaining a video generation model is provided, including: an acquisition module 902, a training module 904 and a determination module 906, wherein:

[0126] The acquisition module 902 is used to acquire a training sample set for a target facial action; the training sample set includes video samples recording the target facial action; any video sample includes a target video frame and a plurality of preceding video frames arranged before the target video frame;

[0127] The training module 904 is used to perform the following training process using the training sample set to train the time series feature extraction network:

[0128] Inputting a plurality of preceding video frames corresponding to the video sample into a time series feature extraction network to obtain time series features corresponding to the plurality of preceding video frames;

[0129] Inputting time series features corresponding to a plurality of preceding video frames into a pre-trained image generator to obtain a predicted video frame corresponding to a target video frame;

[0130] Based on the difference between the target video frame and the predicted video frame, a time series feature extraction network is trained;

[0131] Determine module 906, which is used to obtain a video generation model based on the trained time series feature extraction network and the pre-trained image generator; the video generation model is used to generate a predicted video that records the target facial movement; the coherence of the target facial movement recorded in the predicted video meets the preset requirements.

[0132] In one embodiment, the training module 904 is specifically used to determine the loss information for the time series feature extraction network based on the difference between the target video frame and the predicted video frame; and use an optimizer to update the network parameters of the time series feature extraction network based on the loss information.

[0133] In one embodiment, the acquisition module 902 is specifically used to obtain each video sample recording the target facial action; convert the video format of each video sample into a preset video format to obtain a converted video corresponding to each video sample; crop the video screen of each converted video according to the action characteristics of the target facial action to obtain a cropped video corresponding to each video sample; and generate a training sample set based on the cropped video corresponding to each video sample.

[0134] In one embodiment, the training module 904 is specifically used to grayscale the multiple preceding video frames corresponding to the video sample to obtain a grayscale image sequence corresponding to the multiple preceding video frames; the grayscale image sequence corresponding to the multiple preceding video frames is input into the time series feature extraction network to be trained to obtain the time series features corresponding to the multiple preceding video frames.

[0135] In an exemplary embodiment, Fig.10 As shown, a video generation device is provided, including: an original video acquisition module 1002 and a predicted video generation module 1004, wherein:

[0136] The original video acquisition module 1002 is used to acquire the original video recording the target face action;

[0137] The predicted video generation module 1004 is used to input the video frame sequence corresponding to the original video into the pre-trained video generation model to obtain the predicted video corresponding to the original video; the pre-trained video generation model is generated by a pre-trained time series feature extraction network and a pre-trained image generator; the pre-trained video generation model is obtained by using the video generation model acquisition method as described above; the coherence of the target facial movements recorded in the predicted video meets the preset requirements.

[0138] In one embodiment, the predicted video generation module 1004 is specifically used to input the video frame sequence corresponding to the original video into a pre-trained video generation model, extract the time series features of the video frame sequence through the time series feature extraction network in the pre-trained video generation model; generate predicted video frames of the video frame sequence based on the guidance of the time series features by the image generator in the pre-trained video generation model; predict the video frames to conform to the changing logic of the target facial movements; add the predicted video frames to the video frame sequence, update the video frame sequence, and return to the step of extracting the time series features of the video frame sequence through the time series feature extraction network in the pre-trained video generation model until the number of frames in the video frame sequence is equal to a preset threshold; based on each predicted video frame in the video frame sequence, generate a predicted video corresponding to the original video.

[0139] The acquisition device of the video generation model and each module in the video generation device can be implemented in whole or in part by software, hardware and a combination thereof. The modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the modules.

[0140] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.11 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store acquisition data of a video generation model. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for acquiring a video generation model is implemented.

[0141] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0142] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the above-mentioned method for obtaining a video generation model and the steps of a video generation method. The steps of the method for obtaining a video generation model here may be the steps of the method for obtaining a video generation model in each of the above-mentioned embodiments, and the steps of the video generation method here may be the steps of the video generation method in each of the above-mentioned embodiments.

[0143] In one embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the processor executes the above-mentioned method for obtaining a video generation model and the steps of a video generation method. The steps of the method for obtaining a video generation model here may be the steps in the method for obtaining a video generation model in each of the above-mentioned embodiments, and the steps of the video generation method here may be the steps in the video generation method in each of the above-mentioned embodiments.

[0144] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the processor executes the above-mentioned method for obtaining a video generation model and the steps of a video generation method. The steps of the method for obtaining a video generation model here may be the steps in the method for obtaining a video generation model in each of the above-mentioned embodiments, and the steps of the video generation method here may be the steps in the video generation method in each of the above-mentioned embodiments.

[0145] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0146] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0147] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for obtaining a video generation model, characterized in that: The method comprises: Acquire a training sample set for a target facial action; the training sample set includes video samples recording the target facial action; any of the video samples includes a target video frame and a plurality of preceding video frames arranged before the target video frame; The following training process is performed using the training sample set to train the time series feature extraction network: Inputting the plurality of preceding video frames corresponding to the video sample into the time series feature extraction network to obtain time series features corresponding to the plurality of preceding video frames; Inputting the time series features corresponding to the multiple preceding video frames into a pre-trained image generator to obtain a predicted video frame corresponding to the target video frame; Training the time series feature extraction network according to the difference between the target video frame and the predicted video frame; A video generation model is obtained based on the trained time series feature extraction network and the pre-trained image generator; the video generation model is used to generate a predicted video recording the target facial movement; the coherence of the target facial movement recorded by the predicted video meets preset requirements.

2. The method according to claim 1, characterized in that The training of the time series feature extraction network according to the difference between the target video frame and the predicted video frame comprises: Determining loss information for the time series feature extraction network according to a difference between the target video frame and the predicted video frame; An optimizer is used to update the network parameters of the time series feature extraction network according to the loss information.

3. The method according to claim 1, characterized in that The step of obtaining a training sample set for a target facial action includes: Acquire each of the video samples recording the target person's facial movements; Convert the video format of each video sample into a preset video format to obtain a converted video corresponding to each video sample; According to the action features of the target facial action, cropping the video frames of each of the converted videos to obtain cropped videos corresponding to each of the video samples; The training sample set is generated based on the cropped videos corresponding to the video samples.

4. The method according to claim 1, characterized in that: The step of inputting the plurality of preceding video frames corresponding to the video sample into a time series feature extraction network to obtain time series features corresponding to the plurality of preceding video frames includes: Performing grayscale processing on the multiple preceding video frames corresponding to the video sample to obtain a grayscale image sequence corresponding to the multiple preceding video frames; The grayscale image sequences corresponding to the multiple preceding video frames are input into a time series feature extraction network to be trained to obtain time series features corresponding to the multiple preceding video frames.

5. A video generation method, characterized in that: The method comprises: Obtain the original video recording the target person's facial movements; The video frame sequence corresponding to the original video is input into a pre-trained video generation model to obtain a predicted video corresponding to the original video; the pre-trained video generation model is generated by a pre-trained time series feature extraction network and a pre-trained image generator; the pre-trained video generation model is obtained by using the video generation model acquisition method as described in any one of claims 1 to 4; the coherence of the target facial movements recorded in the predicted video meets preset requirements.

6. The method according to claim 5, characterized in that The step of inputting the video frame sequence corresponding to the original video into a pre-trained video generation model to obtain a predicted video corresponding to the original video includes: Inputting a video frame sequence corresponding to the original video into a pre-trained video generation model, and extracting time series features of the video frame sequence through a time series feature extraction network in the pre-trained video generation model; Generate a predicted video frame of the video frame sequence based on the guidance of the time series features by an image generator in the pre-trained video generation model; the predicted video frame conforms to the change logic of the target face action; Adding the predicted video frame to the video frame sequence, updating the video frame sequence, and returning to the step of extracting the time series features of the video frame sequence through the time series feature extraction network in the pre-trained video generation model until the number of frames in the video frame sequence is equal to a preset threshold; Based on each of the predicted video frames in the video frame sequence, a predicted video corresponding to the original video is generated.

7. A device for acquiring a video generation model, characterized in that: The device comprises: An acquisition module is used to acquire a training sample set for a target facial action; the training sample set includes video samples recording the target facial action; any of the video samples includes a target video frame and a plurality of preceding video frames arranged before the target video frame; The training module is used to perform the following training process using the training sample set to train the time series feature extraction network: Inputting the plurality of preceding video frames corresponding to the video sample into the time series feature extraction network to obtain time series features corresponding to the plurality of preceding video frames; Inputting the time series features corresponding to the multiple preceding video frames into a pre-trained image generator to obtain a predicted video frame corresponding to the target video frame; Training the time series feature extraction network according to the difference between the target video frame and the predicted video frame; A determination module is used to obtain a video generation model based on a trained time series feature extraction network and the pre-trained image generator; the video generation model is used to generate a predicted video recording the target facial action; the coherence of the target facial action recorded in the predicted video meets preset requirements.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images