Video generation method, device, electronic device and storage medium

Abstract

The present disclosure provides a video generation method and device, including: obtaining an initial video output by an expression transfer model; obtaining a face rendering image and expression coefficient corresponding to a video frame in the initial video, wherein the oral region in the face rendering image is drawn with virtual teeth; inputting the face rendering image and expression coefficient corresponding to the video frame into a target image generation model respectively, obtaining a face processing image output by the target image generation model; combining the face processing images corresponding to each of the video frames in the order of the playback time of the expression-driven video, obtaining a face-processed target video. The present disclosure can improve the real and natural degree of facial texture and tooth texture on the basis of ensuring the integrity of the teeth of the target video, thereby greatly improving the quality of the target video.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and in particular, to a video generation method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] With the continuous development of computer technology, various video and image special effects are being used more and more widely, and people's demand for the diversity of special effects is also increasing.

[0003] At present, live photo video is a very personalized and interesting special effects application. Users can input a portrait photo and after selecting a driving video with expression changes, they can process the portrait photo into a live photo video, so that the live photo video has the user's facial features integrated with the expression change effects in the driving video.

[0004] However, in the current scheme, the generated live photo videos often have problems such as incomplete faces, especially teeth, and unclear textures, which greatly affects the quality of the live photo videos. Summary of the invention

[0005] The embodiments of the present disclosure provide a video generation method, device, electronic device, computer-readable storage medium and computer program product to solve the problem that the video generated in the related art often has incomplete facial parts, especially teeth parts, and unclear textures.

[0006] In a first aspect, an embodiment of the present disclosure provides a video generation method, the method comprising:

[0007] Inputting a facial image and an expression-driven video containing expression change features into an expression transfer model to obtain an initial video output by the expression transfer model, wherein the initial video is a video generated by fusing the expression change features in the expression-driven video with the facial features of the facial image;

[0008] Acquire a facial rendering image and an expression coefficient corresponding to a video frame in the initial video, wherein an oral cavity region in the facial rendering image is drawn with virtual teeth;

[0009] Inputting the facial rendering image and expression coefficient corresponding to each of the video frames into a target image generation model respectively, and obtaining a facial processing image corresponding to each of the video frames output by the target image generation model; the target image generation model is used to perform teeth and texture-based processing on the facial area of ​​the facial rendering image according to the expression coefficient;

[0010] The face-processed images corresponding to the video frames are combined in the order of the playback time of the expression-driven video to obtain a target video after face processing.

[0011] In an optional implementation manner, the acquiring of the facial rendering image and the expression coefficient corresponding to the video frame in the initial video includes:

[0012] Identifying facial key points and dental key points of video frames in the initial video;

[0013] According to the facial key points and tooth key points of each video frame, a facial rendering image and expression coefficient corresponding to each video frame are constructed.

[0014] In an optional implementation, constructing a facial rendering image and expression coefficient corresponding to each of the video frames based on the facial key points and tooth key points of each of the video frames includes:

[0015] Performing three-dimensional reconstruction according to facial key points of each of the video frames to obtain three-dimensional facial modeling and expression coefficients corresponding to the video frames;

[0016] Projecting the three-dimensional facial modeling onto a two-dimensional plane to obtain a facial modeling diagram;

[0017] According to the tooth key points of the video frame, the oral cavity area in the facial modeling image is determined, and virtual teeth are drawn in the oral cavity area to obtain the facial rendering image.

[0018] In an optional implementation, the method further includes:

[0019] Obtaining a training video, wherein the training video is a video having multiple frames of exposed teeth, having facial expression changes in the picture, and having a head movement posture offset in the picture that is less than or equal to a preset threshold;

[0020] According to the training video, training paired data is obtained, wherein the training paired data includes: any video frame of the training video, a facial rendering corresponding to the any video frame, and an expression coefficient of the any video frame;

[0021] The initial image generation model is trained according to the training paired data to obtain a target image generation model.

[0022] In an optional implementation, the acquiring training pairing data according to the training video includes:

[0023] Extracting a first video frame and a second video frame from the training video;

[0024] Obtaining a first three-dimensional facial modeling of the first video frame, and a second three-dimensional facial modeling and expression coefficients of the second video frame, wherein the first three-dimensional facial modeling is a three-dimensional modeling that integrates facial features of the first video frame and expression features of the first video frame, and the second three-dimensional facial modeling is a three-dimensional modeling that integrates facial features of the first video frame and expression features of the second video frame;

[0025] Projecting the first three-dimensional facial modeling and the second three-dimensional facial modeling onto a two-dimensional plane to obtain a facial modeling image of the second video frame;

[0026] Determine an oral cavity region in the facial modeling image of the second video frame, and draw virtual teeth in the oral cavity region to obtain a facial rendering image of the second video frame;

[0027] A correspondence between the second video frame, the face rendering of the second video frame, and the expression coefficient of the second video frame is established to obtain the training paired data.

[0028] In an optional implementation manner, the obtaining of the first three-dimensional face modeling of the first video frame, and the second three-dimensional face modeling and expression coefficient of the second video frame includes:

[0029] Recognize key points of the first video frame and the second video frame respectively to obtain first facial key points of the first video frame and second facial key points of the second video frame;

[0030] Acquire a first expression coefficient and a first posture coefficient of the first video frame according to the first facial key point, and acquire a second expression coefficient of the second video frame according to the second facial key point;

[0031] Performing three-dimensional reconstruction according to the first video frame, the first expression coefficient, and the first posture coefficient to obtain the first three-dimensional face model;

[0032] Perform three-dimensional reconstruction according to the first video frame, the second expression coefficient and the first posture coefficient to obtain the second three-dimensional face modeling.

[0033] In an optional implementation manner, before acquiring the training pairing data according to the training video, the method further includes:

[0034] The video frames of the training video are processed by a preset image super-resolution algorithm.

[0035] In a second aspect, the present disclosure also provides a video generation device, the device comprising:

[0036] A migration module is configured to input a facial image and an expression-driven video containing expression change features into an expression migration model to obtain an initial video output by the expression migration model, wherein the initial video is a video generated by fusing the expression change features in the expression-driven video with the facial features of the facial image;

[0037] A first acquisition module is configured to acquire a facial rendering image and an expression coefficient corresponding to a video frame in the initial video, wherein an oral cavity region in the facial rendering image is drawn with virtual teeth;

[0038] The processing module is configured to input the facial rendering image and the expression coefficient corresponding to each of the video frames into a target image generation model respectively, and obtain a facial processing image corresponding to each of the video frames output by the target image generation model; the target image generation model is used to perform teeth and texture-based processing on the facial area of ​​the facial rendering image according to the expression coefficient;

[0039] The generating module is configured to combine the face-processed images corresponding to the video frames according to the play time sequence of the expression-driven video to obtain a target video after face processing.

[0040] In an optional implementation, the first acquisition module includes:

[0041] A recognition submodule, configured to recognize facial key points and tooth key points of a video frame in the initial video;

[0042] The key point analysis submodule is configured to construct a facial rendering image and expression coefficient corresponding to each of the video frames based on the facial key points and tooth key points of each of the video frames.

[0043] In an optional implementation, the key point analysis submodule includes:

[0044] A reconstruction unit is configured to perform three-dimensional reconstruction according to the facial key points of each video frame to obtain a three-dimensional facial model and expression coefficient corresponding to the video frame;

[0045] A projection unit, configured to project the three-dimensional facial modeling onto a two-dimensional plane to obtain a facial modeling image;

[0046] The teeth rendering unit is configured to determine the oral cavity area in the facial modeling image according to the teeth key points of the video frame, and draw virtual teeth in the oral cavity area to obtain the facial rendering image.

[0047] In an optional embodiment, the device further comprises:

[0048] A second acquisition module is configured to acquire a training video, wherein the training video is a video having multiple frames of exposed teeth, a video with facial expression changes in the picture, and a head movement posture offset in the picture that is less than or equal to a preset threshold;

[0049] A third acquisition module is configured to acquire training paired data according to the training video, wherein the training paired data includes: any video frame of the training video, a facial rendering corresponding to the any video frame, and an expression coefficient of the any video frame;

[0050] The training module is configured to train the initial image generation model according to the training paired data to obtain a target image generation model.

[0051] In an optional implementation, the third acquisition module includes:

[0052] A first acquisition submodule is configured to extract a first video frame and a second video frame from the training video;

[0053] A second acquisition submodule is configured to acquire a first three-dimensional facial modeling of the first video frame, and a second three-dimensional facial modeling and expression coefficients of the second video frame, wherein the first three-dimensional facial modeling is a three-dimensional modeling that integrates facial features of the first video frame and expression features of the first video frame, and the second three-dimensional facial modeling is a three-dimensional modeling that integrates facial features of the first video frame and expression features of the second video frame;

[0054] a projection submodule, configured to project the first three-dimensional facial modeling and the second three-dimensional facial modeling onto a two-dimensional plane to obtain a facial modeling image of the second video frame;

[0055] a drawing submodule, configured to determine an oral cavity region in the facial modeling image of the second video frame, and draw virtual teeth in the oral cavity region to obtain a facial rendering image of the second video frame;

[0056] The establishing submodule is configured to establish a correspondence between the second video frame, the face rendering image of the second video frame, and the expression coefficient of the second video frame to obtain the training paired data.

[0057] In an optional implementation, the second acquisition submodule includes:

[0058] a recognition unit configured to perform key point recognition on the first video frame and the second video frame respectively, to obtain a first facial key point of the first video frame and a second facial key point of the second video frame;

[0059] an acquisition unit configured to acquire a first expression coefficient and a first posture coefficient of the first video frame according to the first facial key point, and to acquire a second expression coefficient of the second video frame according to the second facial key point;

[0060] a first establishing unit, configured to perform three-dimensional reconstruction according to the first video frame, the first expression coefficient and the first posture coefficient to obtain the first three-dimensional face model;

[0061] The second establishing unit is configured to perform three-dimensional reconstruction according to the first video frame, the second expression coefficient and the first posture coefficient to obtain the second three-dimensional face modeling.

[0062] In an optional embodiment, the device further comprises:

[0063] The super-resolution module is configured to process the video frames of the training video using a preset image super-resolution algorithm.

[0064] In a third aspect, an embodiment of the present disclosure further provides an electronic device, comprising a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to achieve the video generation.

[0065] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, which, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the video generation.

[0066] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, including a computer program, which implements the video generation when executed by a processor.

[0067] In the disclosed embodiment, after obtaining the initial video generated by fusing the expression change features in the expression-driven video with the facial features of the facial image, the facial rendering image and the expression coefficient of the video frame of the initial video can be obtained through three-dimensional modeling, and the facial rendering image has complete virtual teeth drawn in it, and the facial rendering image retains the clear and accurate facial texture in the video frame. Further, through the target image generation model, the facial rendering image can be converted into a facial processing image with more realistic facial texture and tooth texture based on the expression coefficient. Finally, all facial processing images are encoded in chronological order to obtain the target video, thereby improving the realism and naturalness of the facial texture and tooth texture while ensuring the integrity of the teeth in the target video, thereby greatly improving the quality of the target video.

[0068] The above description is only an overview of the technical solution of the present disclosure. In order to more clearly understand the technical means of the present disclosure, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present disclosure more obvious and easy to understand, the specific implementation methods of the present disclosure are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present disclosure. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0070] Figure 1 is a flowchart of the steps of a video generation method provided by an embodiment of the present disclosure;

[0071] Figure 2 A flowchart of another method for generating a video according to an embodiment of the present disclosure;

[0072] Figure 3 A block diagram of a video generation device provided by an embodiment of the present disclosure;

[0073] Figure 4 A logical block diagram of an electronic device according to an embodiment of the present disclosure;

[0074] Figure 5 A logical block diagram of an electronic device according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0075] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0076] Figure 1 is a flowchart of a video generation method provided by an embodiment of the present disclosure, such as Figure 1 As shown, the method may include:

[0077] Step 101, inputting a facial image and an expression-driven video containing expression change features into an expression transfer model to obtain an initial video output by the expression transfer model.

[0078] The initial video is a video generated by fusing the expression change features in the expression-driven video with the facial features of the facial image.

[0079] In the disclosed embodiment, the target video can be a live photo video, and the special effect of the target video is a facial feature (including facial features, face shape, skin texture, etc.) based on the target object (person, animal, etc.), through the expression change features contained in the preset driving video template, the facial features of the target object are moved to achieve the effect consistent with the expression change of the driving video template. For example, a user can upload a static portrait photo of himself and select a driving video template with a laughing content. After the two are merged, a target video of the user's own face making a laughing expression is obtained.

[0080] Specifically, the initial realization of the target video may include: the user inputs the prepared face picture and the selected expression-driven video containing expression change features into the expression transfer model, and the expression transfer model may merge the expression change features in the expression-driven video with the facial features of the face picture to obtain an initial video in which the face in the face picture moves according to the expression change mode in the expression-driven video. Among them, the expression-driven video may be preset into multiple types, and may include expression change features such as smiling, laughing, crooked mouth, closed eyes, staring, and frowning.

[0081] Furthermore, the expression transfer model can specifically adopt the first-order motion model for image animation (FOMM), which is based on the combination of generative adversarial networks (GAN) to learn the representation of dense motion fields of driving videos and facial images, and can model local positions such as face, hair, and shoulders. It specifically implements a semi-supervised network. The input during training is a video, which is split into pairs of video frames. The cost function used is the difference between the driving frame and the generated frame, plus a motion identity transformation constraint. The network can be divided into two parts: the motion module and the generation module. The motion module can be divided into two parts, the key point detector and the dense motion field. In addition to detecting the key point position, the key point detector also calculates the local affine transformation matrix plus the first-order derivative of this matrix, and simultaneously estimates the key point transformation parameters of the source image and the driving frame. The parameters are all relative to the virtual reference frame. After giving the key point parameters to the dense motion field, the motion of each pixel can be generated. At the same time, the dense motion field also outputs a mask. Finally, the generation module combines the output of the dense motion field with the source image to generate a new image, that is, the target video is obtained.

[0082] However, the initial video generated by the FOMM model also has certain problems. During the training process of the FOMM model, due to the lack of a large amount of training data with high-definition expression of facial details, the video output by the trained FOMM model has problems such as unclear and incomplete teeth and unrealistic facial texture. The embodiments of the present disclosure study and solve these problems.

[0083] Step 102: Obtain a facial rendering image and an expression coefficient corresponding to a video frame in the initial video, wherein virtual teeth are drawn in the oral cavity area in the facial rendering image.

[0084] In the disclosed embodiment, based on the problem that the initial video output by the expression transfer model has unclear and incomplete teeth and the facial texture is not realistic enough, the initial video can be further post-processed to fill in the tooth area and perform realistic transformation of the tooth texture and facial texture.

[0085] Specifically, there are two requirements for the realistic transformation of the tooth texture and facial texture in the initial video. One is that the teeth of the face in the initial video need to be filled to avoid incomplete teeth. The other is that there needs to be a facial realization model that can learn the process of tooth texture and facial texture from a non-realistic state to a realistic state. When the above two conditions are met, the teeth of the video frames in the initial video can be filled to ensure the integrity of the teeth. After that, the tooth texture and facial texture in the initial video can be processed based on the facial realization model to achieve the purpose of applying realistic tooth texture and facial texture to the initial video.

[0086] In this step, data preparation operations are first required, specifically obtaining the facial rendering image corresponding to the video frame in the initial video and the expression coefficient corresponding to the video frame, wherein the facial rendering image is an image with virtual teeth drawn in the oral region of the facial modeling image of the video frame, and its purpose is to realize the facial reconstruction technology based on the 3D deformable statistical model of the human face (3DMM, 3D MorphableModels), and its purpose is to realize the mapping of the face of the video frame to the three-dimensional facial modeling through the dense alignment (3D faceregistration) between the key points of the face, and then project the three-dimensional facial modeling to the two-dimensional plane to obtain the two-dimensional facial modeling image. In practical applications, although it is difficult for 3DMM to model the edge areas such as hair and neck, the expression transfer model has already processed the edge areas such as hair and neck well in the process of generating the initial video, so this step can only process the face through 3DMM.

[0087] In the two-dimensional facial modeling diagram, you can further draw virtual teeth to fill the teeth to ensure the integrity of the teeth. The advantage is that you can restore clear facial textures through three-dimensional reconstruction and express complete and accurate facial expressions. The expression coefficient is obtained by analyzing the facial key points of the video frame during the three-dimensional reconstruction process, and is used to characterize the intensity of expression changes in the video frame.

[0088] Step 103: input the facial rendering image and expression coefficient corresponding to each of the video frames into a target image generation model respectively, and obtain a facial processing image corresponding to each of the video frames output by the target image generation model.

[0089] The target image generation model is used to perform teeth and texture-based processing on the facial area of ​​the facial rendering according to the expression coefficient.

[0090] In the disclosed embodiment, the target image generation model is an image translation model for improving the realism of teeth and full-face texture. The image translation model is a model trained for paired data. Many problems in image processing, computer graphics, and computer vision scenarios can be regarded as "translating" input images into corresponding output images, that is, image translation means conversion between images in different forms. Therefore, in the scenario of the disclosed embodiment, the purpose of the target image generation model is to convert a facial rendering with virtual teeth drawn in the oral region into a facial processed image with more realistic facial texture and tooth texture.

[0091] Specifically, after obtaining the face rendering image and expression coefficient corresponding to the video frame, the face rendering image and expression coefficient of the video frame can be respectively input into the target face realization model, so that the target face realization model processes the tooth texture and face texture of the face rendering image of the video frame with the expression coefficient as the processing condition, thereby obtaining the purpose of applying the real tooth texture and face texture to the face processing image. Since the face image modeling based on the video frame is realized by 3DMM in step 102, and the virtual teeth are drawn in the face rendering image, in the implementation process of this step, on the basis of ensuring the integrity of the teeth, a face processing image with clearer and more realistic texture can be further obtained, thereby improving the clarity, tooth integrity and authenticity of the target video.

[0092] Step 104: Combine the face-processed images corresponding to the video frames according to the playback time sequence of the expression-driven video to obtain a target video after face processing.

[0093] In the disclosed embodiment, for the obtained multiple frames of face processing images, since each frame of the face processing image corresponds to a frame of video, and the video frame corresponds to a playback time point in the time sequence of the expression-driven video, the multiple frames of face processing images can be combined and encoded in the order of the time sequence of the expression-driven video to obtain the target video after face processing. In the target video obtained at this time, the expression change features in the expression-driven video input by the user are perfectly integrated with the facial features of the face picture input by the user, and the tooth area is complete, and the facial texture and tooth texture are closer to real and natural.

[0094] In summary, a video generation method provided by an embodiment of the present disclosure can obtain a face rendering image and an expression coefficient of a video frame of the initial video through three-dimensional modeling after obtaining an initial video generated by fusing the expression change features in an expression-driven video with the facial features of a facial image. Complete virtual teeth are drawn in the face rendering image, and the face rendering image retains the clear and accurate facial texture in the video frame. Further, through a target image generation model, the face rendering image can be converted into a face processing image with more realistic facial texture and tooth texture based on the expression coefficient. Finally, all the face processing images are encoded in chronological order to obtain a target video. This improves the realism and naturalness of the facial texture and tooth texture while ensuring the integrity of the teeth in the target video, thereby greatly improving the quality of the target video.

[0095] Figure 2 is a flowchart of another video generation method provided by an embodiment of the present disclosure. Figure 2 As shown, the method may include:

[0096] Step 201: input a facial image and an expression-driven video containing expression change features into an expression transfer model to obtain an initial video output by the expression transfer model.

[0097] The initial video is a video generated by fusing the expression change features in the expression-driven video with the facial features of the facial image.

[0098] This step may specifically refer to the above step 101, which will not be described in detail here.

[0099] Step 202: Identify facial key points and tooth key points of the video frame in the initial video.

[0100] In the disclosed embodiment, since 3DMM achieves facial modeling through dense alignment between facial key points, the prerequisite for facial modeling through 3DMM is to first identify the facial key points and tooth key points of the video frame in the initial video, wherein the facial key points are used for facial modeling, and the tooth key points are used to determine the oral area for subsequent drawing of virtual teeth.

[0101] Specifically, the facial key points and tooth key points of each video frame can be identified through a convolutional neural network (CNN) model. A video frame can contain 101 facial key points, and the tooth key points are some of the 101 facial key points.

[0102] Step 203: construct a facial rendering image and expression coefficient corresponding to each of the video frames according to the facial key points and tooth key points of each of the video frames.

[0103] In the disclosed embodiment, the facial reconstruction technology based on 3DMM aims to map the face of the video frame to the three-dimensional facial model through dense alignment between facial key points, and then project the three-dimensional facial model to a two-dimensional plane to obtain a two-dimensional facial modeling image.

[0104] Specifically, the dense alignment algorithm matches the facial image to an optimal 3D facial model. These 3D facial models contain more feature points, thereby realizing the facial alignment of dense key points. This can be achieved through a trained dense alignment model. The model takes the facial image as input, and the 3D facial modeling, expression coefficient and posture coefficient after key point alignment as output. The 3D facial modeling is projected onto a 2D plane to obtain a 2D facial modeling diagram. The expression coefficient is obtained by analyzing the facial key points of the video frame during the 3D reconstruction process, and is used to characterize the intensity of expression changes in the video frame.

[0105] Optionally, step 203 may specifically include:

[0106] Sub-step 2031: Perform three-dimensional reconstruction based on the facial key points of each video frame to obtain a three-dimensional facial model and expression coefficient corresponding to the video frame.

[0107] Sub-step 2032: project the three-dimensional facial modeling onto a two-dimensional plane to obtain a facial modeling image.

[0108] Sub-step 2033: determining the oral cavity region in the facial modeling image according to the tooth key points of the video frame, and drawing virtual teeth in the oral cavity region to obtain the facial rendering image.

[0109] Specifically, the entire process of constructing a facial rendering image and expression coefficient corresponding to a video frame based on facial key points and tooth key points of a video frame is first based on a dense alignment algorithm of 3DMM. By matching key points, the face in the video frame is matched to a three-dimensional facial model, and the expression coefficient is obtained in the process. After projecting the three-dimensional facial model onto a two-dimensional plane, a two-dimensional facial modeling image can be obtained, wherein the expression coefficient can be used as a processing condition for subsequent facial realization, and is used to control the transformation of the facial modeling image into a facial processing image during the image translation process.

[0110] Furthermore, the oral cavity area in the facial modeling image can be determined through the tooth key points of the video frame. Then, a set of virtual teeth can be drawn in the oral cavity area of ​​the facial modeling image through CG (virtual animation) drawing technology to obtain a facial rendering image to ensure the integrity of the teeth. In addition, the facial modeling image obtained by three-dimensional reconstruction can be used to restore clear facial texture, thereby expressing the complete and accurate expression features in the original video frame.

[0111] Step 204: input the facial rendering image and expression coefficient corresponding to each of the video frames into a target image generation model respectively, and obtain a facial processing image corresponding to each of the video frames output by the target image generation model.

[0112] The target image generation model is used to perform teeth and texture-based processing on the facial area of ​​the facial rendering according to the expression coefficient.

[0113] This step may specifically refer to the above step 103, which will not be described in detail here.

[0114] Step 205: Combine the face-processed images corresponding to the video frames according to the playback time sequence of the expression-driven video to obtain a target video after face processing.

[0115] This step may specifically refer to the above step 104, which will not be described in detail here.

[0116] Optionally, the method may further include:

[0117] Step 206: Obtain a training video, wherein the training video is a video having multiple frames of exposed teeth, having facial expression changes in the picture, and having an offset of head movement posture in the picture that is less than or equal to a preset threshold.

[0118] In the embodiment of the present disclosure, for the training process of the target image generation model, the training goal is that the target image generation model can use the input expression coefficient as control information to process the input two-dimensional facial rendering image, thereby obtaining a facial processed image with sufficiently realistic tooth texture and facial texture.

[0119] In order to achieve the above training goals, we first need to prepare training data, which can be training videos. The requirements for collecting training videos are: obtain a large number of videos with head postures that are as still as possible, with only facial expressions changing, and the richer the expression changes, the better, and there are many video frames with missing teeth in the training videos. The head posture is as still as possible, which can be specifically manifested as the amplitude of changes in head posture parameters in the video, such as displacement, pitch angle, yaw angle, roll angle, etc., is less than the preset amplitude.

[0120] Among them, the training goal of the target image generation model is to provide realistic face and tooth texture processing. Learning about changes in head posture is not in the training goal. Therefore, selecting training videos with head postures that are as still as possible helps reduce the interference caused by head movements. In addition, the richer the expression changes in the training videos, the more expression change samples can be provided, which helps to improve the applicability of the model. The presence of more video frames with missing teeth in the training videos helps to assist the model in learning about changes in the authenticity of tooth texture.

[0121] Step 207: Acquire training paired data according to the training video, where the training paired data includes: any video frame of the training video, a facial rendering corresponding to the any video frame, and an expression coefficient of the any video frame.

[0122] Since the training goal of the target image generation model is to take the expression coefficient and the facial rendering as input and the facial processed image as output, the video frames in the training video can be further processed based on the input and output of the model, and a corresponding facial rendering is established for each video frame through 3DMM, and virtual teeth are drawn in the oral area of ​​the facial rendering. In the modeling process, the expression coefficient of the facial area of ​​the video frame is obtained, and finally the training paired data consisting of the video frame, the corresponding facial rendering and the expression coefficient can be established.

[0123] Step 208: Train the initial image generation model according to the training pairing data to obtain a target image generation model.

[0124] During the training process, the facial rendering in the training paired data can be used as input, the corresponding expression coefficient can be used as control information, and the video frame can be used as the annotation value to calculate the loss value together with the output value of the model. Finally, based on the loss value and the preset loss function, the parameters of the initial image generation model can be adjusted. After multiple rounds of iterative training, the target image generation model can be obtained.

[0125] Optionally, step 208 may specifically include:

[0126] Sub-step 2081: extracting a first video frame and a second video frame from the training video.

[0127] In the embodiment of the present disclosure, the first video frame and the second video frame may be two random video frames in a training video.

[0128] Sub-step 2082, obtaining a first three-dimensional facial modeling of the first video frame, and a second three-dimensional facial modeling and expression coefficients of the second video frame, wherein the first three-dimensional facial modeling is a three-dimensional modeling that integrates the facial features of the first video frame and the expression features of the first video frame, and the second three-dimensional facial modeling is a three-dimensional modeling that integrates the facial features of the first video frame and the expression features of the second video frame.

[0129] Specifically, the target image generation model is an image translation model for improving the realism of teeth and full-face textures. The image translation model is a model trained on paired data, and its purpose is to achieve conversion between images in different forms. Therefore, in the scenario of the disclosed embodiment, the purpose of the target image generation model is to convert a face rendering with virtual teeth drawn in the oral region into a face processing image with more realistic face texture and tooth texture. During the entire conversion process, by drawing a face rendering with virtual teeth, the texture of the original image is accurately retained, and the missing teeth in the original image are completed by virtual teeth, so that the converted face processing image has more complete teeth and more realistic face and tooth textures.

[0130] Based on the above ideas, the embodiment of the present disclosure needs to first construct, for the first video frame, a first three-dimensional facial modeling (old modeling) that integrates the facial features of the first video frame and the expression features of the first video frame; for the second video frame, construct a second three-dimensional facial modeling (new modeling) that integrates the facial features of the first video frame and the expression features of the second video frame. The subsequent projection process of obtaining the facial modeling map of the second video frame can be completed by the first three-dimensional facial modeling and the second three-dimensional facial modeling. In the process of obtaining the facial modeling map of the second video frame, the posture coefficient and the expression coefficient of the first video frame can be used as a reference to enhance the correlation between the video frames, thereby achieving the purpose of multi-frame optimization.

[0131] Optionally, sub-step 2082 may specifically include:

[0132] Sub-step A1: performing key point recognition on the first video frame and the second video frame respectively to obtain first facial key points of the first video frame and second facial key points of the second video frame.

[0133] In the disclosed embodiment, the training video is further extracted to construct the training pairing data. First, the first video frame and the second video frame need to be randomly extracted from the training video, and corresponding facial rendering images are respectively established for the first video frame and the second video frame through 3DMM. Since 3DMM achieves facial modeling through dense alignment between facial key points, the premise of facial modeling through 3DMM is to first identify the first facial key point of the first video frame and the second facial key point of the second video frame. At the same time, the first tooth key point of the first video frame and the second tooth key point of the second video frame can also be obtained. Among them, the facial key points are used for facial modeling, and the tooth key points are used to determine the oral cavity area for subsequent drawing of virtual teeth. Specifically, the facial key points and tooth key points of each video frame can be identified by a convolutional neural network model. A video frame can contain 101 facial key points, and the tooth key points are part of the 101 facial key points.

[0134] Sub-step A2: acquiring a first expression coefficient and a first posture coefficient of the first video frame according to the first facial key points, and acquiring a second expression coefficient of the second video frame according to the second facial key points.

[0135] In the disclosed embodiment, the expression coefficient is obtained by analyzing the facial key points of the video frame during the 3D reconstruction process, and is used to characterize the intensity of the expression change in the video frame. The posture coefficient is also obtained by analyzing the facial key points of the video frame during the 3D reconstruction process, and is used to characterize the intensity of the posture change in the video frame.

[0136] Sub-step A3: performing three-dimensional reconstruction according to the first video frame, the first expression coefficient and the first posture coefficient to obtain the first three-dimensional facial model.

[0137] Sub-step A4: performing three-dimensional reconstruction according to the first video frame, the second expression coefficient and the first posture coefficient to obtain the second three-dimensional facial model.

[0138] Sub-step 2083: Project the first three-dimensional facial modeling and the second three-dimensional facial modeling onto a two-dimensional plane to obtain a facial modeling image of the second video frame.

[0139] In the disclosed embodiment, for the first video frame, a first three-dimensional facial modeling (old modeling) that integrates the first expression coefficient and the first posture coefficient of the first video frame can be constructed; for the second video frame, the second expression coefficient and the first posture coefficient can be further integrated on the basis of the first video frame to obtain a second three-dimensional facial modeling (new modeling) based on the face of the first video frame, but the expression features are the expression features of the second video frame. The subsequent projection process of obtaining the facial modeling map of the second video frame can be completed by the first three-dimensional facial modeling and the second three-dimensional facial modeling, and the two-dimensional facial modeling map can be deformed pixel by pixel based on the old and new modeling, so that in the process of obtaining the facial modeling map of the second video frame, the posture coefficient and the expression coefficient of the first video frame can be used as a reference to enhance the correlation between the video frames, thereby achieving the purpose of multi-frame optimization.

[0140] In the disclosed embodiment, since the second three-dimensional facial modeling is a three-dimensional modeling established based on the facial features of the first video frame and the expression features of the second video frame, the facial features in the three-dimensional modeling are subsequently fixed to the facial features of the first video frame and remain unchanged, while the expression features change from the expression features of the first video frame to the expression features of the second video frame, and the two three-dimensional models are projected onto a two-dimensional plane, thereby obtaining a facial modeling diagram close to the second video frame.

[0141] Sub-step 2084: determine the oral cavity region in the facial modeling image of the second video frame, and draw virtual teeth in the oral cavity region to obtain the facial rendering image of the second video frame.

[0142] In the embodiment of the present disclosure, the oral cavity area in the facial modeling image of the second video frame can be determined through the second tooth key point of the second video frame, and the teeth in the facial modeling image of the second video frame can be further completed by drawing virtual teeth, thereby ensuring the integrity of the teeth in the facial modeling image of the second video frame.

[0143] Sub-step 2085: establishing a correspondence between the second video frame, the face rendering of the second video frame, and the expression coefficient of the second video frame to obtain the training paired data.

[0144] In this step, a correspondence is established between each second video frame and the corresponding facial rendering image and expression coefficient, so as to obtain training paired data. Each correspondence in the training paired data can realize one input and output in the training process.

[0145] Optionally, before sub-step 207, the method further includes:

[0146] Sub-step 209: Processing the video frames of the training video using a preset image super-resolution algorithm.

[0147] In the embodiment of the present disclosure, in order to improve the accuracy of the training process, the training video needs to have sufficient clarity, so that the details and changes in the video frames of the training video can be accurately captured during the training process. Therefore, the embodiment of the present disclosure can process the video frames of the training video through a preset image super-resolution algorithm to improve the clarity of the training video. The image super-resolution algorithm is a method of image reconstruction. Specifically, the resolution of the original image can be improved through hardware or software methods. In the software method, algorithm ideas such as deep learning and interpolation reconstruction can be adopted.

[0148] For example, for deep learning methods, the high-resolution training image can be first downgraded according to the degradation model to generate a training model. Then, the image can be divided into blocks according to the correspondence between the low-frequency and high-frequency parts of the high-resolution image. Through certain algorithms, learning is performed to obtain prior knowledge and establish a learning model. Finally, based on the input low-resolution block, the best matching high-frequency block is searched in the established training set. The super-resolution method based on deep learning makes full use of the prior knowledge of the image itself, and can still produce high-frequency details without increasing the number of input image samples, thereby obtaining better restoration results than the reconstruction-based method, and can be better applied to the restoration of various images.

[0149] In summary, a video generation method provided by an embodiment of the present disclosure can obtain a face rendering image and an expression coefficient of a video frame of the initial video through three-dimensional modeling after obtaining an initial video generated by fusing the expression change features in an expression-driven video with the facial features of a facial image. Complete virtual teeth are drawn in the face rendering image, and the face rendering image retains the clear and accurate facial texture in the video frame. Further, through a target image generation model, the face rendering image can be converted into a face processing image with more realistic facial texture and tooth texture based on the expression coefficient. Finally, all the face processing images are encoded in chronological order to obtain a target video. This improves the realism and naturalness of the facial texture and tooth texture while ensuring the integrity of the teeth in the target video, thereby greatly improving the quality of the target video.

[0150] Figure 3 is a block diagram of a video generating device provided by an embodiment of the present disclosure, such as Figure 3 As shown, it includes: a migration module 301, a first acquisition module 302, a processing module 303, and a generation module 304.

[0151] The migration module 301 is configured to input a facial image and an expression-driven video containing expression change features into an expression migration model to obtain an initial video output by the expression migration model, wherein the initial video is a video generated by fusing the expression change features in the expression-driven video with the facial features of the facial image;

[0152] A first acquisition module 302 is configured to acquire a facial rendering image and an expression coefficient corresponding to a video frame in the initial video, wherein an oral cavity region in the facial rendering image is drawn with virtual teeth;

[0153] The processing module 303 is configured to input the facial rendering image and the expression coefficient corresponding to each of the video frames into the target image generation model respectively, and obtain the facial processing image corresponding to each of the video frames output by the target image generation model; the target image generation model is used to perform teeth and texture-based processing on the facial area of ​​the facial rendering image according to the expression coefficient;

[0154] The generation module 304 is configured to combine the face-processed images corresponding to the video frames according to the play time sequence of the expression-driven video to obtain a target video after face processing.

[0155] In an optional implementation, the first acquisition module includes:

[0156] A recognition submodule, configured to recognize facial key points and tooth key points of a video frame in the initial video;

[0157] The key point analysis submodule is configured to construct a facial rendering image and expression coefficient corresponding to each of the video frames based on the facial key points and tooth key points of each of the video frames.

[0158] In an optional implementation, the key point analysis submodule includes:

[0159] A reconstruction unit is configured to perform three-dimensional reconstruction according to the facial key points of each video frame to obtain a three-dimensional facial model and expression coefficient corresponding to the video frame;

[0160] A projection unit, configured to project the three-dimensional facial modeling onto a two-dimensional plane to obtain a facial modeling image;

[0161] The teeth rendering unit is configured to determine the oral cavity area in the facial modeling image according to the teeth key points of the video frame, and draw virtual teeth in the oral cavity area to obtain the facial rendering image.

[0162] In an optional implementation, the device further includes:

[0163] A second acquisition module is configured to acquire a training video, wherein the training video is a video having multiple frames of exposed teeth, a video with facial expression changes in the picture, and a head movement posture offset in the picture that is less than or equal to a preset threshold;

[0164] A third acquisition module is configured to acquire training paired data according to the training video, wherein the training paired data includes: any video frame of the training video, a facial rendering corresponding to the any video frame, and an expression coefficient of the any video frame;

[0165] The training module is configured to train the initial image generation model according to the training paired data to obtain a target image generation model.

[0166] In an optional implementation, the third acquisition module includes:

[0167] A first acquisition submodule is configured to extract a first video frame and a second video frame from the training video;

[0168] A second acquisition submodule is configured to acquire a first three-dimensional facial modeling of the first video frame, and a second three-dimensional facial modeling and expression coefficients of the second video frame, wherein the first three-dimensional facial modeling is a three-dimensional modeling that integrates facial features of the first video frame and expression features of the first video frame, and the second three-dimensional facial modeling is a three-dimensional modeling that integrates facial features of the first video frame and expression features of the second video frame;

[0169] a projection submodule, configured to project the first three-dimensional facial modeling and the second three-dimensional facial modeling onto a two-dimensional plane to obtain a facial modeling image of the second video frame;

[0170] a drawing submodule, configured to determine an oral cavity region in the facial modeling image of the second video frame, and draw virtual teeth in the oral cavity region to obtain a facial rendering image of the second video frame;

[0171] The establishing submodule is configured to establish a correspondence between the second video frame, the face rendering image of the second video frame, and the expression coefficient of the second video frame to obtain the training paired data.

[0172] In an optional implementation, the second acquisition submodule includes:

[0173] a recognition unit configured to perform key point recognition on the first video frame and the second video frame respectively, to obtain a first facial key point of the first video frame and a second facial key point of the second video frame;

[0174] an acquisition unit configured to acquire a first expression coefficient and a first posture coefficient of the first video frame according to the first facial key point, and to acquire a second expression coefficient of the second video frame according to the second facial key point;

[0175] a first establishing unit, configured to perform three-dimensional reconstruction according to the first video frame, the first expression coefficient and the first posture coefficient to obtain the first three-dimensional face model;

[0176] The second establishing unit is configured to perform three-dimensional reconstruction according to the first video frame, the second expression coefficient and the first posture coefficient to obtain the second three-dimensional face modeling.

[0177] In an optional implementation, the device further includes:

[0178] The super-resolution module is configured to process the video frames of the training video using a preset image super-resolution algorithm.

[0179] In summary, a video generating device provided by an embodiment of the present disclosure can, after obtaining an initial video generated by fusing the expression change features in an expression-driven video with the facial features of a facial image, obtain the facial rendering image and expression coefficient of the video frame of the initial video through three-dimensional modeling. Complete virtual teeth are drawn in the facial rendering image, and the facial rendering image retains the clear and accurate facial texture in the video frame. Further, through the target image generation model, the facial rendering image can be converted into a facial processing image with more realistic facial texture and tooth texture based on the expression coefficient. Finally, all facial processing images are encoded in chronological order to obtain the target video, thereby improving the realism and naturalness of the facial texture and tooth texture while ensuring the integrity of the teeth in the target video, thereby greatly improving the quality of the target video.

[0180] Figure 4 is a block diagram of an electronic device 600 according to an exemplary embodiment. For example, the electronic device 600 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0181] Reference Figure 4 , the electronic device 600 may include one or more of the following components: a processing component 602 , a memory 604 , a power component 606 , a multimedia component 608 , an audio component 610 , an input / output (I / O) interface 612 , a sensor component 614 , and a communication component 616 .

[0182] The processing component 602 generally controls the overall operation of the electronic device 600, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 602 may include one or more processors 620 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 602 may include one or more modules to facilitate the interaction between the processing component 602 and other components. For example, the processing component 602 may include a multimedia module to facilitate the interaction between the multimedia component 608 and the processing component 602.

[0183] The memory 604 is used to store various types of data to support the operation of the electronic device 600. Examples of such data include instructions for any application or method operating on the electronic device 600, contact data, phone book data, messages, pictures, multimedia, etc. The memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0184] The power supply component 606 provides power to the various components of the electronic device 600. The power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 600.

[0185] The multimedia component 608 includes a screen that provides an output interface between the electronic device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 608 includes a front camera and / or a rear camera. When the electronic device 600 is in an operating mode, such as a shooting mode or a multimedia mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0186] The audio component 610 is used to output and / or input audio signals. For example, the audio component 610 includes a microphone (MIC), and when the electronic device 600 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is used to receive an external audio signal. The received audio signal can be further stored in the memory 604 or sent via the communication component 616. In some embodiments, the audio component 610 also includes a speaker for outputting audio signals.

[0187] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0188] The sensor assembly 614 includes one or more sensors for providing various aspects of status assessment for the electronic device 600. For example, the sensor assembly 614 can detect the open / closed state of the electronic device 600, the relative positioning of the components, such as the display and keypad of the electronic device 600, and the sensor assembly 614 can also detect the position change of the electronic device 600 or a component of the electronic device 600, the presence or absence of contact between the user and the electronic device 600, the orientation or acceleration / deceleration of the electronic device 600, and the temperature change of the electronic device 600. The sensor assembly 614 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 614 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 614 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0189] The communication component 616 is used to facilitate wired or wireless communication between the electronic device 600 and other devices. The electronic device 600 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 616 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 616 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0190] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to implement a video generation method provided in an embodiment of the present disclosure.

[0191] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, and the instructions can be executed by a processor 620 of an electronic device 600 to perform the above method. For example, the non-transitory storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0192] Figure 5 is a block diagram of an electronic device 700 according to an exemplary embodiment. For example, the electronic device 700 may be provided as a server. Figure 5 The electronic device 700 includes a processing component 722, which further includes one or more processors, and a memory resource represented by a memory 732 for storing instructions that can be executed by the processing component 722, such as an application. The application stored in the memory 732 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 722 is configured to execute instructions to perform a video generation method provided by an embodiment of the present disclosure.

[0193] The electronic device 700 may also include a power supply component 726 configured to perform power management of the electronic device 700, a wired or wireless network interface 750 configured to connect the electronic device 700 to a network, and an input / output (I / O) interface 758. The electronic device 700 may operate based on an operating system stored in the memory 732, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0194] The embodiment of the present disclosure also provides a computer program product, including a computer program, and when the computer program is executed by a processor, the video generating method is implemented.

[0195] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0196] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A video generation method, characterized in that: The method comprises: Inputting a facial image and an expression-driven video containing expression change features into an expression transfer model to obtain an initial video output by the expression transfer model, wherein the initial video is a video generated by fusing the expression change features in the expression-driven video with the facial features of the facial image; Acquire a facial rendering image and an expression coefficient corresponding to a video frame in the initial video, wherein an oral cavity region in the facial rendering image is drawn with virtual teeth; Inputting the facial rendering image and expression coefficient corresponding to each of the video frames into a target image generation model respectively, and obtaining a facial processing image corresponding to each of the video frames output by the target image generation model; the target image generation model is used to perform teeth and texture-based processing on the facial area of ​​the facial rendering image according to the expression coefficient; The face-processed images corresponding to the video frames are combined in the order of the playback time of the expression-driven video to obtain a target video after face processing.

2. The method according to claim 1, characterized in that The acquiring of the facial rendering image and the expression coefficient corresponding to the video frame in the initial video includes: Identifying facial key points and dental key points of video frames in the initial video; According to the facial key points and tooth key points of each video frame, a facial rendering image and expression coefficient corresponding to each video frame are constructed.

3. The method according to claim 2, characterized in that According to the facial key points and tooth key points of each video frame, a facial rendering image and expression coefficient corresponding to each video frame are constructed, including: Performing three-dimensional reconstruction according to facial key points of each of the video frames to obtain three-dimensional facial modeling and expression coefficients corresponding to the video frames; Projecting the three-dimensional facial modeling onto a two-dimensional plane to obtain a facial modeling diagram; According to the tooth key points of the video frame, the oral cavity area in the facial modeling image is determined, and virtual teeth are drawn in the oral cavity area to obtain the facial rendering image.

4. The method according to claim 1, characterized in that: The method further comprises: Obtaining a training video, wherein the training video is a video having multiple frames of exposed teeth, having facial expression changes in the picture, and having a head movement posture offset in the picture that is less than or equal to a preset threshold; According to the training video, training paired data is obtained, wherein the training paired data includes: any video frame of the training video, a facial rendering corresponding to the any video frame, and an expression coefficient of the any video frame; The initial image generation model is trained according to the training paired data to obtain a target image generation model.

5. The method according to claim 4, characterized in that The step of obtaining training pairing data according to the training video includes: Extracting a first video frame and a second video frame from the training video; Obtaining a first three-dimensional facial modeling of the first video frame, and a second three-dimensional facial modeling and expression coefficients of the second video frame, wherein the first three-dimensional facial modeling is a three-dimensional modeling that integrates facial features of the first video frame and expression features of the first video frame, and the second three-dimensional facial modeling is a three-dimensional modeling that integrates facial features of the first video frame and expression features of the second video frame; Projecting the first three-dimensional facial modeling and the second three-dimensional facial modeling onto a two-dimensional plane to obtain a facial modeling image of the second video frame; Determine an oral cavity region in the facial modeling image of the second video frame, and draw virtual teeth in the oral cavity region to obtain a facial rendering image of the second video frame; A correspondence between the second video frame, the face rendering of the second video frame, and the expression coefficient of the second video frame is established to obtain the training paired data.

6. The method according to claim 5, characterized in that The step of obtaining the first three-dimensional face modeling of the first video frame and the second three-dimensional face modeling and expression coefficient of the second video frame includes: Recognize key points of the first video frame and the second video frame respectively to obtain first facial key points of the first video frame and second facial key points of the second video frame; Acquire a first expression coefficient and a first posture coefficient of the first video frame according to the first facial key point, and acquire a second expression coefficient of the second video frame according to the second facial key point; Performing three-dimensional reconstruction according to the first video frame, the first expression coefficient, and the first posture coefficient to obtain the first three-dimensional face model; Perform three-dimensional reconstruction according to the first video frame, the second expression coefficient and the first posture coefficient to obtain the second three-dimensional face modeling.

7. The method according to claim 4, characterized in that Before acquiring the training pairing data according to the training video, the method further includes: The video frames of the training video are processed by a preset image super-resolution algorithm.

8. A video generating device, characterized in that: The device comprises: A migration module is configured to input a facial image and an expression-driven video containing expression change features into an expression migration model to obtain an initial video output by the expression migration model, wherein the initial video is a video generated by fusing the expression change features in the expression-driven video with the facial features of the facial image; A first acquisition module is configured to acquire a facial rendering image and an expression coefficient corresponding to a video frame in the initial video, wherein an oral cavity region in the facial rendering image is drawn with virtual teeth; The processing module is configured to input the facial rendering image and the expression coefficient corresponding to each of the video frames into a target image generation model respectively, and obtain a facial processing image corresponding to each of the video frames output by the target image generation model; the target image generation model is used to perform teeth and texture-based processing on the facial area of ​​the facial rendering image according to the expression coefficient; The generating module is configured to combine the face-processed images corresponding to the video frames according to the play time sequence of the expression-driven video to obtain a target video after face processing.

9. The device according to claim 8, characterized in that The first acquisition module includes: A recognition submodule, configured to recognize facial key points and tooth key points of a video frame in the initial video; The key point analysis submodule is configured to construct a facial rendering image and expression coefficient corresponding to each of the video frames based on the facial key points and tooth key points of each of the video frames.

10. The device according to claim 9, characterized in that The key point analysis submodule includes: A reconstruction unit is configured to perform three-dimensional reconstruction according to the facial key points of each video frame to obtain a three-dimensional facial model and expression coefficient corresponding to the video frame; A projection unit, configured to project the three-dimensional facial modeling onto a two-dimensional plane to obtain a facial modeling image; The teeth rendering unit is configured to determine the oral cavity area in the facial modeling image according to the teeth key points of the video frame, and draw virtual teeth in the oral cavity area to obtain the facial rendering image.

11. The device according to claim 8, characterized in that The device also includes: A second acquisition module is configured to acquire a training video, wherein the training video is a video having multiple frames of exposed teeth, a video with facial expression changes in the picture, and a head movement posture offset in the picture that is less than or equal to a preset threshold; A third acquisition module is configured to acquire training paired data according to the training video, wherein the training paired data includes: any video frame of the training video, a facial rendering corresponding to the any video frame, and an expression coefficient of the any video frame; The training module is configured to train the initial image generation model according to the training paired data to obtain a target image generation model.

12. The device according to claim 11, characterized in that The third acquisition module includes: A first acquisition submodule is configured to extract a first video frame and a second video frame from the training video; A second acquisition submodule is configured to acquire a first three-dimensional facial modeling of the first video frame, and a second three-dimensional facial modeling and expression coefficients of the second video frame, wherein the first three-dimensional facial modeling is a three-dimensional modeling that integrates facial features of the first video frame and expression features of the first video frame, and the second three-dimensional facial modeling is a three-dimensional modeling that integrates facial features of the first video frame and expression features of the second video frame; a projection submodule, configured to project the first three-dimensional facial modeling and the second three-dimensional facial modeling onto a two-dimensional plane to obtain a facial modeling image of the second video frame; a drawing submodule, configured to determine an oral cavity region in the facial modeling image of the second video frame, and draw virtual teeth in the oral cavity region to obtain a facial rendering image of the second video frame; The establishing submodule is configured to establish a correspondence between the second video frame, the face rendering image of the second video frame, and the expression coefficient of the second video frame to obtain the training paired data.

13. The device according to claim 12, characterized in that The second acquisition submodule includes: a recognition unit configured to perform key point recognition on the first video frame and the second video frame respectively, to obtain a first facial key point of the first video frame and a second facial key point of the second video frame; an acquisition unit configured to acquire a first expression coefficient and a first posture coefficient of the first video frame according to the first facial key point, and to acquire a second expression coefficient of the second video frame according to the second facial key point; a first establishing unit, configured to perform three-dimensional reconstruction according to the first video frame, the first expression coefficient and the first posture coefficient to obtain the first three-dimensional face model; The second establishing unit is configured to perform three-dimensional reconstruction according to the first video frame, the second expression coefficient and the first posture coefficient to obtain the second three-dimensional face modeling.

14. The device according to claim 11, characterized in that The device also includes: The super-resolution module is configured to process the video frames of the training video using a preset image super-resolution algorithm.

15. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 7.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images