Video generation method and device, equipment, storage medium and program product

Abstract

The invention relates to the technical field of video processing, and discloses a video generation method and device, equipment, a storage medium and a program product, and the method comprises the steps: obtaining a target audio, a to-be-processed video and a plurality of images of a target object, and enabling the to-be-processed video to be obtained through the mask processing of a lip region in the video of the target object; generating a driving key point of the target object based on the target audio; screening the plurality of images based on the tooth image quality of the lip region in the plurality of images, and determining a reference image; key point extraction is carried out on a target object in the reference image to obtain reference key points; and based on the reference image, the reference key point and the driving key point, predicting a lip region in the to-be-processed video, and determining a target video corresponding to the target audio. The problem that the driving effect of the voice-driven lip shape is poor can be solved, and the video quality of the speaking video generated by the voice-driven lip shape is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of video processing technology, and in particular to a video generation method, device, equipment, storage medium and program product. Background Art

[0002] At present, the method of driving lip shape based on speech in video images mainly selects a prior image randomly or based on the distance of lip key points; the prior image contains prior information such as the teeth, expression, and action of the target object. Then, the lip shape is driven by the prior image and speech to generate a speaking video corresponding to the speech. However, since the selected prior image may have poor image detail quality in the lips, the lip shape driving effect in the generated speaking video is poor. Summary of the invention

[0003] In view of this, the present disclosure provides a video generation method, apparatus, device, storage medium and program product to solve the problem of poor driving effect of speech-driven lip shape.

[0004] In a first aspect, the present disclosure provides a video generation method, the method comprising:

[0005] Acquire target audio, a video to be processed, and multiple images of a target object, wherein the video to be processed is obtained by performing mask processing on a lip region in a video of the target object;

[0006] generating driving key points of the target object based on the target audio;

[0007] Based on the quality of the teeth images in the lip area of ​​the multiple images, the multiple images are screened to determine a reference image;

[0008] Extracting key points of the target object in the reference image to obtain reference key points;

[0009] Based on the reference image, the reference key points and the driving key points, the lip area in the video to be processed is predicted to determine a target video corresponding to the target audio.

[0010] In a second aspect, the present disclosure provides a video generation device, the device comprising:

[0011] A data acquisition module, used to acquire target audio, a video to be processed, and multiple images of a target object, wherein the video to be processed is obtained by performing mask processing on the lip region in the video of the target object;

[0012] A first operation module, configured to generate a driving key point of the target object based on the target audio;

[0013] An image screening module, used for screening the multiple images based on the quality of the teeth image in the lip area in the multiple images to determine the reference image;

[0014] A second operation module is used to extract key points of the target object in the reference image to obtain reference key points;

[0015] The lip driving module is used to predict the lip area in the video to be processed based on the reference image, the reference key points and the driving key points, and determine the target video corresponding to the target audio.

[0016] In a third aspect, the present disclosure provides an electronic device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the video generation method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0017] In a fourth aspect, the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the video generation method of the first aspect or any corresponding embodiment thereof.

[0018] In a fifth aspect, the present disclosure provides a computer program product, including computer instructions, where the computer instructions are used to enable a computer to execute the video generation method of the first aspect or any corresponding embodiment thereof.

[0019] The video generation method provided by the embodiment of the present disclosure generates driving key points of the target object through target audio. Then, based on the tooth image quality of the lip area in the multiple images of the target object, the multiple images are screened to determine the reference image. Therefore, more tooth details can be provided to enrich the image details of the lips, thereby ensuring the lip shape quality in the reference image. Then, the key points of the target object in the reference image are extracted to obtain the reference key points. When the lip area in the processed video is predicted based on the driving key points corresponding to the target audio, the reference image and the reference key points are used as additional information to guide the prediction of the lip area to determine the target video corresponding to the target audio. Since the tooth image quality of the reference image providing prior information is good, more tooth details can be provided, and the blurring of teeth in the generated target video can be effectively avoided, thereby improving the lip shape driving effect of the voice-driven lip shape. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 is a flowchart of a first video generation method according to an embodiment of the present disclosure;

[0022] Figure 2 is a flow chart of a second video generation method according to an embodiment of the present disclosure;

[0023] Figure 3 is a model schematic diagram of a target key point prediction model according to an embodiment of the present disclosure;

[0024] Figure 4 is a flowchart of a third video generation method according to an embodiment of the present disclosure;

[0025] Figure 5 is a model schematic diagram of a target video generation model and a target optical flow model according to an embodiment of the present disclosure;

[0026] Figure 6 is a schematic diagram of a spatial adaptive normalization layer according to an embodiment of the present disclosure;

[0027] Figure 7 is a schematic diagram of an adaptive instance normalization layer according to an embodiment of the present disclosure;

[0028] Figure 8 is a flowchart of a fourth video generation method according to an embodiment of the present disclosure;

[0029] Fig. 9 is a structural block diagram of a video generating device according to an embodiment of the present disclosure;

[0030] Fig.10 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0032] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0033] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0034] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0035] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0036] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0037] At present, the methods of driving virtual lip shape based on speech are mainly divided into two methods: based on a single image and based on video images. Compared with a single image, the target object in the video has more prior information, such as teeth, expressions, movements, etc. How to use this prior information to generate a target object image consistent with the original video has brought challenges to the method of driving lip shape based on speech based on video images.

[0038] In the related art, the prior image is mainly selected in the video image by random selection or based on the distance of the lip key points. Then, the lip shape is driven by the prior image and the voice to generate a speaking video corresponding to the voice. However, since the selected prior image may have poor image detail quality of the lips, the lip shape quality in the generated speaking video is poor. Moreover, when multiple prior images are selected, the video generation model will choose to refer to different prior images for different video frames, which may cause the problem of tooth flashing in the generated speaking video.

[0039] It can be seen from this that in the related technology of voice-driven lip shape based on video images, the lip shape driving effect is poor.

[0040] In view of this, according to an embodiment of the present disclosure, a video generation method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0041] In this embodiment, a first video generation method is provided, which can be used in the above-mentioned electronic devices, such as mobile phones, tablet computers, etc. Figure 1 is a flow chart of a first video generation method according to an embodiment of the present disclosure, such as Figure 1 As shown, the process includes the following steps:

[0042] Step S101, obtaining target audio, a video to be processed, and multiple images of a target object, wherein the video to be processed is obtained by performing mask processing on the lip region in the video of the target object.

[0043] Specifically, the target audio may be obtained by recording with a recording device. Alternatively, the target audio may be obtained by editing an existing audio with audio editing software. Alternatively, the corresponding target audio may be synthesized with the target text and the target timbre.

[0044] Specifically, the target object may be a virtual portrait or a drawn cartoon character. In addition, the target object may also be other objects with lips and teeth, which are not limited here.

[0045] Specifically, the video of the target object contains facial information of the target object. The video to be processed can be obtained by masking the lip area in the video of the target object, or by masking the lower half of the facial area containing the lip area in the video of the target object.

[0046] Specifically, the multiple images of the target object are images that can represent the image of the target object, and are mainly used to describe the facial information of the target object, such as lip shape, lip size, lipstick color, etc. The multiple images can be video images in a video of the target object, or can be other images of the target object that contain the facial information of the target object, which is not limited here.

[0047] Step S102: generating driving key points of the target object based on the target audio.

[0048] Specifically, the target audio can be preprocessed, such as denoising, filtering, sampling rate adjustment, etc., to obtain preprocessed audio. The preprocessed audio is subjected to speech feature extraction to obtain audio feature data; wherein the audio feature data is data related to lip shape changes carried in the preprocessed audio, and the audio feature data includes at least one of phonemes, pitches, and frequencies. Then, the audio feature data is processed by a pre-trained mapping model from audio features to facial key points to obtain driving key points. The mapping model can adopt a deep learning model such as a generative adversarial model, a sequence-to-sequence model, and a multimodal learning model. Alternatively, according to the principles of phonetics, a mapping relationship is constructed to map audio features to specific facial key point positions, and the driving key points are obtained based on the audio feature data and the constructed mapping relationship.

[0049] Specifically, the driving key points are used to characterize the facial information of the target object under the drive of the target audio. The facial information includes at least one of the position of the facial key area, facial posture, and facial expression. The facial key area includes the area corresponding to at least one of the eyebrows, glasses, nose, lips, and ears.

[0050] Step S103, based on the tooth image quality of the lip area in the multiple images, the multiple images are screened to determine a reference image.

[0051] Specifically, the quality of the tooth image can be determined by at least one of the clarity of the tooth image in the lip area in the multiple images, the contrast between the teeth and other parts, the noise level of the tooth image, and the degree of retention of the tooth details. Alternatively, a pre-constructed image quality assessment model is trained by using multiple sample tooth images and the corresponding sample tooth image qualities, so that the image quality assessment model learns the mapping relationship between the sample tooth images and the sample tooth image qualities, and obtains a target image quality assessment model. Then, the tooth images corresponding to the lip area in the multiple images are input into the target image quality assessment model to evaluate the tooth images, and obtain the tooth image qualities corresponding to the multiple images.

[0052] Specifically, the image with the best tooth image quality among multiple images can be selected as the reference image. Alternatively, the multiple images can be screened based on the lip opening range of the target object in the image and the tooth image quality to determine the reference image. Alternatively, the multiple images can be screened based on the pixel points in the lip area of ​​the image and the tooth image quality to determine the reference image. Alternatively, the multiple images can be screened based on the lip opening range of the target object in the image, the pixel points in the lip area and the tooth image quality to determine the reference image.

[0053] Step S104: extract key points of the target object in the reference image to obtain reference key points.

[0054] Specifically, the target key point detection model can be used to extract key points of the target object in the reference image to obtain reference key points; wherein the target key point detection model is obtained by training a preset key point detection model based on the sample image of the sample object and the corresponding sample key points. Alternatively, the key points of the target object in the reference image can be extracted by facial recognition technology to obtain reference key points. Alternatively, a method based on feature descriptors is used to describe the facial key areas of the target object in the reference image, and the positions of the key points are determined by matching the feature descriptors to obtain reference key points.

[0055] Specifically, the reference key points are used to characterize the facial information of the target object in the reference image, and the facial information includes at least one of the position, facial posture, and facial expression of the facial key areas. The facial key areas include the areas corresponding to at least one of the eyebrows, glasses, nose, lips, and ears.

[0056] Step S105, based on the reference image, the reference key points and the driving key points, the lip area in the video to be processed is predicted to determine the target video corresponding to the target audio.

[0057] Specifically, the driving key points and the video to be processed are input into the target video generation model, and the reference image and the reference key points are used as additional inputs or conditions to guide the target video generation model to predict the lip area in the video to be processed to obtain the target video. Alternatively, the driving key points and the video image of the video to be processed are input into the target image generation model, and the reference image and the reference key points are used as additional inputs or conditions to guide the target image generation model to predict the lip area in the video image of the video to be processed to obtain the target video.

[0058] Optionally, the target video generation model is a key point rendering model, a contour transformation model or a generative adversarial model, and the target image generation model can adopt a generative adversarial model, a diffusion model or an end-to-end image segmentation model based on a convolutional neural network.

[0059] The video generation method provided in this embodiment generates the driving key points of the target object through the target audio. Then, based on the tooth image quality of the lip area in the multiple images of the target object, the multiple images are screened to determine the reference image. Therefore, more tooth details can be provided to enrich the image details of the lips, thereby ensuring the lip shape quality in the reference image. Then, the key points of the target object in the reference image are extracted to obtain the reference key points. When the lip area in the processed video is predicted based on the driving key points corresponding to the target audio, the reference image and the reference key points are used as additional information to guide the prediction of the lip area to determine the target video corresponding to the target audio. Since the tooth image quality of the reference image providing prior information is good, more tooth details can be provided, and the blurring of teeth in the generated target video can be effectively avoided, thereby improving the lip shape driving effect of the voice-driven lip shape.

[0060] In this embodiment, a second video generation method is provided, which can be used in electronic devices such as mobile phones, tablet computers, etc. Figure 2 is a flow chart of a second video generation method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0061] Step S201, obtaining target audio, a video to be processed, and multiple images of a target object, wherein the video to be processed is obtained by performing mask processing on the lip region in the video of the target object. Please refer to the above step S101, which will not be repeated here.

[0062] Step S202: generating driving key points of the target object based on the target audio.

[0063] In some optional implementations, the above step S202 includes: inputting the target audio into a target key point prediction model to perform key point prediction to obtain driving key points, and the target key point prediction model is trained based on sample audio and sample key points of the target object.

[0064] Specifically, the target key point prediction model is obtained by training a preset key point prediction model based on sample audio and sample key points of the target object.

[0065] Furthermore, the target key point prediction model is obtained by training a preset key point prediction model based on the sample audio, the sample key points of the target object, and the sample posture prior key points of the target object.

[0066] Specifically, the target audio is input into the target key point prediction model for key point prediction to obtain driving key points, including: extracting initial key points and posture prior key points in the video of the target object; inputting the target audio, initial key points and posture prior key points into the target key point prediction model for key point prediction to obtain driving key points.

[0067] Specifically, Figure 3 As shown, the target key point prediction model includes a first convolutional neural network, a second convolutional neural network, a third convolutional neural network and a decoder of a transformer model (also known as a Transformer model). The initial key point of the target object is input into the first convolutional neural network for feature extraction to obtain feature data of the initial key point. The target audio is input into the second convolutional neural network for feature extraction to obtain feature data of the target audio. The posture prior key point of the target object is input into the third convolutional neural network for feature extraction to obtain feature data of the posture prior key point. Then, the feature data of the initial key point, the feature data of the target audio and the feature data of the posture prior key point are input into the decoder of the transformer model for decoding to obtain the driving key point.

[0068] Optionally, the first convolutional neural network is a one-dimensional convolutional neural network, the second convolutional neural network is a two-dimensional convolutional neural network, and the third convolutional neural network is a one-dimensional convolutional neural network.

[0069] It should be noted that the driving key points are two-dimensional key points.

[0070] The video generation method provided in this embodiment uses the pre-trained target key point prediction model and the target audio to perform key point prediction and obtain driving key points, thereby improving the generation efficiency of driving key points.

[0071] Step S203, based on the tooth image quality of the lip area in the multiple images, the multiple images are screened to determine a reference image.

[0072] Specifically, the above step S203 includes:

[0073] Step S2031, obtaining the lip opening range of the target object in multiple images.

[0074] Specifically, the lip opening amplitude is also called the lip opening amplitude. The lip opening amplitude is used to characterize the opening size of the lips.

[0075] Specifically, the lip opening range of the target object in each image is determined by a facial feature detection algorithm, or the lip contour is extracted by an edge detection algorithm, and the lip opening range is determined according to the lip contour.

[0076] Step S2032: determine the image in which the lip opening amplitude is greater than a preset amplitude threshold among the multiple images as the first candidate image.

[0077] Specifically, the value of the preset amplitude threshold needs to ensure that the image can clearly show the details of the teeth in the lip area, and the specific value can be selected according to actual conditions.

[0078] Step S2033, determining the quality of the teeth image in the lip area in the first candidate image, and determining the first candidate image with the best quality of the teeth image in the lip area as the reference image.

[0079] Specifically, the tooth image quality of the lip region in the first candidate image is determined by an Image Quality Assessment (IQA) model.

[0080] It should be noted that, when there are multiple first candidate images, the first candidate image with the best tooth image quality in the lip region is determined as the reference image. When there is only one first candidate image, the first candidate image can be directly determined as the reference image.

[0081] The video generation method provided in this embodiment first preliminarily screens multiple images according to the lip opening range to obtain a first candidate image, so as to reduce the number of images that need to determine the tooth image quality. Then, the first candidate image with the best tooth image quality in the lip area is selected from all the first candidate images as a reference image. Therefore, the screening efficiency of the reference image can be effectively improved.

[0082] In some optional implementations, the above step S2033 includes:

[0083] Step a1, obtaining pixel values ​​of the lip area in the first candidate image.

[0084] Specifically, the pixel value of the lip region may be read from the image attributes of the first candidate image, or the pixel value of the lip region in the first candidate image may be acquired by calling image processing software.

[0085] Step a2: based on the statistical analysis result of the pixel values, the first candidate image is screened to obtain a second candidate image.

[0086] It is understandable that the size of the pixel value will affect the clarity of the image. The lower the pixel value, the blurrier the image, and the less detail information can be obtained from the image. Therefore, the first candidate image can be screened again according to the statistical analysis results of the pixel value to obtain the second candidate image.

[0087] Specifically, the statistical analysis result of the pixel value includes at least one of the average pixel value, the pixel value variance, and the histogram. In addition, the statistical analysis result of the pixel value may also include other analysis data related to the image quality.

[0088] In some optional implementations, the above step a2 includes:

[0089] Step a21, based on the statistical analysis result of the pixel values, determine the average pixel value of the lip area in the first candidate image.

[0090] Step a22, based on the size of the average pixel value, the first candidate image is screened to obtain a second candidate image.

[0091] Specifically, a preset number of first candidate images are screened as second candidate images in descending order of average pixel values.

[0092] For example, first candidate images with top 10 average pixel values ​​are selected from the first candidate images as second candidate images.

[0093] In addition, the first candidate image whose average pixel value is greater than a preset pixel threshold may be used as the second candidate image.

[0094] Step a3, determining the quality of the teeth image in the lip area in the second candidate image, and determining the second candidate image with the best quality of the teeth image in the lip area as the reference image.

[0095] It should be noted that, after the second candidate image is determined in the first candidate image, the dental image quality of the lip region in the second candidate image can be obtained. Alternatively, the dental image quality of each first candidate image can be obtained first, and then, after the second candidate image is determined, the dental image quality of the second candidate image can be determined based on the dental image quality of the first candidate image obtained in advance.

[0096] The video generation method provided in this embodiment screens the first candidate image based on the statistical analysis results of the pixel values ​​of the lip area in the first candidate image to obtain the second candidate image. Therefore, the number of images that need to determine the quality of the tooth image can be further reduced. Then, the second candidate image with the best tooth image quality in the lip area is screened from all the second candidate images as the reference image. Therefore, the screening efficiency of the reference image can be effectively improved. At the same time, the lip shape quality in the reference image is further guaranteed.

[0097] Step S204, extract key points of the target object in the reference image to obtain reference key points. Please refer to the above step S104, which will not be repeated here.

[0098] Step S205: Based on the reference image, the reference key points and the driving key points, the lip region in the processed video is predicted to determine the target video corresponding to the target audio. Please refer to the above step S105, which will not be repeated here.

[0099] In this embodiment, a third video generation method is provided, which can be used in electronic devices such as mobile phones, tablet computers, etc. Figure 4 is a flow chart of a third video generation method according to an embodiment of the present invention. Figure 4 As shown, the process includes the following steps:

[0100] Step S401, obtaining target audio, a video to be processed and multiple images of a target object, wherein the video to be processed is obtained by performing mask processing on the lip region in the video of the target object. Please refer to the above step S101, which will not be repeated here.

[0101] Step S402, generating driving key points of the target object based on the target audio, which can be referred to in the above step S102, and will not be described in detail here.

[0102] Step S403, based on the tooth image quality of the lip region in the multiple images, the multiple images are screened to determine a reference image. Please refer to the above step S103, which will not be described in detail here.

[0103] Step S404: extract key points of the target object in the reference image to obtain reference key points. Please refer to the above step S104, which will not be described in detail here.

[0104] Step S405: Based on the reference image, the reference key points and the driving key points, the lip area in the video to be processed is predicted to determine the target video corresponding to the target audio.

[0105] In some optional implementations, the above step S405 includes:

[0106] Step S4051, extracting the lip area in the reference image to obtain a lip image.

[0107] Specifically, the lip region in the reference image may be cropped to obtain a lip image.

[0108] Step S4052, masking is performed on the lip region in the reference image to obtain a target image.

[0109] Specifically, the area corresponding to the lip image in the reference image is masked to obtain the target image.

[0110] It can be understood that the target image is used to represent information other than the lip image in the reference image.

[0111] Step S4053, predicting the lip area in the video to be processed based on the driving key points and the lip image, and aligning the information represented by the lip image, the target image and the reference key points during the prediction process to determine the target video corresponding to the target audio.

[0112] Specifically, the driving key points, lip images and the video to be processed are input into the target video generation model, and the lip images, target images and reference key points are used as additional inputs or conditions to guide the target video generation model to predict the lip area in the video to be processed to obtain the target video. Alternatively, the driving key points, lip images and the video image of the video to be processed are input into the target image generation model, and the lip images, target images and reference key points are used as additional inputs or conditions to guide the target image generation model to predict the lip area in the video image of the video to be processed to obtain the target video.

[0113] The video generation method provided in this embodiment divides the reference image into a lip image and a target image that does not contain a lip image. Then, the lip area in the video to be processed is predicted based on the driving key points and the lip image. And in the prediction process, the information represented by the lip image, the target image and the reference key points are aligned to obtain the target video. Therefore, when predicting the lip area in the video to be processed, only the information of one lip image is referenced, avoiding the problem of teeth flickering between multiple frames of video images of the target video, ensuring the lip shape coherence between multiple frames of video images of the target video, so as to further improve the lip shape driving effect of voice-driven lip shape and improve the video quality of the target video.

[0114] In some optional implementations, the above step S4053 includes:

[0115] Step b1, input the lip image, target image, reference key points and driving key points into the target alignment model, align the lip image, target image and driving key points to obtain the image alignment result.

[0116] Specifically, the target alignment model can adopt a generative model, a discriminative model or a target optical flow model.

[0117] In some optional implementations, the target alignment model is a target optical flow model. The above step b1 includes:

[0118] Step b11: input the lip image, the target image and the reference key points into the target optical flow model for feature extraction to obtain first feature data.

[0119] Specifically, Figure 5As shown, the target optical flow model includes a first convolution layer, a second convolution layer, a first spatial adaptive normalization layer, a second spatial adaptive normalization layer, a first pixel reorganization layer, a third spatial adaptive normalization layer, and a third convolution layer, which are connected in sequence.

[0120] Specifically, convolution operations are performed on the lip image, the target image, and the reference key points through the first convolution layer and the second convolution layer of the target optical flow model to obtain first feature data.

[0121] Step b12: input the driving key points into the target optical flow model, perform optical flow estimation based on the first feature data and the driving key points, and obtain an optical flow field.

[0122] Specifically, the driving key points are input into the target optical flow model, and the optical flow is estimated based on the first feature data and the driving key points to obtain the optical flow field, including: performing spatially adaptive normalization (Spatially-Adaptive Denormalization, SPADE for short) on the first feature data and the driving key points based on the first spatial adaptive normalization layer of the target optical flow model to obtain a first spatial adaptive normalization result. Performing spatially adaptive normalization on the first spatial adaptive normalization result based on the second spatial adaptive normalization layer to obtain a second spatial adaptive normalization result. Performing pixel shuffle on the second spatial adaptive normalization result based on the first pixel reorganization layer to obtain a pixel reorganization result. Performing spatially adaptive normalization on the pixel reorganization result based on the third spatial adaptive normalization layer to obtain a third spatial adaptive normalization result. Performing convolution operation on the third spatial adaptive normalization result based on the third convolution layer to obtain the optical flow field.

[0123] Step b13: Based on the optical flow field, spatial transformation is performed on the lip image, the target image and the first feature data to obtain an image alignment result.

[0124] Specifically, an image affine transformation (Warp) is performed on the lip image, the target image and the first feature data based on the optical flow field to obtain an image alignment result.

[0125] Specifically, the lip image, the target image and the first feature data are transformed or distorted using the optical flow field, so that the final image alignment result is consistent with the facial information represented by the driving key points.

[0126] The video generation method provided in this embodiment uses the target optical flow model to extract features from the lip image, the target image, and the reference key points to obtain the first feature data. The target optical flow model is used to perform optical flow estimation on the driving key points and the first feature data to obtain the optical flow field. Then, based on the optical flow field, the lip image, the target image, and the first feature data are spatially transformed to obtain the image alignment result. Therefore, the target optical flow model can be used to capture the facial information of the target object, and the facial information represented by the lip image, the target image, the reference key points, and the driving key points can be effectively aligned.

[0127] Step b2: input the lip image, the video to be processed, the driving key points and the image alignment result into the target video generation model, predict the lip area in the video to be processed, and obtain the target video corresponding to the target audio.

[0128] The video generation method provided in this embodiment inputs the lip image, the target image, the reference key points and the driving key points into the target alignment model, aligns the lip image, the target image and the driving key points, and obtains the image alignment result. Therefore, the image alignment result can be made consistent with the facial posture and expression represented by the driving key points, and the information of the lip area of ​​the target object is reflected in the image alignment result. Then, the lip image, the video to be processed, the driving key points and the image alignment result are input into the target video generation model, and the lip area in the video to be processed is predicted to obtain the target video. Therefore, the authenticity of the lip shape in the target video finally generated can be ensured.

[0129] In some optional implementations, the above step b2 includes:

[0130] Step b21, inputting the video to be processed and the driving key points into the encoder of the target video generation model for feature extraction to obtain second feature data.

[0131] Step b22, input the second feature data, the lip image, the first feature data and the image alignment result into the decoder of the target video generation model, predict the lip area in the processed video, and obtain the target video corresponding to the target audio.

[0132] In some optional implementations, the above step b22 further includes: extracting features from the target audio to obtain audio feature data. Inputting the second feature data, the lip image, the first feature data, the audio feature data, and the image alignment result into a decoder of the target video generation model, predicting the lip area in the processed video, and obtaining a target video corresponding to the target audio.

[0133] It is worth noting that the audio feature data is input into the target video generation model to predict the lip area in the processed video. Therefore, the details and lip shape of the lip can be enhanced, so that the lip shape of the target video is more synchronized with the target audio.

[0134] The video generation method provided in this embodiment inputs the video to be processed and the driving key points into the encoder of the target video generation model for feature extraction to obtain the second feature data. Then, the second feature data, the lip image, the first feature data and the image alignment result are input into the decoder of the target video generation model to predict the lip area in the video to be processed to obtain the target video. Therefore, only the information of one lip image is obtained in the target video generation model, which can effectively avoid the occurrence of tooth flickering between the video images of the target video. At the same time, the authenticity of the target video can be further improved.

[0135] Specifically, Figure 5 As shown, the target video generation model includes a fourth convolution layer, a fifth convolution layer, a fourth spatial adaptive normalization layer, a first adaptive instance normalization (Adaptive Instance Normalization, AdaIN) layer, a second pixel reorganization layer, a fifth spatial adaptive normalization layer, a second adaptive instance normalization layer, a sixth spatial adaptive normalization layer, and a sixth convolution layer, which are sequentially connected. Among them, the fourth convolution layer and the fifth convolution layer serve as encoders of the target video generation model, and the remaining layers serve as decoders of the target video generation model.

[0136] Specifically, the audio feature data of the target audio can be input into the target video generation model through the second adaptive instance normalization layer.

[0137] Specifically, Figure 6 As shown in FIG. 1 , the workflow of the spatial adaptive normalization layer is as follows: convolution operations are performed on the input key points through different convolutions to generate a first scale parameter and a first offset parameter with spatial dimensions. Also, instance normalization (Instance Norm) is performed on the input feature data. Then, the instance normalization result is multiplied by the corresponding first scale parameter and the corresponding first offset parameter is added to obtain the spatial adaptive normalization result.

[0138] Specifically, Figure 7As shown in FIG. 1 , the workflow of the adaptive instance normalization layer is as follows: feature extraction is performed on the input audio through different multilayer perceptrons (MLPs) to generate a second scale parameter and a second offset parameter. In addition, instance normalization is performed on the input feature data. Then, the instance normalization result is multiplied by the corresponding second scale parameter and the corresponding second offset parameter is added to obtain the adaptive instance normalization result.

[0139] As a specific example, Figure 8 As shown, the video of the target object is the original video, and the video image of the original video is input into a three-dimensional deformable face model (3D Morphable Model, abbreviated as 3DMM). According to the identification information and posture corresponding to the target object, the facial information of the target object is determined. The facial information of the target object is projected based on the target audio to obtain the driving key points. In addition, the lip area in the video image of the original video is masked to obtain the video image of the video to be processed. The video image of the video to be processed and the driving key points are input into the encoder of the target video generation model for feature extraction to obtain the second feature data. And, the video image with the best tooth image quality in the lip area of ​​the original video is used as the reference image, and the lip area of ​​the reference image is extracted to obtain the lip image. And the area corresponding to the lip image in the reference image is masked to obtain the target image. And, the key point extraction of the target object in the reference image is obtained to obtain the reference key point. The driving key point, the reference key point, the target image, and the lip image are input into the target optical flow model for optical flow estimation to obtain the optical flow field and the first feature data. Based on the optical flow field, the lip image, the target image and the first feature data are spatially transformed to obtain an image alignment result. Then, the image alignment result, the first feature data, the second feature data and the lip image are input into the decoder of the target video generation model to predict the lip area in the video image of the processed video to obtain the video image of the target video.

[0140] As one of the specific implementation methods, the Identity-Preserving Talking Face Generation with Landmark and Appearance Priors (IP_LAP) model based on landmarks and appearance priors is taken as an example. Among them, the IP_LAP model includes a Transformer-based key point generator, an alignment module for rendering key points to videos, and a translation module. The IP_LAP model is improved by using the video generation method disclosed in the present invention, and the key point generator of the IP_LAP model is used as the target key point prediction model, the alignment module is used as the target alignment model, and the translation module is used as the target video generation model. Then, after adopting the video generation method disclosed in the present invention, the input data of the alignment module in the IP_LAP model is changed from the original multiple reference images and the key points of the multiple reference images to the lip image, the target image and the reference key points. The input data of the translation module in the IP_LAP model is changed from the original video to be processed and the driving key points to the lip image, the video to be processed and the driving key points. Since the improved IP_LAP model uses the image with the best tooth image quality of the target object as the reference image, the improved IP_LAP model will only learn the information of a lip image, which can effectively avoid the problem of tooth flickering and effectively enhance the tooth image quality and stability of the target object in the target video output by the IP_LAP model.

[0141] As a specific application example, a target application is installed on a mobile phone, and the target application is used to generate a video. In the target application, the target audio can be collected using the recording function, or a text can be input in the target application, such as "The weather is really nice today", and the corresponding timbre is selected, and the input text is converted into the target audio according to the selected timbre. Then, the target application uses the video generation method disclosed in the present invention to generate a target video corresponding to the target audio. The generated target video is displayed or played on the target display page of the target application.

[0142] It is worth noting that in the video generation method disclosed in the present invention, multiple images of the target object are screened based on the tooth image quality, and the image with good tooth image quality is used as the reference image. Therefore, the tooth quality of the target object in the speech video generated by voice-driven lip shape can be effectively improved. At the same time, by masking the lip area of ​​the target object in the input image, the stability of the target object's teeth in the speech video can be effectively enhanced.

[0143] In this embodiment, a video generation device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0144] This embodiment provides a video generation device, such as Fig. 9 As shown, including:

[0145] The data acquisition module 901 is used to acquire target audio, a video to be processed, and multiple images of a target object, where the video to be processed is obtained by performing mask processing on the lip region in the video of the target object;

[0146] A first operation module 902, used to generate driving key points of a target object based on target audio;

[0147] An image screening module 903 is used to screen the multiple images based on the quality of the teeth image in the lip area in the multiple images to determine the reference image;

[0148] The second operation module 904 is used to extract key points of the target object in the reference image to obtain reference key points;

[0149] The lip driving module 905 is used to predict the lip area in the video to be processed based on the reference image, the reference key points and the driving key points, and determine the target video corresponding to the target audio.

[0150] In some optional implementations, the first operation module 902 includes:

[0151] The driving reference point operation unit is used to input the target audio into the target key point prediction model for key point prediction to obtain the driving key point. The target key point prediction model is trained based on the sample audio and the sample key points of the target object.

[0152] In some optional implementations, the image screening module 903 includes:

[0153] An opening amplitude acquisition unit is used to acquire the lip opening amplitude of the target object in multiple images;

[0154] A first image screening unit, configured to determine an image in which the lip opening amplitude is greater than a preset amplitude threshold value among the multiple images as a first candidate image;

[0155] The second image screening unit is used to determine the tooth image quality of the lip area in the first candidate image, and determine the first candidate image with the best tooth image quality in the lip area as the reference image.

[0156] In some optional embodiments, the second image screening unit includes:

[0157] A pixel value acquisition subunit, used to acquire pixel values ​​of the lip region in the first candidate image;

[0158] A pixel value screening subunit, used for screening the first candidate image based on the statistical analysis result of the pixel value to obtain a second candidate image;

[0159] The image quality screening subunit is used to determine the tooth image quality of the lip area in the second candidate image, and determine the second candidate image with the best tooth image quality in the lip area as the reference image.

[0160] In some optional embodiments, the pixel value screening subunit is specifically used to: determine the average pixel value of the lip area in the first candidate image based on the statistical analysis results of the pixel values; and screen the first candidate image based on the size of the average pixel value to obtain the second candidate image.

[0161] In some optional implementations, the lip driving module 905 includes:

[0162] An image extraction unit, used for extracting the lip region in the reference image to obtain a lip image;

[0163] A mask processing unit, used for performing mask processing on the lip region in the reference image to obtain a target image;

[0164] The lip driving unit is used to predict the lip area in the processed video based on the driving key points and the lip image, and in the prediction process, align the lip image, the target image and the information represented by the reference key points to determine the target video corresponding to the target audio.

[0165] In some optional embodiments, the lip drive unit includes:

[0166] The information alignment subunit is used to input the lip image, the target image, the reference key points and the driving key points into the target alignment model, align the lip image, the target image and the driving key points, and obtain the image alignment result;

[0167] The video generation subunit is used to input the lip image, the video to be processed, the driving key points and the image alignment results into the target video generation model, predict the lip area in the video to be processed, and obtain the target video corresponding to the target audio.

[0168] In some optional embodiments, the target alignment model is a target optical flow model; then, the information alignment subunit is specifically used to: input the lip image, the target image and the reference key points into the target optical flow model for feature extraction to obtain first feature data; input the driving key points into the target optical flow model, perform optical flow estimation based on the first feature data and the driving key points to obtain an optical flow field; based on the optical flow field, perform spatial transformation on the lip image, the target image and the first feature data to obtain an image alignment result.

[0169] In some optional embodiments, the video generation subunit is specifically used to: input the video to be processed and the driving key points into the encoder of the target video generation model for feature extraction to obtain second feature data; input the second feature data, the lip image, the first feature data and the image alignment result into the decoder of the target video generation model, predict the lip area in the video to be processed, and obtain the target video corresponding to the target audio.

[0170] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0171] The video generating device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0172] The present disclosure also provides an electronic device having the above Fig. 9 The video generating device shown.

[0173] See also Fig.10 , Fig.10 is a structural block diagram of an electronic device provided by an optional embodiment of the present disclosure, such as Fig.10As shown, the electronic device includes: one or more processors 1001, memory 1002, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.10 A processor 1001 is taken as an example.

[0174] The processor 1001 may be a central processing unit, a network processor or a combination thereof. The processor 1001 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0175] The memory 1002 stores instructions executable by at least one processor 1001, so that the at least one processor 1001 executes the method shown in the above embodiment.

[0176] The memory 1002 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 1002 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 1002 may optionally include a memory remotely arranged relative to the processor 1001, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0177] The memory 1002 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 1002 may also include a combination of the above types of memory.

[0178] The electronic device also includes an input device 1003 and an output device 1004. The processor 1001, the memory 1002, the input device 1003 and the output device 1004 may be connected via a bus or other means. Fig.10 The example of connecting through bus is taken in the following.

[0179] The input device 1003 can receive input digital or character information, and generate key signal input related to the user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick, etc. The output device 1004 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0180] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0181] A part of the present disclosure may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present disclosure through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes, but is not limited to, source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0182] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A video generation method, It is characterized in that The method comprises: Acquire target audio, a video to be processed, and multiple images of a target object, wherein the video to be processed is obtained by performing mask processing on a lip region in a video of the target object; generating driving key points of the target object based on the target audio; Based on the quality of the teeth images in the lip area of ​​the multiple images, the multiple images are screened to determine a reference image; Extracting key points of the target object in the reference image to obtain reference key points; Based on the reference image, the reference key points and the driving key points, the lip area in the video to be processed is predicted to determine a target video corresponding to the target audio.

2. The video generation method according to claim 1, It is characterized in that The method of screening the multiple images based on the quality of the teeth image in the lip area in the multiple images to determine the reference image includes: Acquire the lip opening amplitude of the target object in the multiple images; Determine, among the multiple images, an image in which the lip opening amplitude is greater than a preset amplitude threshold as a first candidate image; The tooth image quality of the lip region in the first candidate image is determined, and the first candidate image with the best tooth image quality in the lip region is determined as the reference image.

3. The video generation method according to claim 2, It is characterized in that The step of determining the tooth image quality of the lip region in the first candidate image and determining the first candidate image having the best tooth image quality in the lip region as the reference image comprises: Obtaining pixel values ​​of the lip area in the first candidate image; Based on the statistical analysis result of the pixel values, the first candidate image is screened to obtain a second candidate image; The tooth image quality of the lip region in the second candidate image is determined, and the second candidate image with the best tooth image quality in the lip region is determined as the reference image.

4. The video generation method according to claim 3, It is characterized in that The step of screening the first candidate image based on the statistical analysis result of the pixel values ​​to obtain the second candidate image includes: Determine an average pixel value of the lip region in the first candidate image based on a statistical analysis result of the pixel values; Based on the size of the average pixel value, the first candidate image is screened to obtain the second candidate image.

5. The video generation method according to claim 1, It is characterized in that The step of predicting the lip region in the video to be processed based on the reference image, the reference key point, and the driving key point, and determining a target video corresponding to the target audio includes: Performing image extraction on the lip region in the reference image to obtain a lip image; Performing mask processing on the lip region in the reference image to obtain a target image; The lip area in the video to be processed is predicted based on the driving key points and the lip image, and during the prediction process, the lip image, the target image and the information represented by the reference key points are aligned to determine the target video corresponding to the target audio.

6. The video generation method according to claim 5, It is characterized in that The step of predicting the lip region in the video to be processed based on the driving key points and the lip image, and aligning the lip image, the target image, and the information represented by the reference key points during the prediction process to determine a target video corresponding to the target audio, includes: Inputting the lip image, the target image, the reference key points and the driving key points into a target alignment model, aligning the lip image, the target image and the driving key points to obtain an image alignment result; The lip image, the video to be processed, the driving key points and the image alignment result are input into a target video generation model, the lip area in the video to be processed is predicted, and a target video corresponding to the target audio is obtained.

7. The video generation method according to claim 6, It is characterized in that The target alignment model is a target optical flow model; the lip image, the target image, the reference key points and the driving key points are input into the target alignment model, and the lip image, the target image and the driving key points are aligned to obtain an image alignment result, including: Inputting the lip image, the target image, and the reference key points into the target optical flow model to extract features, thereby obtaining first feature data; Inputting the driving key points into the target optical flow model, performing optical flow estimation based on the first feature data and the driving key points, and obtaining an optical flow field; Based on the optical flow field, spatial transformation is performed on the lip image, the target image, and the first feature data to obtain the image alignment result.

8. The video generation method according to claim 7, It is characterized in that The step of inputting the lip image, the video to be processed, the driving key points and the image alignment result into a target video generation model, predicting the lip area in the video to be processed, and obtaining a target video corresponding to the target audio includes: Inputting the to-be-processed video and the driving key points into the encoder of the target video generation model for feature extraction to obtain second feature data; The second feature data, the lip image, the first feature data and the image alignment result are input into the decoder of the target video generation model, the lip area in the video to be processed is predicted, and the target video corresponding to the target audio is obtained.

9. The video generation method according to claim 1, It is characterized in that The generating the driving key points of the target object based on the target audio comprises: The target audio is input into a target key point prediction model for key point prediction to obtain the driving key point, wherein the target key point prediction model is trained based on sample audio and sample key points of the target object.

10. A video generating device, It is characterized in that The device comprises: A data acquisition module, used to acquire target audio, a video to be processed, and multiple images of a target object, wherein the video to be processed is obtained by performing mask processing on the lip region in the video of the target object; A first operation module, configured to generate a driving key point of the target object based on the target audio; An image screening module, used for screening the multiple images based on the quality of the teeth image in the lip area in the multiple images to determine the reference image; A second operation module is used to extract key points of the target object in the reference image to obtain reference key points; The lip driving module is used to predict the lip area in the video to be processed based on the reference image, the reference key points and the driving key points, and determine the target video corresponding to the target audio.

11. An electronic device, It is characterized in that include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the video generation method according to any one of claims 1 to 9 by executing the computer instructions.

12. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the video generation method according to any one of claims 1 to 9.

13. A computer program product, It is characterized in that The method comprises computer instructions for causing a computer to execute the video generating method according to any one of claims 1 to 9.

Images