A method for generating a singing video from a drama character picture based on a GAN network

Abstract

The present invention discloses a method for generating a singing video from a drama character picture based on a GAN network, including the steps of: 1. Collecting Qinqiang opera singing videos to synthesize a new dataset; 2. Locating the position of the face in the new dataset, cropping the video with the face as the center, and importing it into a depth perception generative adversarial network; 3. The facial depth network selects two consecutive frames from the opera videos in the training set and recovers the dense 3D facial geometry from the facial videos; 4. The facial depth network generates respective depth maps for the RGB image of the drama makeup and the RGB image frames of the opera videos, connects the depth maps with the corresponding RGB images to form RGB-D, and the facial key point detection network outputs K facial key points; 5. Generating distorted features; 6. Generating refined distorted features of the image; 7. Decoding the refined distorted features to generate an image; 8. Using the mimsave function to convert the image into a video and adding sound with ffmpeg to generate a singing video, making the expressions of the drama makeup characters in the video natural and coherent.

Description

Technical Field

[0001] The present invention relates to video generation technology, and specifically to a method for generating a singing video from a picture of a drama character based on a GAN network. Background Art

[0002] As an intangible cultural heritage of China, traditional Chinese opera art is an important carrier of the inheritance of Chinese civilization. Spreading, inheriting, and promoting traditional Chinese opera art is of far-reaching significance for inheriting our excellent culture, enriching people's spiritual world, and enhancing national confidence. Among various means of expression in traditional Chinese opera, music, especially singing, has always been the dominant factor. Singing comes first among singing, speaking, acting, and acrobatic fighting. Due to singing, the spoken parts cannot maintain the natural intonation of daily language. It needs to be chanted, emphasizing the rhythm and cadence of the language intonation, forming musical beauty. Various forms of spoken parts, such as free spoken parts, rhymed spoken parts, rhythmic recitations (such as introductions and poems, dialogues), and rhythmic chantings (such as "gan paizi" and number plates), are different degrees of combinations of language and music; the beauty of the high and low, strong and weak, cadenced tones of Chinese characters is the basis for the musical beauty of the spoken parts in traditional Chinese opera. Generating a singing video using pictures of characters with theatrical makeup can not only help hearing-impaired users identify the content of traditional Chinese opera using lip-reading technology, but also has great application potential in some interesting fields such as movies, photography, and entertainment.

[0003] Existing speaker video generation technologies directly process face image data under normal circumstances. However, the facial makeup of traditional Chinese opera actors is relatively thick, especially the painted faces of the roles in traditional Chinese opera, resulting in very poor video generation effects. Therefore, generating a singing video using pictures of characters with theatrical makeup is a technical problem. Deep learning has made great contributions to speaker video generation. In recent years, generative adversarial networks have been constantly innovating, such as Vid2Vid, DAVS, and StyleHEAT, etc. Speaker video generation methods are roughly divided into three categories: ① audio-driven speaker video generation method, ② image-driven speaker video generation method, and ③ landmark-driven speaker video generation method. However, the above methods cannot be used to generate a singing video from a picture of a character with theatrical makeup. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for generating a singing video from a picture of a drama character based on a GAN network, which can generate a singing video with natural and coherent expressions from a picture of a character with theatrical makeup.

[0005] To achieve the above purpose, the present invention is implemented by adopting the following technical solutions:

[0006] A method for generating a singing video from a picture of a drama character based on a GAN network, comprising the following steps:

[0007] Step 1: Collect the Qinqiang opera singing videos and add them to the open-source dataset VoxCeleb to synthesize a new dataset.

[0008] Step 2: First, use face recognition to locate the faces in the new dataset, then use ffmpeg to crop the videos centered on the faces to generate a preprocessed dataset. Then, divide the preprocessed dataset into a training set, a validation set, and a test set, and import them into the deep perception generative adversarial network in batches.

[0009] Step 3: The facial depth network selects two consecutive frames from the opera-driven video frames in the training set in a self-supervised manner to learn the per-pixel facial depth map, and finally recovers the dense 3D facial geometry from the facial video.

[0010] Step 4: The facial depth network generates the depth maps of the RGB images of the theatrical makeup and the RGB image frames of the opera videos respectively, and connects the generated depth maps with the corresponding RGB images to form RGB-D. Then, through the facial key point detection network, learn the geometric and appearance features of the RGB-D facial key points and output K facial key points.

[0011] Step 5: The feature warping module in the face synthesis network receives the facial key points generated in Step 4 and generates a motion field for warping the source image feature map, fuses the facial motion and appearance information, and generates warped features.

[0012] Step 6: The depth encoder generates an encoding result from the facial depth map of the RGB image frame of the input opera video, and performs a linear projection on the encoding result and the warped features. Convert them into three latent feature maps F q , F k , F v , and fuse F q with F k to generate a dense guidance for face generation and obtain refined warped features for generating images.

[0013] Step 7: Decode the refined warped features to generate an image, and repeat Steps 4 to 7 to generate multiple images.

[0014] Step 8: Use the mimsave function of imageio to turn the generated multiple images into a video, and at the same time use ffmpeg to add sound to the video, and finally generate a singing video with sound.

[0015] Furthermore, the size of the video after cropping in Step 2 is 256*256.

[0016] Further, the facial depth network in step 3 consists of an encoder and a decoder. The encoder is a ResNet18 network without the last fully connected layer and pooling layer.

[0017] Further, the key point detection network in step 4 concatenates the depth map of size 1×256×256 and the corresponding RGB image of size 3×256×256 to form an RGB-D of size 4×256×256.

[0018] Further, step 4 also includes using the equivariance loss L E and the key point distance loss L D to evaluate the facial key point detection network. When a 2D transformation is applied to the image, the equivariance loss L E makes the key points predicted by the key point detection network change according to the applied 2D transformation to ensure the consistency of specific key points of the image. When the distance between two corresponding key points is lower than a predefined threshold, the key point distance loss L D penalizes the model.

[0019] Further, the working process of the feature warping module in step 5 is as follows:

[0020] Step 5.1), Use the dense 2D motion field to warp the downsampled image to generate an initial warped feature map;

[0021] Step 5.2), The occlusion estimator inputs the initial warped feature map into the feature encoder to predict the motion flow mask and the occlusion map. Among them: the motion flow mask assigns different confidence values to the estimated dense 2D motion field, so that the motion field is masked; the occlusion map corrects the rotation amplitude of the head by masking the feature map area;

[0022] Step 5.3), Use the masked motion field to warp the appearance feature map extracted by the feature encoder from the source image;

[0023] Step 5.4), Fuse the appearance feature map with the occlusion map to generate a warped feature.

[0024] Further, the three latent feature maps F q , F k , F v in step 6 respectively represent the query, key, and value in the self-attention mechanism.

[0025] Further, step 7 also includes downsampling the opera-driven video frames and the generated images to 4 different resolutions, using the trained VGG network to extract features from the downsampled opera-driven video frames and the generated images, calculating the L1 distance between the opera-driven video frames and the generated images, and using the perceptual loss L PAs part of the total loss, the Gan loss function is used to ensure that the generated image is similar to its corresponding opera-driven video frame.

[0026] Furthermore, the combined loss function L of the total loss is expressed as:

[0027]

[0028] In the formula: I g represents the generated image, I d represents the driving image, λ P , λ G , λ E , λ D all represent hyperparameters, and represent a set of sparse key points in the face, K is the number of detected facial key points, the subscript s represents the source image, the subscript d represents the driving image, L P represents the perceptual loss, L G represents the Gan loss, L E represents the isometry loss, L D represents the key point distance loss.

[0029] The present invention has the following beneficial effects:

[0030] The present invention first performs face recognition on the first frame of the opera video and crops the opera video centered on the face. This can not only reduce the noise information of the background, thereby maintaining the true facial structure of the characters with theatrical makeup, which is a key factor in generating high-quality singing videos of characters with theatrical makeup; but also improve the training speed of the network. In addition, using the per-pixel facial depth map provides dense 3D geometric information for the key point detection network, so as to guide the model to focus on the fine-grained key parts of the face, such as the eyes and mouth, during image generation, making the expressions of the opera characters in the generated singing video of the opera characters natural and smooth, and the mouth shapes accurate, with good visual effects. It can be seen that the method for generating a singing video using pictures of characters with theatrical makeup based on the depth perception GAN network of the present invention not only has a complete overall architecture, but also shows excellent performance in generating singing videos from pictures of characters with theatrical makeup, solving the problems of the untrue and discontinuous speaking videos of opera characters, and having a positive significance for the inheritance and promotion of Chinese intangible cultural heritage. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 : Architecture diagram of the facial depth network of the present invention;

[0032] Figure 2 : Architecture diagram of the key point estimator of the present invention;

[0033] Figure 3:Diagram of the feature distortion module in the face synthesis network of the present invention;

[0034] Figure 4 :Diagram of the cross-modal attention module in the face synthesis network of the present invention;

[0035] Figure 5 :Effect diagram of generating a singing video from an opera female role picture of the present invention;

[0036] Figure 6 :Effect of generating a singing video from an opera clown picture of the present invention;

[0037] Figure 7 :Flowchart of facial key point detection of the present invention;

[0038] Figure 8 :Flowchart of synthesizing facial key points into a video using the face synthesis network of the present invention. Detailed implementation manner

[0039] The following further elaborates on the specific content of the present invention in conjunction with embodiments, but it is not a limitation to the present invention.

[0040] As Figures 1 to 8 shown, a method for generating a singing video from a drama character picture based on a GAN network includes the following steps:

[0041] Step 1: Collect Qinqiang opera singing videos on the Internet and add the collected videos to the open-source dataset VoxCeleb to synthesize a new dataset;

[0042] Step 2: First, use face recognition to locate the position of the face in the new dataset, and then use ffmpeg to crop the video to a size of 256*256 with the face as the center to generate a preprocessed dataset. Then, divide the preprocessed dataset into a training set, a validation set, and a test set, and import them into the deep perception generative adversarial network in batches;

[0043] Step 3: The facial depth network selects two consecutive frames from the opera-driven video frames of the training set in a self-supervised manner to learn the per-pixel facial depth map, and finally recovers the dense 3D facial geometry from the facial video;

[0044] The facial depth network (as Figure 1 shown) consists of an encoder and a decoder. The encoder is a ResNet18 network, and it does not have the last fully connected layer and pooling layer;

[0045] Step 4: As Figure 7As shown, the facial depth network generates respective depth maps for the input RGB image of the theatrical makeup and the RGB image frames of the Chinese opera video, and connects the generated depth map of size 1×256×256 and the corresponding RGB image of size 3×256×256 to form an RGB-D for input to the facial keypoint detection network. Then, through the facial keypoint detection network (such as Figure 2 as shown), learn the geometric and appearance features of the RGB-D facial keypoints and output K facial keypoints;

[0046] Use the equivariance loss L E and the keypoint distance loss L D to evaluate the facial keypoint detection network. When applying a 2D transformation to an image, the keypoints predicted by the keypoint detection network should change according to the applied transformation. Therefore, using the equivariance loss L E can ensure the consistency of specific image keypoints. At the same time, the facial keypoints detected by the keypoint detection network should be clustered around a small neighborhood. If the distance between two corresponding keypoints is below a predefined threshold, use the keypoint distance loss L D to penalize the model;

[0047] Step 5: The feature warping module in the face synthesis network accepts the facial keypoints generated in Step 4 and generates a motion field for warping the feature map of the source image, fusing the facial motion and appearance information to generate warped features;

[0048] The structure of the feature warping module is as Figure 3 shown, where: D is the downsampling operation, W is the warping operation, · is the element-wise multiplication, + and - represent addition and subtraction operations respectively, and its specific working process is as follows:

[0049] Step 5.1): Use the dense 2D motion field to warp the downsampled image to generate an initial warped feature map;

[0050] Step 5.2): The occlusion estimator inputs the initial warped feature map into the feature encoder to predict the motion flow mask and the occlusion map. Among them: the motion flow mask assigns different confidence values to the estimated dense 2D motion field, so that the motion field is masked; the occlusion map corrects the rotation amplitude of the head by masking the feature map area;

[0051] Step 5.3): Use the masked motion field to warp the appearance feature map extracted from the source image by the feature encoder;

[0052] Step 5.4): Fuse the appearance feature map with the occlusion map to generate warped features;

[0053] Step 6: The cross-modal attention module in the face synthesis network (such as Figure 4As shown in the figure), the obtained distorted features and the facial depth map of the opera character image are further used to learn the dense depth perception attention map, so as to refine the distorted features, thereby generating better generated images. Specifically: the depth encoder generates an encoded result from the facial depth map of the RGB image frame of the input opera video, and linearly projects the encoded result and the distorted features, and converts them into three latent feature maps F through three 1×1 convolutional layers q ,F k ,F v , the three latent feature maps F q ,F k ,F v represent the query, key, and value in the self-attention mechanism respectively. Therefore, the geometry-related query feature F q generated by the depth map can be fused with the appearance-related key feature F k to generate a dense guidance for face generation and obtain the final refined distorted features for generation;

[0054] Step 7: Decode the refined distorted features to generate an image, and repeat steps 4 to 7 to generate multiple images; and use the perceptual loss L P to ensure that the generated image is similar to the features in its corresponding opera-driven video frame. The specific process is as follows:

[0055] 7.1) Downsample the corresponding opera-driven video frame and the generated image to 4 different resolutions, namely 256×256, 128×128, 64×64, and 32×32;

[0056] 7.2) Use the trained VGG network to extract features from the opera-driven video frame and the generated image downsampled in step 7.1), calculate the L1 distance between the opera-driven video frame and the generated image, and use the perceptual loss L P as part of the total loss; the combined loss function L is expressed as:

[0057]

[0058] In the formula: I g represents the generated image, I d represents the driving image, λ P ,λ G ,λ E ,λ D all represent hyperparameters, and represent a set of sparse key points in the face, K is the number of detected facial key points, the subscript s represents the source image, the subscript d represents the driving image, L P represents the perceptual loss, L G represents the Gan loss, LE Denote the equal variance loss as L D Denote the key point distance loss;

[0059] 7.3), Use the Gan loss function (which consists of the least square loss and feature matching loss introduced in pix2pixHD) to ensure that the generated image is similar to its corresponding opera-driven video frame;

[0060] Step 8, Use the mimsave function of imageio to convert the generated multiple images into a video, and at the same time use ffmpeg to add sound to the video, and finally generate a singing video with sound (see the generation effects of the female lead and clown roles in Figure 5 and Figure 6 ).

Claims

1. A method for generating a singing video from a drama character picture based on a GAN network, characterized in that, it includes the following steps: Step 1: Collect Qinqiang opera singing videos and add them to the open-source dataset VoxCeleb to synthesize a new dataset; Step 2: First, use face recognition to locate the positions of faces in the new dataset, then use ffmpeg to crop the videos centered on the faces to generate a preprocessed dataset, and then divide the preprocessed dataset into a training set, a validation set, and a test set, and import them into the deep perception generative adversarial network in batches; Step 3: The facial depth network selects two consecutive frames from the opera-driven video frames in the training set in a self-supervised manner to learn the per-pixel facial depth map, and finally recovers the dense 3D facial geometry from the facial video; Step 4: The facial depth network generates respective depth maps for the RGB image of the drama makeup and the RGB image frames of the opera video, and connects the generated depth maps with the corresponding RGB images to form RGB-D. Then, through the facial key point detection network, learn the geometric and appearance features of the RGB-D facial key points and output K facial key points; Step 5: The feature warping module in the face synthesis network receives the facial key points generated in Step 4 and generates a motion field for warping the feature map of the source image, fuses the facial motion and appearance information, and generates warped features; Step 6: The depth encoder generates an encoded result from the facial depth map of the RGB image frames of the input opera video, and linearly projects the encoded result and the warping features, and converts them into three latent feature maps F through three 1×1 convolutional layers q ,F k ,F v , and fuse F q with F k to generate a dense guidance for face generation and obtain a refined warping feature for generating an image; Step 7: Decode the refined warped features to generate an image, and repeat Steps 4 to 7 to generate multiple images; Step 8: Use the mimsave function of imageio to convert the generated multiple images into a video, and at the same time use ffmpeg to add sound to the video, and finally generate a singing video with sound.

2. The method for generating a singing video from a drama character picture based on a GAN network according to claim 1, characterized in that, the size of the video after cropping in Step 2 is 256*256.

3. The method for generating a singing video from a drama character picture based on a GAN network according to claim 1, characterized in that, the facial depth network in Step 3 consists of an encoder and a decoder, and the encoder is a ResNet18 network without the last fully connected layer and pooling layer.

4. The method for generating a singing video from a drama character picture based on a GAN network according to claim 1, characterized in that, the key point detection network in Step 4 connects the depth map with a size of 1×256×256 and the corresponding RGB image with a size of 3×256×256 to form an RGB-D with a size of 4×256×256.

5. The method for generating a singing video from a drama character picture based on a GAN network according to claim 1, characterized in that, Step 4 further includes using an isovariance loss L E and a key-point distance loss L D to evaluate a facial key-point detection network. When applying a 2D transformation to an image, the isovariance loss L E causes the key points predicted by the key-point detection network to change according to the applied 2D transformation, thereby ensuring the consistency of specific key points of the image. When the distance between two corresponding key points is lower than a predefined threshold, the key-point distance loss L D penalizes the model.

6. The method for generating a singing video from a drama character picture based on a GAN network according to claim 1, characterized in that, the working process of the feature warping module in Step 5 is as follows: Step 5.1): Use the dense 2D motion field to warp the downsampled image to generate an initial warped feature map; Step 5.2), Occlusion Estimator, input the initial warped feature map into the feature encoder to predict the motion flow mask and the occlusion map, where: the motion flow mask assigns different confidence values to the estimated dense 2D motion field, so that the motion field is masked; the occlusion map corrects the rotation amplitude of the head by masking the feature map area; Step 5.3), warp the appearance feature map extracted by the feature encoder from the source image with the masked motion field; Step 5.4), fuse the appearance feature map with the occlusion map to generate the warped feature.

7. The method for generating a singing video from a drama character picture based on a GAN network according to claim 1, characterized in that The three potential feature maps F in step 6 q , F k , F v respectively represent the query, key, and value in the self-attention mechanism.

8. The method for generating a singing video from a drama character picture based on a GAN network according to claim 1, characterized in that Step 7 further includes downsampling the opera-driven video frames and the generated images to four different resolutions, extracting features from the downsampled opera-driven video frames and the generated images using the trained VGG network, calculating the L1 distance between the opera-driven video frames and the generated images, and taking the perceptual loss L P As part of the total loss, the Gan loss function is used to ensure that the generated images are similar to their corresponding opera-driven video frames.

9. The method for generating a singing video from a drama character picture based on a GAN network according to claim 8, characterized in that the combined loss function L of the total loss is expressed as: Where: I g represents the generated image, I d represents the driving image, λ P , λ G , λ E , λ D all represent hyperparameters, and represent a set of sparse key points in the face, K is the number of detected facial key points, the subscript s represents the source image, the subscript d represents the driving image, L P represents the perceptual loss, L G represents the Gan loss, L E represents the isotropic loss, L D represents the key point distance loss.

Images