A comic video generation method based on domain adaptation

Abstract

The present invention provides a method for generating a comic video based on domain adaptation, comprising the following steps: first, use a face video dataset to train a motion-driven network; for a given source image and a driving frame input to the pre-trained motion-driven network, it can generate and output a target video of the given source image according to the motion of the driving frame; then, use the face video dataset and the comic dataset to train a cross-domain network; introduce a domain discriminator, and use a gradient reversal layer to confuse the face video dataset and the comic dataset to maximize the domain classification error. The present invention first uses a face video dataset to train for the image dynamicization task, and then adapts it to the comic domain using the domain adaptation method, solving the problems of overfitting and poor generalization ability that are likely to occur when directly using the small number of comic datasets for training, and does not require per-video optimization, ensuring the quality and stability of the generated video.

Description

Technical Field

[0001] The present invention relates to the technology of generating comic videos, and particularly to the technology of generating comic videos driven by face videos based on domain adaptation. Background Art

[0002] With the development of deep learning, the work of face stylization has received extensive attention. Nowadays, live streaming and short videos have become the mainstream of people's entertainment life, and video style conversion also has great market and commercial application value. The most natural way to extend the style transfer algorithm from images to videos is to use the technology of image style transfer to complete the conversion of videos frame by frame. However, it is very difficult to ensure the consistency of the style between video frames, and there are usually problems of jitter and discontinuity. This is mainly caused by the instability of single-image style conversion.

[0003] Currently, the methods for generating face comic videos mainly include three categories. The first category is the method based on key frame extraction. This kind of method first selects the most representative frame as the key frame, performs style transfer on it, and then interpolates the style-transferred key frame by extracting the features of other frames. This method needs to optimize and iterate for each person's corresponding video, with low efficiency, and it is difficult to constrain the selection criteria of key frames. The second category ensures the continuity and stability of the stylized video by extracting and propagating short-term temporal information, but this method needs to balance the stability and clarity of the results. The third category is based on comic modeling, using parameters to represent expressions, extracting and quantifying the expression actions in the face video, and forming a video with the corresponding comics according to the same motion parameters. Such methods are very complex to implement, with a large amount of calculation, and there are also certain errors in comic modeling and parameter quantization.

[0004] The image animation method published in NeurIPS in 2019 provides a new idea for image-to-video generation. Image animation will animate the objects in the source image according to the motion of the driving video to generate a video sequence. However, for the problem that the number of comic datasets is small and there is no corresponding video data, it is necessary to first complete the task of image animation in the face video dataset and then transfer it to the comic domain. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide an effective and high-quality method for generating comic videos in view of the insufficient number of comic datasets and the problems that the optimization and iteration time of the comic video generation network is too long, and the continuity, stability and generation quality need to be improved.

[0006] The technical solution adopted by the present invention to solve the above technical problems is a method for generating comic videos based on domain adaptation, including the following steps:

[0007] Step 1: Train the motion-driven network using the face video dataset; the motion-driven network includes a key point detector, a dense motion field module, and a generation module; specifically, it includes the following steps:

[0008] 1-1 Align the faces in the face video dataset and unify the sizes.

[0009] 1-2 Use the key point detector to learn the positions of a set of key points and their local affine transformations for the source image and the driving frames in the face video dataset respectively.

[0010] 1-3 The dense motion field module learns the motion of each pixel based on the positions of the key points and their local affine transformations, and outputs a mask indication image, which is used to indicate the parts that need to be obtained through image deformation and the parts that can be obtained through image inpainting.

[0011] 1-4 The generation module extracts the motion features of the source image and combines the mask indication image to render the source image to obtain the target image, and integrates all the target images into the target video.

[0012] Step 2: Train the cross-domain network using the face video dataset and the comic dataset; the cross-domain network includes a key point detector, a domain discriminator, a gradient reversal layer, a dense motion field module, and a generation module, and the gradient reversal layer is set between the key point detector and the domain discriminator; the initial network parameters of the cross-domain network use the network parameters of the trained motion-driven network, and select a face video and a comic dataset with a larger number than the comic dataset as the training dataset; specifically, it includes the following steps:

[0013] 2-1 Align the faces in the face video dataset and the comic dataset and unify the sizes.

[0014] 2-2 The face video dataset enters the key point detector, and the key point detector learns the positions of a set of key points and their local affine transformations for the source image and the driving frames in the face video dataset respectively; the key point detector outputs the positions of the detected key points and their local affine transformations to the dense motion field module and the domain discriminator respectively; the domain discriminator is used to judge whether the results generated by the key point detector are real.

[0015] The comic dataset enters the key point detector, and the key point detector learns the positions of a set of key points and their local affine transformations for the source image and the driving frames in the comic dataset respectively; the key point detector outputs the positions of the detected key points and their local affine transformations to the dense motion field module on one side, and sends them to the domain discriminator through the gradient reversal layer on the other side.

[0016] The 2-3 dense motion field module learns the motion of each pixel through the positions of key points and their local affine transformations, and outputs a mask indication image, which is used to indicate the parts that need to be obtained through image deformation and the parts that can be obtained through image inpainting;

[0017] The 2-4 generation module extracts the motion features of the source image and combines the mask indication image to render the source image to obtain the target image, and integrates all the target images into the target video;

[0018] Step 3: After the cross-domain network training is completed, input the comic image into the cross-domain network, and through the trained key point detector, dense motion field module and generation module, the generated comic video can be obtained.

[0019] The beneficial effects of the present invention are as follows: First, use the face video dataset to train for the image animation task, and then adapt it to the comic domain using the domain adaptation method, which solves the problems of overfitting and poor generalization ability that are likely to occur when directly using the small number of comic datasets for training. This method does not require per-video optimization, and the pre-training method using the face video dataset also ensures the quality and stability of the generated video. Description of the Drawings

[0020] Figure 1 : A motion-driven network pre-trained using a face video dataset.

[0021] Figure 2 : A network model for comic domain adaptation. Detailed Embodiments

[0022] The embodiments are implemented on a TITAN XP server. It mainly includes two steps:

[0023] Use the face video dataset to train the motion-driven network, regarded as the pre-training process of the cross-domain network;

[0024] Use the comic dataset to train the cross-domain network to complete domain adaptation.

[0025] Specifically:

[0026] Step 1 is as Figure 1 shown. The motion-driven network includes a key point detector, a dense motion field module and a generation module. Use the face video dataset to pre-train the motion-driven network. For a given face photo (source image) and another face driving video (driving frame) input by the pre-trained motion-driven network, generate and output the dynamic video (target video) of the given face photo according to the motion of the driving video. Specifically, it includes the following steps:

[0027] 1-1 Align the face video dataset, and unify it to the same size of 256x256;

[0028] 1-2 Use a keypoint detector to learn the positions of a set of keypoints and their local affine transformations for the source image and the driving frames in the face video dataset respectively;

[0029] 1-3 The dense motion field module learns the motion of each pixel through the positions of the keypoints and their local affine transformations, and outputs a mask indication image, which is used to indicate the parts that need to be obtained through image deformation and the parts that can be obtained through image inpainting;

[0030] 1-4 The generation module extracts the motion features of the source image and combines the mask indication image to render the source image to obtain the target image, and integrates all the target images into the target video;

[0031] Specifically, the keypoint detection network is implemented based on the Mobilenet-v2 module. First, use a convolution conv to expand the number of channels to 64, and then use four Mobilenet-v2 blocks to extract keypoint features to ensure accuracy and greatly reduce the computational amount. After obtaining the keypoint features, use the softmax activation as the prediction result of the keypoints. Then, use the keypoint features to first input a convolution conv, and then calculate its affine transformation matrix and add the first-order derivative of this matrix (the Jacobian matrix) as the parameters of the local affine transformation of the keypoint neighborhood.

[0032] The dense motion field module first converts the keypoint information of the source image and the driving frame into a region respectively, then takes the difference to record the changes between the keypoints and their neighborhoods, and performs upsampling to extract the keypoint changes. Finally, take the positions of the keypoints in the source image and the features where the keypoints change as inputs, use an encoder-decoder module (including three downsampling convolutional convs and three upsampling convolutional convs) and a convolutional conv to estimate the mask, and finally use the Softmax activation function to obtain the regions that need to be redrawn and the regions that need to be erased.

[0033] The generation module first uses two downsampling convolutional convs and BatchNorm normalization to extract features from the source image, then uses bilinear interpolation to sample and align the motion of the source image and the keypoints, then multiplies the obtained feature map by the mask calculated by the dense motion field module, and finally inputs it into 3 residual blocks and two upsampling and BatchNorm normalization layers. Finally, use a 7x7 convolutional kernel for conv convolution and use the Sigmoid function for activation to obtain the target frame.

[0034] Step 2: Introduce a domain discriminator on the basis of the motion-driven network to construct a cross-domain network, and introduce a gradient reversal layer between the keypoint detector and the domain discriminator, asFigure 2 As shown, a model for domain adaptation using a comic dataset is obtained, including a keypoint detector, a domain discriminator, a gradient reversal layer, a dense motion field module, and a generation module. The initial network parameters for training the cross-domain network use the network parameters of the trained motion-driven network.

[0035] The features extracted by the keypoint detector network are input into the domain discriminator through the gradient flipping layer. During backpropagation, the gradient is reversed to confuse the face dataset and the comic dataset, maximizing the domain classification error so that the features extracted by the keypoint network contain the common features of faces and comics. When training the cross-domain network, face videos and comic datasets are randomly selected in a quantity 10 times that of the comic dataset.

[0036] 2-1 Perform face alignment on both the face video dataset and the comic dataset, and unify them to the same size of 256x256;

[0037] 2-2 The face video dataset enters the keypoint detector. The keypoint detector learns the positions of a set of keypoints and their local affine transformations for the source image and the driving frame in the face video dataset respectively; the keypoint detector outputs the positions of the detected keypoints and their local affine transformations to the dense motion field module and the domain discriminator respectively; among them, the data input into the domain discriminator first passes through a fully connected layer fc to convert the number of channels to 100, then performs normalization bn, is input into the relu activation function, and then passes through a fully connected layer fc to become 2 channels, and finally obtains the result of domain discrimination through the Softmax activation function;

[0038] The comic dataset enters the keypoint detector. The keypoint detector learns the positions of a set of keypoints and their local affine transformations for the source image and the driving frame in the comic dataset respectively; the positions of the detected keypoints and their local affine transformations output by the keypoint detector pass through the gradient reversal layer to the domain discriminator; the domain discriminator is used to judge whether the results generated by the keypoint detector are real, and a GAN Loss is introduced for constraint. In addition, the L1 distance between the features extracted by VGG19 of the generated target image and the real target image is constrained using the reconstruction loss. To increase the stability of keypoint estimation, it is constrained that the keypoints of the target image move backward to be the same as the keypoints of the source image;

[0039] 2-3 The dense motion field module learns the motion of each pixel through the positions of the keypoints and their local affine transformations, and outputs a mask indication image, which is used to indicate the part that needs to be obtained by image deformation and the part that can be obtained by image inpainting;

[0040] The 2-4 generation module extracts the motion features of the source image and combines the mask indication image to render the source image to obtain the target image, and integrates all the target images into the target video; at this time, when the source image comes from the comic dataset, a video of the comic image is generated.

[0041] After the cross-domain network is trained, the domain discriminator is no longer used. The comic image is input into the cross-domain network, and the generated comic video can be obtained through the trained key point detector, dense motion field module and generation module.

Claims

1. A method for generating comic videos based on domain adaptation, characterized in that It includes the following steps: Step 1: Train the motion-driven network using the face video dataset; the motion-driven network includes a key point detector, a dense motion field module, and a generation module; Specifically, it includes the following steps: 1-1 Align the faces in the face video dataset and unify the sizes; 1-2 Use the key point detector to learn the positions of a set of key points and their local affine transformations for the source image and the driving frames in the face video dataset respectively; 1-3 The dense motion field module learns the motion of each pixel through the positions of the key points and their local affine transformations, and outputs a mask indication image, which is used to indicate the parts that need to be obtained through image deformation and the parts that can be obtained through image inpainting; 1-4 The generation module extracts the motion features of the source image and combines the mask indication image to render the source image to obtain the target image, and integrates all the target images into the target video; Step 2: Train the cross-domain network using the face video dataset and the comic dataset; the cross-domain network includes a key point detector, a domain discriminator, a gradient reversal layer, a dense motion field module, and a generation module, and the gradient reversal layer is set between the key point detector and the domain discriminator; the initial network parameters of the cross-domain network use the network parameters of the trained motion-driven network, and select a face video dataset and a comic dataset with a quantity more than that of the comic dataset as the training dataset; specifically, it includes the following steps: 2-1 Align the faces in the face video dataset and the comic dataset and unify the sizes; 2-2 The face video dataset enters the key point detector, and the key point detector learns the positions of a set of key points and their local affine transformations for the source image and the driving frames in the face video dataset respectively; the key point detector outputs the positions of the detected key points and their local affine transformations to the dense motion field module and the domain discriminator respectively; the domain discriminator is used to judge whether the results generated by the key point detector are real; The comic dataset enters the key point detector, and the key point detector learns the positions of a set of key points and their local affine transformations for the source image and the driving frames in the comic dataset respectively; the key point detector outputs the positions of the detected key points and their local affine transformations to the dense motion field module on one side, and sends them to the domain discriminator through the gradient reversal layer on the other side; 2-3 The dense motion field module learns the motion of each pixel through the positions of the key points and their local affine transformations, and outputs a mask indication image, which is used to indicate the parts that need to be obtained through image deformation and the parts that can be obtained through image inpainting; 2-4 The generation module extracts the motion features of the source image and combines the mask indication image to render the source image to obtain the target image, and integrates all the target images into the target video; Step 3: After the cross-domain network is trained, input the comic image into the cross-domain network, and the generated comic video can be obtained through the trained key point detector, dense motion field module, and generation module.

2. The method according to claim 1, wherein The selection method of the training dataset in Step 2 is: randomly select a face video dataset and a comic dataset with 10 times the quantity of the comic dataset as the training dataset.

3. The method according to claim 1, characterized in that, The specific processing of the input data by the domain discriminator is as follows: The input data first passes through a fully connected layer to convert the number of channels to 100, then after normalization operation, it is input into the relu activation function, and then passes through another fully connected layer to become 2 channels, and finally the domain discrimination result is obtained through the Softmax activation function.

Images