A method and device for generating ink and wash video based on instance segmentation and reference frames

Abstract

The application provides a kind of ink-wash video generation method and device based on instance segmentation and reference frame, including obtaining the instance segmentation of each frame in real video, and according to the optical flow estimation between frames, the similarity of each frame is calculated, and the frame to be converted and the reference frame are selected;The optical flow estimation and the frame to be converted, the reference frame and its instance segmentation are input into the ink style network model, to obtain the ink video frame of the frame to be converted, so as to obtain the ink video of the real video.The ink style generation ability of the model is improved, better results are achieved compared with the prior art in the aspects of white space, different object stroke entanglement and different scale object processing, ink style with reference is realized, error accumulation in traditional sequential conversion is avoided, and compared with the prior art, flicker and inconsistency in the generated ink video are reduced.

Description

Technical Field

[0001] This invention belongs to the field of video stylization, specifically relating to a method and apparatus for generating ink-wash video based on instance segmentation and reference frames. Background Technology

[0002] Ink-wash video generation aims to generate corresponding ink-wash style videos from any given real-shot video, achieving the goal of batch generation of ink-wash videos. Ink-wash video generation can be broadly divided into two parts: first, ensuring each generated frame possesses a suitable ink-wash style; and second, maintaining the coherence and consistency of the stylized video. While research has addressed both aspects separately, attempts to appropriately combine them to achieve satisfactory results are still lacking.

[0003] Regarding the former, most current image ink painting stylization systems are based on adjustments to general stylization methods, failing to adequately consider the characteristics of the ink painting style itself. This results in the following problems with the generated style quality: 1) difficulty in correctly selecting white space areas; 2) strokes of different objects easily becoming entangled; 3) difficulty in simultaneously processing objects of multiple scales. Due to these issues, ink painting video generation systems based on these methods struggle to meet the needs of practical applications.

[0004] Regarding the latter, due to the characteristic of ink painting having a lot of blank space, the results produced by existing video stylization methods contain a lot of flickering and inconsistencies, making it difficult to directly transfer them to ink painting video generation tasks. Summary of the Invention

[0005] To address the aforementioned problems, this invention proposes a method and apparatus for generating ink-wash style videos based on instance segmentation and reference frames. Compared to existing image ink-wash style methods, this method achieves a better ink-wash style while maintaining the coherence and consistency of the ink-wash videos. This invention can convert real-shot videos into ink-wash style videos with the same content, enabling batch generation of ink-wash videos and improving subjective visual quality and artistic effect.

[0006] The technical solution adopted in this invention is as follows:

[0007] A method for generating ink-wash videos based on instance segmentation and reference frames, comprising the following steps:

[0008] 1) Obtain the instance segmentation map of each real video frame in the real video, and calculate the similarity between any two real video frames based on the optical flow estimation between all real video frames.

[0009] 2) Based on the similarity level, select the real video frame to be converted and one or more reference frames for each round, and input the real video frame to be converted and the corresponding reference frames into the ink painting stylization network model. The ink painting video frame of the real video frame to be converted is obtained through the following steps:

[0010] 2.1) Extract real video features and corresponding reference features based on the real video frames to be converted, each reference frame and the corresponding instance segmentation map, and align each reference feature with the real video features using optical flow estimation.

[0011] 2.2) Based on the difference between each reference feature and the real video frame features, calculate the reference feature score of each reference feature, and fuse all reference features through the reference feature scores to obtain the fused reference feature;

[0012] 2.3) Decode the modified real video features obtained from the real video features and the fused reference features to obtain the ink-wash video frame of the real video frame to be converted.

[0013] 3) Repeat step 2) to generate the ink wash video of the real video.

[0014] Furthermore, the following steps are used to select the real video frames to be converted and several reference frames for each round:

[0015] 1) Based on the similarity level, select the frame with the lowest similarity to all converted frames and the highest similarity to all unconverted frames as the real video frame to be converted in this round;

[0016] 2) Based on the similarity, select several converted frames that have the highest similarity to the real video frames to be converted as reference frames. When the first ink painting video frame is generated, the number of reference frames is 0.

[0017] Furthermore, the real video features are obtained through the following steps:

[0018] 1) Combine real video frames and corresponding instance segmentation maps, encode and stitch the results to obtain global features;

[0019] 2) For each instance i and its corresponding region image x in each instance segmentation map. i , region image x i After being stitched together with the corresponding instance segmentation map in the channel dimension, the image is enlarged to the image size of the real video frame to be converted;

[0020] 3) Extract image features from the magnified image and shrink the image features back to the region image x. i The feature size is used to obtain the instance feature H. i ;

[0021] 4) Integrate global features with instance features H i This yields the true characteristics of the video.

[0022] Further, the fused reference features are obtained through the following steps:

[0023] 1) Apply the softmax transform to each reference feature score at each position to obtain the corresponding coefficients;

[0024] 2) The reference features are weighted and summed at each position according to the coefficients to obtain the fused reference features.

[0025] Furthermore, the modified real video features are obtained through the following steps:

[0026] 1) The difference between the fused reference features and the real video frame features is scored to obtain the fused reference feature score;

[0027] 2) Use the weighted summation of fused reference feature scores to modify the features of real video frames, and obtain the modified real video features.

[0028] Furthermore, the structure of the ink-wash styled network model includes: an encoder, a reference frame feature fusion unit, a feature modifier, and a decoder. The encoder structure includes: several convolutional layers and several residual blocks, wherein each convolutional layer is followed by a batch normalization module and a linear rectified function. Starting from the second convolutional layer, each convolutional layer is downsampled with a stride greater than 1. The reference frame feature fusion unit and the feature modifier structure respectively include: a convolutional layer, several staggered residual blocks and convolutional layers, and a convolutional layer. The decoder structure is symmetrical to the encoder, including: several residual blocks and several convolutional layers, wherein each convolutional layer is followed by a linear rectified function, and except for the last convolutional layer, the remaining convolutional layers are upsampled to nearest neighbors, and a batch normalization module is added after each convolutional layer.

[0029] Furthermore, the ink-wash styled network model F is trained through the following steps:

[0030] 1) Collect several sets of training data for iterative training. Each set of training data includes: a real sample video, a real sample frame taken from the real sample video, several sample reference frames taken from the real sample video, and a randomly selected real ink painting.

[0031] 2) Obtain the instance segmentation map of the real sample frame and each sample reference frame, calculate the optical flow of the real sample frame and each sample reference frame, and input the real sample frame, each sample reference frame, and the corresponding instance segmentation map and optical flow of the real sample frame and each sample reference frame into the ink painting stylization network model to obtain an ink painting video sample frame.

[0032] 3) Input the sample frames of the ink painting video and the real ink painting into an ink painting discriminator and an ink painting style discriminator for discrimination. The structure of the ink painting discriminator and the ink painting style discriminator includes: several convolutional layers. Except for the first and last layers, each convolutional layer is followed by a batch normalization module and a linear rectified function. The first layer is followed by only a linear rectified function and downsampling is performed with a stride greater than 1.

[0033] 4) Input the real ink painting into the real image generator to obtain the generated real image. Then, input the real video sample frames and the generated real image into a real image discriminator for discrimination. The structure of the real image generator includes an encoder and a decoder. The encoder structure includes several convolutional layers and several residual blocks. Each convolutional layer is followed by a batch normalization module and a linear rectified function. Starting from the second convolutional layer, each convolutional layer is downsampled with a stride greater than 1. The decoder structure is symmetrical to the encoder and includes several residual blocks and several convolutional layers. Each convolutional layer is followed by a linear rectified function. Except for the last convolutional layer, the remaining convolutional layers are upsampled with nearest neighbor. A batch normalization module is added after each convolutional layer. The structure of the real image discriminator includes several convolutional layers. Except for the first and last layers, each convolutional layer is followed by a batch normalization module and a linear rectified function. The first layer is followed only by a linear rectified function and downsampled with a stride greater than 1.

[0034] 5) Input the ink painting video sample frames into the real image generator to obtain the reconstructed real video frames, and compare the reconstructed real video frames with the real sample frames;

[0035] 6) Input the generated real image and its instance segmentation map into the ink painting stylization network model to obtain the reconstructed ink painting, and compare the reconstructed ink painting with the real ink painting.

[0036] 7) Through the discrimination and comparison in steps 3)-6), the obtained adversarial loss L is used. adv Circular reconstruction loss L cycle Consistency loss L cons Temporal loss L tmp White space loss L whiten Contour loss L contour Line loss L stroke and ink loss L ink Calculate the parameters of the ink painting style network model.

[0037] Furthermore, combating losses L adv =E x [log(1-D Y (F(x)))]+Ey [log(1-D X (B(y)))], where x and y are the results sampled from the real dataset and the ink painting dataset, respectively, and E x and E y D are the mathematical expectations calculated using the probability distributions of the two methods mentioned above. Y D is a device for identifying ink paintings. X F is the real image discriminator, B is the ink painting stylization network model, and C is the real image generator; L is the cyclic reconstruction loss. cycle =γE x [‖B(F(x))-x‖1]+E y [‖F(B(y))-y‖1], where γ is a coefficient used to balance the cyclic reconstruction loss L cycle The two parts; consistency loss L cons =L flow +L scale ,in y ω =ω(F(x),f),H ω =ω(E(x),f ↓ ), where ω(I,w) is the transformation of image I according to optical flow w, and f is the optical flow between a real sample frame and a sample reference frame. ↓ M is the result of downsampling the optical flow matrix to the size of the feature matrix. o For masking optical flow, Where H n The features extracted from the region containing the nth instance after magnification, where s is the size restoration operation and N is the number of instances; temporal loss. Where H′ represents the modified features of the real sample frame, {r1,…,r k} is the reference frame, {f1,…,f k} represents the optical flow between the corresponding reference frame and the current frame, {M o,1 ,…,M o,k} represents the masking mask corresponding to the optical flow, M o,c for Loss of white space Where M f For the foreground mask, M b M is the background mask. b =1-M f Contour loss Where Haar(I) represents the Haar wavelet transform of image I to extract its high-frequency components, and M... c The edge mask is set to 1 in the object contour region and 0 in other regions, calculated using the instance segmentation map; line loss. Where Edg is the edge extractor HED, T is the number of pixels in the image, and μ = T. - / T, where T - The sum of weights for non-edge points in the Edg detection results; ink loss L ink =E y [log D Y (Gauss(y))]+E x [log(1-D Y (Gauss(F(x))))], where Gauss represents the Gaussian blur operation.

[0038] A storage medium storing a computer program, wherein the computer program is configured to execute the method described above when run.

[0039] An electronic device includes a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer to perform the methods described above.

[0040] Compared with existing technologies, this invention improves the model's ability to generate ink painting style by adding instance segmentation input, adjusting feature extraction methods, and guiding training with an appropriate loss function. It achieves better results than existing technologies in terms of white space, entanglement of strokes of different objects, and handling of objects of different scales. By fusing features from multiple reference frames and modifying features from real video frames, it achieves referenced ink painting style and selects the conversion order and reference frames in an appropriate manner, avoiding error accumulation in traditional sequential conversion. Compared with existing methods, it reduces flickering and inconsistencies in the generated ink painting videos. Attached Figure Description

[0041] Figure 1 This is an overall framework diagram of the ink-wash styled network model used in the embodiments of the present invention.

[0042] Figure 2 This is a flowchart detailing the encoder in a network model for a single image, showcasing the ink-wash style.

[0043] Figure 3 This is a network structure diagram of the ink-wash styled network model used in the embodiments of the present invention.

[0044] Figure 4 This is a network structure diagram of the ink painting style discriminator, ink painting discriminator, real image generator, and real image discriminator used in the embodiments of the present invention.

[0045] Figure 5 This refers to the input video frame in this embodiment of the invention.

[0046] Figure 6 for Figure 5 The generated ink-wash video frames.

[0047] Figure 7 This is a comparison diagram of the conversion of a single image into an ink wash image using the present invention and existing methods.

[0048] Figure 8 This is a comparison diagram of the conversion of real video to ink wash video using the present invention and existing methods. Detailed Implementation

[0049] To make the above-mentioned features and advantages of the present invention more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings. It should be noted that the specific number of layers, modules, functions, and layer settings given in the following embodiments are merely preferred implementations and are not intended to limit the scope. Those skilled in the art can select the appropriate number and set different layers according to actual needs, which should be understood.

[0050] This embodiment discloses a method for generating ink-wash style videos based on instance segmentation and reference frames. It utilizes a training dataset to adversarially train an ink-wash style network model, an ink-wash style discriminator, an ink-wash painting discriminator, a real image generator, and a real image discriminator. The ink-wash style network model includes an encoder, a reference frame feature fusion unit, a feature modifier, and a decoder. The encoder comprises several consecutive convolutional layers and several residual blocks. Each convolutional layer is followed by a batch normalization module and a linear rectified function. Starting from the second convolutional layer, each convolutional layer is downsampled with a stride greater than 1. The reference frame feature fusion unit and the feature modifier each consist of a convolutional layer, several interleaved residual blocks and convolutional layers, and a single convolutional layer. The decoder has a structure symmetrical to the encoder, consisting of several residual blocks and several convolutional layers. Each convolutional layer is followed by a linear rectified function, and except for the last convolutional layer, nearest neighbor upsampling is performed before the remaining convolutional layers. A batch normalization module is added after the convolutional layers. The ink painting style discriminator, ink painting discriminator, and real image discriminator are all composed of several convolutional layers. Except for the first and last layers, each convolutional layer is followed by a batch normalization module and a linear rectified function. The first layer is followed by only a linear rectified function and downsampling is performed with a stride greater than 1. The real image generator consists of an encoder and decoder with the same structure as in the ink painting style network model.

[0051] Taking the conversion of a target video into a Chinese ink painting style video as an example, the specific explanation is as follows:

[0052] Step 1: Collect a large number of real videos, images selected from real videos, and a large number of randomly selected ink painting images. The photographic images form the real dataset, and the ink painting images form the ink painting dataset.

[0053] Step 2: Build a network model for ink painting stylization, an ink painting style discriminator, an ink painting discriminator, a real image generator, and a real image discriminator.

[0054] Network structure such as Figure 1 As shown in Figures 2, 3, and 4, the model consists of an encoder E and a reference frame feature fusion unit W in the ink painting stylization network model. r We define the feature modifier W, decoder G, and the entire ink painting styled network model as F. In addition, there is an ink painting style discriminator D. ink D, a device for identifying ink paintings Y Real image generator B, real image discriminator D X .

[0055] The encoder E consists of three consecutive convolutional layers and six residual blocks. Each convolutional layer is followed by a batch normalization module and a Rectified Linear Unit (ReLU). The second and third convolutional layers use convolutions with a stride of 2 for downsampling. In operation, the input frame and instance segmentation map are first concatenated along the channel dimension and then processed through this network to obtain the global features H. G Then, for each instance i and its corresponding region image x i , will x i The corresponding instance segmentation maps are concatenated along the channel dimensions and then enlarged to the input frame size. Features are then extracted using this network and the images are scaled back to the region image x. i The original size is denoted as instance feature H. i Finally, the output of this network is obtained by fusing the instance segmentation graphs. Where S i Let be the instance segmentation map of instance i. It is a single-channel 0 / 1 matrix with the same size as the original image, where 1 is in the region where instance i is located and 0 is in the rest of the region.

[0056] Reference Frame Feature Fuder W r It consists of one convolutional layer, five staggered residual blocks and convolutional layers, and finally one more convolutional layer. When used, the difference between each reference feature and the feature of the real video frame is input separately to generate a score matrix of the same size as the reference features. Then, a softmax transformation is applied to all reference feature scores at all positions. Using this as coefficients, all reference features are weighted and summed at each position to obtain the fused reference feature R. f .

[0057] Feature modifier W, its structure and W r The same applies; it receives the difference between the fused reference features and the real video frame features and scores it. This score is a matrix w of the same size as the fused reference features and the real video frame features. Based on the score, the fused reference features are used to modify the real video frame features to obtain the modified real video frame features. Where H′ represents the real video frame features obtained by the encoder.

[0058] Decoder G, whose structure is symmetrical to encoder, consists of 6 residual blocks and 3 convolutional layers. Each convolutional layer is followed by a linear rectified function, and except for the last convolutional layer, the remaining convolutional layers are subjected to nearest neighbor upsampling. This module is used to convert the modified real video frame features into ink painting style frames.

[0059] Realistic Image Generator B consists of 3 convolutional layers, 12 residual blocks, and then another 3 convolutional layers. Each convolutional layer is followed by a linear rectified function, with ReLU used in all but the last convolutional layer, and tanh used in the last convolutional layer. Furthermore, except for the first and last convolutional layers, each convolutional layer is followed by a batch normalization module; this module is used for converting ink wash images to realistic photographs.

[0060] Ink painting style discriminator D ink D, a device for identifying ink paintings Y Real Image Discriminator D X Each convolutional layer consists of 6 convolutional layers. Except for the first and last layers, each convolutional layer is followed by a batch normalization module and a LeakyReLU function. The first layer is followed only by a LeakyReLU function. The first convolutional layer uses a convolution with a stride of 2 for downsampling. These modules are used to determine whether the input image has an ink painting style, whether it belongs to an ink painting, and whether it belongs to a real photograph. They are used for adversarial training with the generative models F and B.

[0061] Step 3: Train the ink painting style network model F and the ink painting style discriminator D. ink D, a device for identifying ink paintings Y Real image generator B and real image discriminator D X Among them, the generative model F, B and the discriminative model D ink D Y D X Optimize alternately.

[0062] For generative models F and B, their loss function includes: adversarial loss L adv Circular reconstruction loss L cycle Consistency loss L cons Temporal loss L tmp White space loss L whiten Contour loss L contour Line loss L stroke And ink loss L ink .

[0063] Among them, L adv =Ex [log(1-D Y (F(x)))]+E y [log(1-D X (B(y)))],L cycle =γE x [‖B(F(x))-x‖1]+E y [‖F(B(y))-y‖1], where x and y are the results sampled from the real dataset and the ink painting dataset, respectively, and E x and E y D are the mathematical expectations calculated using the probability distributions of the two methods mentioned above. Y D is a device for identifying ink paintings. X F is the real image discriminator, B is the ink painting stylization network model, and γ is a coefficient used to balance L. cycle The loss function is divided into two parts. The purpose of this part is to guide the ink painting stylization network model to generate a realistic ink painting style and to guide the realistic image generator to generate a realistic photo style. The generation of the two is the inverse operation of each other.

[0064] L cons =L flow +L scale Part 1 Where y ω =ω(F(x),f),H ω =ω(F(x),f ↓ ), ω(I,w) means transforming image I according to optical flow w, f is the optical flow between a sampled video frame and another frame, f ↓ M is the result of downsampling the optical flow matrix to the size of the feature matrix. o This is a masking layer for the optical flow. L flow The aim is to guide the model in establishing optical flow consistency between the features learned by the encoder and the ink-wash image generated by the decoder. Another part... Where H n L represents the features extracted from the region containing the nth instance mentioned in the encoder introduction after magnification, where s is the size restoration operation. scale The aim is to guide the model to establish scale consistency between the features learned by the encoder and the ink painting images generated by the decoder.

[0065]

[0066] Where H′ represents the modified real video frame features, {r1,…,r k} is the reference frame, {f1,…,f k} represents the optical flow between the corresponding reference frame and the current frame, {M o,1 ,…,Mo,k} represents the masking mask corresponding to the optical flow, M o,c for The purpose of this loss function is to guide the model to generate an ink painting image consistent with the reference frame in regions where optical flow is reliable, and to generate an image consistent with the image without a reference frame in the remaining regions.

[0067] Where M f Foreground mask, directly generated from instance style results, M b M is the background mask. b =1-M f The purpose of this loss function is to guide the model to leave blank spaces in the background area, thus applying more ink wash to the foreground.

[0068] Where Haar(I) represents the Haar wavelet transform of image I to extract its high-frequency components, and M... c The edge mask is set to 1 in the object contour region and 0 in other regions, and is calculated using the instance segmentation map. The purpose of this loss function is to mitigate the entanglement of strokes between different objects.

[0069] Where Edg is the edge extractor HED, T is the number of pixels in the image, and μ = T. - / T, where T - This is the sum of the weights of non-edge points in the Edg detection results. This coefficient is used to balance the losses for edge and non-edge points. The purpose of this loss function is to constrain the generated ink wash image to have similar lines and contours to the original image.

[0070] L ink =E y [log D Y (Gauss(y))]+E x [log(1-D Y [(Gauss(F(x))))], where Gauss represents the Gaussian blur operation. The purpose of this loss function is to optimize the ink-wash texture of the generated ink-wash image.

[0071] For discriminant model D X D Y D ink In other words, its losses include L adv_X =E y [log(1-D Y (y))]+E x [log D Y (F(x))],L adv_Y =E x [log(1-D X(x))]+E y [log D X (B(y))] and L adv_ink =E y [log(1-D ink (Gauss(y)))]+E x [log D ink [Gauss(F(x))], the purpose of this loss function is to guide the discriminative model to distinguish between generated ink painting images and real ink painting images; to guide the discriminative model to distinguish between generated real images and real images; and to guide the discriminative model to distinguish between generated ink painting style and real ink painting style.

[0072] The training steps include:

[0073] Instance segmentation (marking the regions where different objects are located) and optical flow between real video frames and reference frames are calculated and input into the ink painting stylization network model to obtain a transformed ink painting video frame.

[0074] The generated ink painting video frames and ink painting images are simultaneously input into an ink painting discriminator and an ink painting style discriminator for discrimination, and the loss (L) is calculated. adv and L ink This is used for adversarial training between the discriminator and the generator, thereby making the generated results more realistic.

[0075] Inputting a traditional Chinese ink painting into a realistic image generator produces a real-life image.

[0076] The real video frames and the generated real images are input into the real image discriminator for discrimination.

[0077] The generated ink-wash video frames are input into the real image generator to obtain the reconstructed real video frames;

[0078] The reconstructed real video frames are compared with the input real video frames, and the loss (L) is calculated. cycle This ensures that the reconstructed real video frame is consistent with the original, thus ensuring that the image content is not lost during the generation process. In other words, the generated ink-wash video frame and the input real video frame have the same content structure.

[0079] The generated real images and their instance segments are input into the ink painting stylization network model to obtain the reconstructed ink painting;

[0080] The reconstructed ink painting is compared with the input ink painting, and the loss (L) is also calculated. cycle This ensures consistency of content during the generation process.

[0081] Step 4: Reasoning stage, input the actual video footage (see...) Figure 5) Calculate its instance segmentation and perform pairwise optical flow estimation across all frames, ultimately outputting the desired ink-wash style video (see Figure 6 For ease of demonstration, the image only shows some frames from the video, including:

[0082] 1) Calculate optical flow between all frames, and measure the similarity between two frames by the magnitude of the optical flow between them;

[0083] 2) Select the frame that is least similar to all converted frames and most similar to all other unconverted frames, and use it as the real video frame for that round;

[0084] 3) Select several converted frames that are most similar to the selected real video frames as reference frames. In special cases, there are no reference frames in the first round.

[0085] 4) The features of the reference frame used in the decoder of the ink painting stylization network model are used as reference features, and together with the selected real video frame, the instance segmentation of the real video frame, and the optical flow between the real video frame and the reference frame, they are input into the ink painting stylization network model to obtain a corresponding ink painting video frame. In the absence of a reference frame, the features extracted from the real video frame will be directly input into the decoder of the ink painting stylization network to generate the corresponding ink painting frame.

[0086] 5) Return to step 2) until all frames have been converted, and finally assemble the converted frames into an ink painting video in the previous order.

[0087] Experimental data:

[0088] We compared our work with existing methods for converting single images into ink-wash style images (see...). Figure 7 ChipGAN is an existing image ink painting method, while AdaIN, WCT, and ChipGAN are existing image stylization methods. On a single image, we can achieve better ink painting effects than existing methods. We also compared our results with existing image ink painting and video stylization methods, converting real video to ink painting video (see...). Figure 8 The adjacent columns are magnified views of the video frames and their corresponding regions. ChipGAN is an existing image ink painting method, which we apply to each frame of the video to perform video ink painting. Linear and Compound are existing video stylization methods. It can be seen that our method can achieve both excellent ink painting style and stable video effect compared to existing methods.

[0089] It is easy to understand that the specific structures of each network in this method, such as the number of convolutional layers, the types of nonlinear activation functions, and the regularization methods, can be replaced by other structures; the optical flow estimation method, instance segmentation method, and edge detection method used in this method can be replaced by other methods with the same or similar functions.

[0090] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention. The scope of protection of the present invention should be determined by the claims.

Claims

1. A method for generating ink-wash video based on instance segmentation and reference frames, comprising the following steps: 1) Obtain the instance segmentation map of each real video frame in the real video, and calculate the similarity between any two real video frames based on the optical flow estimation between all real video frames. 2) Based on the similarity, select the real video frame to be converted and one or more reference frames for each round, and input the real video frame to be converted and the corresponding reference frames into the ink painting stylization network model. The ink painting video frame of the real video frame to be converted is obtained through the following steps: 2.1) Extract real video features and corresponding reference features based on the real video frames to be converted, each reference frame and the corresponding instance segmentation map, and align each reference feature with the real video features using optical flow estimation. 2.2) Based on the difference between each reference feature and the real video frame features, calculate the reference feature score of each reference feature, and fuse all reference features through the reference feature scores to obtain the fused reference feature; 2.3) Decode the modified real video features obtained from the real video features and the fused reference features to obtain the ink-wash video frame of the real video frame to be converted. 3) Repeat step 2) to generate the ink wash video of the real video.

2. The method as described in claim 1, characterized in that, The following steps are used to select the real video frames to be converted and several reference frames for each round: 1) Based on the similarity level, select the frame with the lowest similarity to all converted frames and the highest similarity to all unconverted frames as the real video frame to be converted in this round; 2) Based on the similarity, select several converted frames that have the highest similarity to the real video frames to be converted as reference frames. When the first ink painting video frame is generated, the number of reference frames is 0.

3. The method as described in claim 1, characterized in that, The following steps are used to obtain real video features: 1) Combine real video frames and corresponding instance segmentation maps, encode and stitch the results to obtain global features; 2) For each instance i and its corresponding region image x in each instance segmentation map. i , region image x i After being stitched together with the corresponding instance segmentation map in the channel dimension, the image is enlarged to the image size of the real video frame to be converted; 3) Extract image features from the magnified image and shrink the image features back to the region image x. i The feature size is used to obtain the instance feature H. i ; 4) Integrate global features with instance features H i This allows us to obtain the true characteristics of the video.

4. The method as described in claim 1, characterized in that, The fusion reference features are obtained through the following steps: 1) Apply the softmax transform to each reference feature score at each position to obtain the corresponding coefficients; 2) The reference features are weighted and summed at each position according to the coefficients to obtain the fused reference features.

5. The method as described in claim 1, characterized in that, The modified real video features are obtained through the following steps: 1) The difference between the fused reference features and the real video frame features is scored to obtain the fused reference feature score; 2) Use the weighted summation of fused reference feature scores to modify the features of real video frames, and obtain the modified real video features.

6. The method as described in claim 1, characterized in that, The structure of the ink painting stylized network model includes: an encoder, a reference frame feature fusion unit, a feature modifier, and a decoder. The encoder consists of several convolutional layers and several residual blocks, with each convolutional layer followed by a batch normalization module and a linear rectified function. Starting from the second convolutional layer, each convolutional layer is downsampled with a stride greater than 1. The reference frame feature fusion unit and the feature modifier each consist of: a convolutional layer, several staggered residual blocks and convolutional layers, and a single convolutional layer. The decoder is symmetrical to the encoder, consisting of several residual blocks and several convolutional layers, with each convolutional layer followed by a linear rectified function. Except for the last convolutional layer, the remaining convolutional layers are upsampled to nearest neighbors, and a batch normalization module is added after each convolutional layer.

7. The method as described in claim 1, characterized in that, The ink painting stylization network model F is trained using the following steps: 1) Collect several sets of training data for iterative training. Each set of training data includes: a real sample video, a real sample frame taken from the real sample video, several sample reference frames taken from the real sample video, and a randomly selected real ink painting. 2) Obtain the instance segmentation map of the real sample frame and each sample reference frame, calculate the optical flow of the real sample frame and each sample reference frame, and input the real sample frame, each sample reference frame, and the corresponding instance segmentation map and optical flow of the real sample frame and each sample reference frame into the ink painting stylization network model to obtain an ink painting video sample frame. 3) Input the sample frames of the ink painting video and the real ink painting into an ink painting discriminator and an ink painting style discriminator for discrimination. The structure of the ink painting discriminator and the ink painting style discriminator includes: several convolutional layers. Except for the first and last layers, each convolutional layer is followed by a batch normalization module and a linear rectified function. The first layer is followed by only a linear rectified function and downsampling is performed with a stride greater than 1. 4) Input the real ink painting into the real image generator to obtain the generated real image. Then, input the real video sample frames and the generated real image into a real image discriminator for discrimination. The structure of the real image generator includes an encoder and a decoder. The encoder structure includes several convolutional layers and several residual blocks. Each convolutional layer is followed by a batch normalization module and a linear rectified function. Starting from the second convolutional layer, each convolutional layer is downsampled with a stride greater than 1. The decoder structure is symmetrical to the encoder and includes several residual blocks and several convolutional layers. Each convolutional layer is followed by a linear rectified function. Except for the last convolutional layer, the remaining convolutional layers are upsampled with nearest neighbor. A batch normalization module is added after each convolutional layer. The structure of the real image discriminator includes several convolutional layers. Except for the first and last layers, each convolutional layer is followed by a batch normalization module and a linear rectified function. The first layer is followed only by a linear rectified function and downsampled with a stride greater than 1. 5) Input the ink painting video sample frames into the real image generator to obtain the reconstructed real video frames, and compare the reconstructed real video frames with the real sample frames; 6) Input the generated real image and its instance segmentation map into the ink painting stylization network model to obtain the reconstructed ink painting, and compare the reconstructed ink painting with the real ink painting. 7) Through the discrimination and comparison in steps 3)-6), the obtained adversarial loss L is used. adv Circular reconstruction loss L cycle Consistency loss L cons Temporal loss L tmp White space loss L whiten Contour loss L contour Line loss L stroke and ink loss L ink Calculate the parameters of the ink painting style network model.

8. The method as described in claim 7, characterized in that, Combat loss L adv =E x [log(1-D Y (F(x)))]+E y [log(1-D X (B(y)))], where x and u are the results sampled from the real dataset and the ink painting dataset, respectively, and E x and E u These are the mathematical expectations calculated using the probability distributions of the two methods mentioned above, D. Y D is a device for identifying ink paintings. X F is the real image discriminator, B is the ink painting stylization network model, and C is the real image generator; L is the cyclic reconstruction loss. cycle =γE x [‖B(F(x))-x‖1]+E y [‖F(B(y))-y‖1], where γ is a coefficient used to balance the cyclic reconstruction loss L cycle The two parts; consistency loss L cons =L flow +L scale ,in y ω =ω(F(x),f),H ω =ω(E(x),f ↓ ), where ω(I,w) is the transformation of image I according to optical flow w, and f is the optical flow between a real sample frame and a sample reference frame. ↓ M is the result of downsampling the optical flow matrix to the size of the feature matrix. o For masking the optical flow, Where H n The features extracted from the region containing the nth instance after magnification, where s is the size restoration operation and N is the number of instances; temporal loss. Where H′ represents the modified features of the real sample frame, {r1,…,r k } is the reference frame, {f1,…,f k } represents the optical flow between the corresponding reference frame and the current frame, {M o,1 ,…,M o,k } represents the masking mask corresponding to the optical flow, M o,c for Loss of white space Where M f For the foreground mask, M b M is the background mask. b =1-M f Contour loss Where Haar(I) represents the Haar wavelet transform of image I to extract its high-frequency components, and M... c The edge mask is set to 1 in the object contour region and 0 in other regions, calculated using the instance segmentation map; line loss. Where Edg is the edge extractor HED, T is the number of pixels in the image, μ = T_ / T, where T_ is the sum of the weights of non-edge points in the Edg detection result; ink loss L ink =E y [logD Y (Gauss(y))]+E x [log(1-D Y (Gauss(F(x))))], where Gauss represents the Gaussian blur operation.

9. A storage medium storing a computer program, wherein, The computer program is configured to execute the method of any one of claims 1-8 at runtime.

10. An electronic device comprising a memory and a processor, the memory storing a computer program, the processor being configured to run the computer program to perform the method as claimed in any one of claims 1-8.

Images