End-to-end video style transfer method, system and storage medium for suppressing temporal noise amplification
By constructing a video style migration network model, using multi-level attention feature migration module and self-similarity loss, the problem of time domain noise amplification in the prior art is solved, and the time consistency and style effect of stylized videos are improved.
Patent Information
- Application Number
- CN202210241789.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-11
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2042-03-11
AI Technical Summary
Existing video style transfer methods are difficult to effectively deal with time domain noise amplification, resulting in poor time consistency of stylized videos, especially in videos with simple backgrounds.
Using end-to-end video style migration method, by obtaining image and style training sets, a video style migration network model is built, including content encoder, style encoder, multi-level attention feature migration module and decoder, using self-similarity loss and Reconstruct loss-constrained time domain noise to improve the time consistency of stylized videos.
The time consistency improvement and satisfactory style effect of stylized videos are achieved. The time domain noise amplification is effectively suppressed through multi-level attention feature migration module and self-similarity loss. The generated stylized video is more in line with the characteristics of the human eye visual system. .
Smart Images

Figure CN114692733B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to an end-to-end video style transfer method, system and storage medium for suppressing temporal noise amplification. Background Art
[0002] Artistic images are visually appealing and impressive. In the past, creating artistic images required hours of effort by experts. Style transfer technology can automatically transform real-world scene images / videos into works of art. Nowadays, style transfer has been widely used in augmented reality and animation production. Using this technology, production costs can be reduced and production time can be saved.
[0003] With the development of technology, the performance and speed of existing video style transfer methods have been greatly improved. However, most existing methods cannot handle stylized video flickering caused by temporal noise amplification of the source video well.
[0004] For video sequences with a small amount of temporal noise, the video style transfer network tends to amplify the temporal noise in the input video, seriously affecting the quality of the stylized video. Due to the persistence effect of the human eye, this instability is not easy to be observed in stylized videos with complex backgrounds and large movements, but is more easily captured by the human eye in videos with simple backgrounds and single scenes. Summary of the invention
[0005] In order to overcome the technical defects of the prior art, the present invention provides an end-to-end video style transfer method, system and storage medium for suppressing temporal noise amplification. The present invention not only enhances the temporal consistency of the stylized video, but also generates a satisfactory stylization effect.
[0006] The present invention adopts the following technical solution:
[0007] An end-to-end video style transfer method for suppressing temporal noise amplification includes the following:
[0008] Obtain an image training set, obtain two consecutive frames of images and randomly generated optical flow information;
[0009] Acquire a style training set, wherein the style training set includes artistic painting style images;
[0010] Constructing a video style transfer network model, wherein the video style transfer network model includes a content encoder, a style encoder, a multi-level attention feature transfer module and a decoder;
[0011] The generated two-frame image and style image are input into the video style transfer network model for training to obtain a trained video style transfer model;
[0012] Input any test video and style image into the trained video style transfer model to obtain a stylized video.
[0013] Furthermore, the two frames of images and the style image generated are input into the video style transfer network model for training to obtain a trained video style transfer model, specifically:
[0014] Two consecutive frames of images are input into the content encoder to obtain content features;
[0015] Input the style image into the style encoder to obtain the style features;
[0016] The content features and style features are input into the multi-level attention feature transfer module, and its output is connected to the decoder to finally output the stylized result;
[0017] Calculate the total loss value of the stylized result, and use the total loss value to train the video style transfer network model until all the video frames in the training set are taken to obtain a trained video style transfer model.
[0018] Furthermore, the content encoder and the style encoder are both VGG19 networks, including several convolutional layers, each of which is connected to a ReLU function, and downsampling is performed using maximum pooling.
[0019] Furthermore, the multi-level attention feature transfer module includes a style attention module and a multi-head attention module.
[0020] Furthermore, the decoder includes a plurality of adaptive instance normalization layers, upsampling layers, residual blocks and convolutional layers.
[0021] Furthermore, the total loss value is calculated as follows:
[0022] L=λ s L s +λ c L c +λ temp L temp +λ ss L ss +λ recon L recon +λ tv L tv ,
[0023] Among them, λ s , c , temp , ss , recon and λ tv is the weight term, L s is the style loss value, L c is the content loss value, Ltemp is the time domain continuity loss value, L ss is the self-similarity loss value, L recon is the reconstruction loss value, L tv is the total variation loss value.
[0024] Furthermore, the self-similarity loss value L ss , by obtaining the temporal error grayscale image of the source video frame and the temporal error grayscale image of the corresponding frame of the stylized video, and calculating the structural similarity between them to obtain L ss .
[0025] Furthermore, the reconstruction loss value L recon L is obtained by calculating the mean absolute error and structural similarity between the image reconstructed by the model and the original image. recon .
[0026] A system based on the video style transfer method, comprising:
[0027] Image training set module: used to obtain image training sets, obtain two consecutive frames of images and randomly generated optical flow information;
[0028] Segmentation training set module: used to obtain style training set;
[0029] The video style transfer network model inputs style images and continuous frame images to obtain stylized videos.
[0030] A storage medium stores a computer program, which implements the video style transfer method when executed by a processor.
[0031] Beneficial effects of the present invention:
[0032] (1) The present invention can achieve good style transfer by matching the semantics of multi-level content features and style features through a multi-level attention feature transfer module;
[0033] (2) The present invention proposes a self-similarity loss to constrain the temporal noise between adjacent frames of the stylized video, suppress the temporal noise in the source video that is amplified by the video style transfer model, and improve the temporal consistency of the stylized video;
[0034] (3) The present invention proposes a new reconstruction loss to make the image reconstructed by the model more consistent with the characteristics of the human visual system. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 It is a work flow chart of the present invention;
[0036] Figure 2 It is a structural schematic diagram of the video style transfer network model of the present invention. DETAILED DESCRIPTION
[0037] The present invention will be further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.
[0038] Example 1
[0039] like Figure 1 As shown, an end-to-end video style transfer method for suppressing temporal noise amplification includes the following steps:
[0040] S1 obtains the image training set and obtains two consecutive frames of images and randomly generated optical flow information;
[0041] Specifically, obtain the image training set C, image I t As the t-th frame image, according to the randomly generated optical flow information, the t-th frame I is distorted t , generate the t+1th frame I t+1 , thus obtaining two consecutive frames of images and standard optical flow information.
[0042] S2 obtains a style training set, which is mainly composed of artistic painting style images. Other styles can also be replaced according to user needs, such as ink painting, sketching, abstraction and other styles that the user considers.
[0043] S3 builds a video style transfer network model. The network structure is as follows Figure 2 As shown in the figure, it includes a content encoder (Encoder), a style encoder (EncoderStyle), a multi-level attention feature transfer module (Multi-levelAttention Transform Module) and a decoder (Decoder). Both the content encoder and the style encoder are based on the VGG19 network, including 9 convolutional layers, each of which is followed by a ReLU function and uses maximum pooling for downsampling.
[0044] The multi-level attention feature transfer module includes a style attention module and a multi-head attention module. The style attention module inputs the content features and style features of the previous layer output, and the multi-head attention module divides the previous layer output into four groups of sub-features according to the number of channels, and then inputs them into four heads respectively. The output of the style attention module and the output of the multi-head attention are fused to obtain the result of the multi-level attention feature transfer module.
[0045] The decoder includes 4 adaptive instance normalization layers, 3 upsampling layers and residual blocks, and ends with a convolutional layer to obtain a stylized output result.
[0046] S4 training video style transfer model
[0047] Input the video frame into the content encoder to obtain content features;
[0048] Input the style image into the style encoder to obtain the style features;
[0049] Input content features and style features into the multi-level attention feature transfer module;
[0050] The output of the feature transfer module is output to the decoder. The style features of different layers are sequentially input to the adaptive instance normalization layer through skip connections, and the final output is obtained through the last layer of convolution.
[0051] Calculate the total loss value L of the stylized result:
[0052] The video style transfer network model is trained using the total loss value L until all video frames in the training set are taken to obtain a trained video style transfer model.
[0053] The total loss function is L = λ s L s +λ c L c +λ temp L temp +λ ss L ss +λ recon L recon +λ tv L tv ,
[0054] Among them, λ s , c , temp , ss , recon and λ tv is the weight term, usually set to λ comp =60,λ ss =60,λ s =20,λ c =1,λ recon =20,λ tv =20.
[0055] L s is the style loss value, L c is the content loss value, L comp is the time domain continuity loss value, L ss is the self-similarity loss value, L temp is the reconstruction loss value, L tv is the total variation loss value;
[0056] Style loss term L s , calculated by the following formula:
[0057]
[0058]
[0059] represents the feature extracted from the i-th layer in the pre-trained VGG19, μ represents the mean of the feature, and σ represents the variance of the feature. s Represents the style image, I cs Represents the style transfer result graph, using ReLU1_1, ReLU2_1, ReLU3_1, and ReLU4_1 layers to calculate the style loss.
[0060] Content loss term L c , calculated by the following formula:
[0061]
[0062] represents the features extracted from the i-th layer in the pre-trained VGG19, I c Represents the content image, I cs Represents the style transfer result map, using ReLU4_1 to calculate the content loss.
[0063] Time domain continuity loss term L temp , calculated by the following formula:
[0064]
[0065] F represents the style transfer model, represents random optical flow, X m represents the nth frame image, Δ represents local jitter or noise, and S represents the style image.
[0066] Self-similarity loss term L ss , by obtaining the temporal error grayscale image of the source video frame and the temporal error grayscale image of the corresponding frame of the stylized video, and calculating the structural similarity between them to obtain L ss .
[0067] Reconstruction loss term L recon , by calculating the mean absolute error and structural similarity between the image reconstructed by the network and the original image, we can get L recon .
[0068] Total variation loss term L tv , calculated by the following formula:
[0069] L tv =||δ x (I cs )-δ y (I cs)||
[0070] δ x represents the horizontal residual, δ y Represents the residual in the vertical direction.
[0071] The specific training process is:
[0072] Each time, select an image I from the content training set C t , generate a random optical flow F with the same length and width as the target content image, and use the random optical flow to I t Distortion Generation I t+1 ; Input the training sample into the style transfer network to generate the stylized result (I′ t , I′ t+1 );Calculate the total loss value of the stylized result: L = λ s L s +λ c L c +λ temp L temp +λ ss L ss +λ recon L recon +λ tv L tv ,
[0073] Among them, λ s , c , temp , ss , recon and λ tv is the weight term, L s is the style loss value, L c is the content loss value, L temp is the time domain continuity loss value, L ss is the self-similarity loss value, L recon is the reconstruction loss value, L tv is the total variation loss value;
[0074] In the S5 test phase, video frames and arbitrary style images are input into the found model, and the model will automatically synthesize stable stylized videos.
[0075] Example 2
[0076] A video style transfer system, comprising:
[0077] Image training set module: used to obtain image training sets, obtain two consecutive frames of images and randomly generated optical flow information;
[0078] Segmentation training set module: used to obtain style training set;
[0079] The video style transfer network model inputs the style image and continuous frame images to obtain the stylized video.
[0080] The video style transfer network model includes a content encoder, a style encoder, a multi-level attention feature transfer module and a decoder.
[0081] Example 3
[0082] A storage medium stores a computer program, which implements the video style transfer method when executed by a processor.
[0083] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.
Claims
1. An end-to-end video style transfer method for suppressing temporal noise amplification, characterized in that: The process includes: obtaining an image training set, obtaining two consecutive frames of images and randomly generated optical flow information; Acquire a style training set, wherein the style training set includes artistic painting style images; Constructing a video style transfer network model, wherein the video style transfer network model includes a content encoder, a style encoder, a multi-level attention feature transfer module and a decoder; The generated two-frame image and style image are input into the video style transfer network model for training to obtain the trained video style transfer model: Two consecutive frames of images are input into the content encoder to obtain content features; Input the style image into the style encoder to obtain the style features; The content features and style features are input into the multi-level attention feature transfer module, and its output is connected to the decoder to finally output the stylized result; Calculate the total loss value of the stylized result, and use the total loss value to train the video style transfer network model until all video frames in the training set are taken to obtain a trained video style transfer model; The total loss value is calculated as follows: L=λ s L s +λ c L c +λ temp L temp +λ ss L ss +λ recon L recon +λ tv L tv , where λ s , c , temp , ss , recon and λ tv is the weight term, L s is the style loss value, L c is the content loss value, L temp is the time domain continuity loss value, L ss is the self-similarity loss value, L recon is the reconstruction loss value, L tv is the total variation loss value; The self-similarity loss value L ss , obtained by obtaining the temporal error grayscale image of the source video frame and the temporal error grayscale image of the corresponding frame of the stylized video, and calculating the structural similarity between them; Input any test video and style image into the trained video style transfer model to obtain a stylized video.
2. The video style transfer method according to claim 1, characterized in that: The content encoder and style encoder are both VGG19 networks, including several convolutional layers, each of which is connected to a ReLU function and down-sampling is performed using maximum pooling.
3. The video style transfer method according to claim 1, characterized in that: The multi-level attention feature transfer module includes a style attention module and a multi-head attention module.
4. The video style transfer method according to any one of claims 1 to 3, characterized in that: The decoder includes several adaptive instance normalization layers, upsampling layers, residual blocks and convolutional layers.
5. The video style transfer method according to claim 1, characterized in that: Self-similarity loss value L ss , by obtaining the temporal error grayscale image of the source video frame and the temporal error grayscale image of the corresponding frame of the stylized video, and calculating the structural similarity between them to obtain L ss .
6. The video style transfer method according to claim 1, characterized in that: Reconstruction loss value L recon L is obtained by calculating the mean absolute error and structural similarity between the image reconstructed by the model and the original image. recon .
7. A system based on the video style transfer method according to any one of claims 1 to 6, characterized in that: include: Image training set module: used to obtain image training sets, obtain two consecutive frames of images and randomly generated optical flow information; Segmentation training set module: used to obtain style training set; The video style transfer network model inputs style images and continuous frame images to obtain stylized videos.
8. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video style transfer method according to any one of claims 1 to 6 is implemented.
