A progressive multi-branch video style transfer method based on confidence-weighted loss

Abstract

The present application relates to the field of video image technology. The present application provides a progressive multi-branch video style transfer method based on confidence weighted loss. The technical solution steps designed by the present invention include: S10: receiving a stylized image I s and content frame (I t‑1 , I t , I t+1 ) and input into the VGG19 neural network. The style encoder and content encoder based on the VGG19 neural network extract the style features F s and content feature F c ; S20: The style feature F s and content feature F c Feature capture is performed based on MBSTS to obtain new style features F sN and new content features F cN And fusion output fusion feature F cSN , the fusion feature F cSN The final stylized feature F is output by splicing along the feature map channel direction cs ; S30: the final stylized feature F cs The decoder based on the VGG19 neural network decodes and outputs the video image. This application uses a video style transfer network with a multi-layer branch semantic transfer structure to effectively minimize the visual flicker and discontinuity in the video stylization process by reducing the stylization error between frames.

Description

Technical Field

[0001] The present invention belongs to the field of video image processing, and in particular relates to a progressive multi-branch video style migration method based on confidence weighted loss. Background Art

[0002] Style transfer technology plays an important role in the field of computer vision. It can convert images into various artistic styles, thus enriching the diversity of visual expression. It plays a key role in many fields such as augmented reality, advertising, and entertainment. With the popularity of short videos, video style transfer technology has received unprecedented attention. The core of video style transfer is to transfer the content style of a video to another specific style. This is not only a re-creation of visual art, but also an innovation in the way content is expressed. Through this technology, it is easy to give videos new visual effects, whether it is imitating the color tone of classic movies or incorporating elements of modern art, it can bring a new visual experience to the audience.

[0003] The problem of temporal and content structure consistency is the biggest technical challenge in transferring the style of static images to video content. Although the traditional frame-by-frame style transfer method is simple, it is difficult to ensure the coherence of the video in the temporal dimension. It ignores the consistency between frames and will cause serious flickering artifacts when the video is played. To solve this problem, scholars introduced optical flow constraints and temporal consistency losses to optimize the style transfer process. At the same time, feedforward networks and temporal regularization techniques have also been developed to enhance the stability of video style transfer. Some scholars have proposed multi-channel correction networks, while others have ensured the consistency of style transfer by adjusting the self-attention mechanism. Although the above methods all attempt to improve temporal consistency by considering the coherence between frames, the problem of video flickering still exists when the input video contains temporal noise. Some scholars have adopted a temporal motion compensated bilateral filter. This method uses traditional denoising technology and requires analysis of multiple video frames and a lot of calculations. It improves the current frame by copying pixels from the static area of ​​the previous frame and calculates the mean square error between adjacent frames, but this method lacks versatility and requires resetting the threshold for each frame to detect errors. In addition, it is difficult to maintain the consistency of content structure between modes during video style transfer. If not handled properly, the converted video may have incoherent styles, making the overall visual effect appear fragmented or unnatural. Summary of the invention

[0004] In view of the above-mentioned defects of the prior art, the present invention proposes a progressive multi-branch video style transfer method based on confidence-weighted loss. The technical solution steps designed by the present invention include:

[0005] S10: Receive stylized image I s and content frame (It-1 , I t , I t+1 ) and input into the VGG19 neural network. The style encoder and content encoder based on the VGG19 neural network extract the style features F s and content feature F c ;

[0006] S20: The style feature F s and content feature F c Feature capture is performed based on MBSTS to obtain new style features F sN and new content features F cN And fusion output fusion feature F cSN , the fusion feature F cSN The final stylized feature F is output by splicing along the feature map channel direction cs ;

[0007] S30: The final stylized feature F cs The decoder based on the VGG19 neural network decodes and outputs the video image.

[0008] Preferably, the new style feature F is obtained in S20 sN and new content features F cN include:

[0009] The MBSTS includes a feature extraction layer and a feature fusion layer;

[0010] The feature extraction layer includes a 1×1 convolution layer, three progressive modules and a pooling layer, and the three progressive modules include PM1, PM2 and PM3;

[0011] The style feature F s and content feature F c Input feature extraction layer to capture features and obtain new style features F sN and new content features F cN , the formula is as follows:

[0012] F cN =Branch_N(F c )

[0013] F sN =Branch_N(F s )

[0014] In the formula, Branch_N() represents the feature extraction layer, N = 1, 2, 3, 4, 5.

[0015] Preferably, the fusion output fusion feature F in S20 cSN , the formula is as follows:

[0016] F csN =CSAM(F cN , F sN )

[0017] Where CSAM() represents the channel space attention module CSA of the feature fusion layer.

[0018] Preferably, the output of the final stylized feature F in S20 cs , the formula is as follows:

[0019] F cs =f 1×1 (cat(F cs1 , …, F csN ))

[0020] In the formula, f 1×1 represents a 1×1 convolutional layer, and cat represents concatenation along the channel direction of the feature map.

[0021] Preferably, the S30 includes:

[0022] The decoder of the VGG19 neural network includes the main decoder D m and auxiliary decoder D a , the main decoder D m and auxiliary decoder D a Alternate training, shared input fusion feature F cs and style features F s , auxiliary decoder D a Generate output O auxiliary , main decoder D m Output main frame O main and abnormal frame O minor , the formula is as follows:

[0023] [O main , O minor ]=D m (F cs , F s )

[0024] O auxiliary =D a (F cs , F s )

[0025] In the formula, [O main , O minor ] is the output video image.

[0026] Preferably, the auxiliary decoder D a The loss function of the training process is as follows:

[0027] Lauxiliary =λ s L s +λ c L c +λ temp Ltenp+λ SSL L SSL +λ SSRL L SSRL +λ recon L recon +λ tv L tv

[0028] In the formula, λ s , c , temp , SSL , recon and λ tv is the weight term, L s is the style loss value, L c is the content loss value, L comp is the time domain continuity loss value, L SSL is the self-similarity loss value, L temp is the reconstruction loss value, L tv is the total variation loss value.

[0029] Preferably, the main decoder D m The loss function of the training process is as follows:

[0030] L main =λ s L s +λ c L c +λ cr L cr .

[0031] Preferably, the progressive multi-branch video style transfer method based on confidence-weighted loss also includes:

[0032] Setting a confidence-weighted loss function on the output video image to select a primary mode for each pixel based on the video image in the plurality of modes and ignoring the secondary mode;

[0033] The setting of the confidence weighted loss function includes calculating a confidence map of the main mode, and updating the confidence weighted loss function based on the confidence map of the main mode.

[0034] Preferably, the confidence map of the main mode is calculated using the following formula:

[0035]

[0036] In the formula, O main is the main frame, Ominor is the outlier frame, O auxiliary is an auxiliary frame, d1 and d2 are respectively main With O auxiliary and O minor With O auxiliary For the distance in relative space, ε is set to 0.1 and 6 is set to 0.01.

[0037] Preferably, the confidence map based on the main mode updates the confidence weighted loss function formula as follows:

[0038] d′1=L data (C t ·O main , C t ·O auxiliary )

[0039] d′2=L data ((1-C t )·O minor , (1-Ct ) ·O auxiliary )

[0040] L cr =d′1+αd′2

[0041] In the formula, d′2 and d′2 represent O main Upper O main and O minor The distance relationship between projections, L data is the L1 Manhattan distance, and α is the preset value.

[0042] Beneficial effects:

[0043] 1. The present invention can generate stable and natural style videos under the conditions of high-speed motion, focus adjustment, brightness change and complex style, can effectively maintain the content structure of video frames, and suppress the generation of random artifacts that cause frame flickering. It is significantly superior to other existing video style transfer methods in terms of temporal stability and consistency;

[0044] 2. The present invention produces higher visual quality and lower temporal error scores in stylized video metrics. Experiments on a variety of complex styles and video visualizations show that smoother video content is generated with more natural overall consistency;

[0045] 3. The proposed method produces lower optical flow alignment errors and obtains the highest subjective score in the AST task, which is supported by both quantitative and qualitative results. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic diagram of the network structure of a preferred embodiment of the present invention;

[0047] Figure 2 It is a schematic diagram of the MBSTS feature extraction layer of a preferred embodiment of the present invention;

[0048] Figure 3 It is a schematic diagram of the MBSTS feature fusion layer of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0049] The embodiments of the present invention are described in detail below. The following embodiments are implemented on the premise of the technical solution of the present invention, and detailed implementation methods and specific operation processes are given, but the protection scope of the present invention is not limited to the following embodiments.

[0050] The present invention designs a progressive multi-branch video style transfer method based on confidence weighted loss, such as Figure 1 As shown, the technical solution includes the following steps, specifically including:

[0051] S10: Receive stylized image I s and content frame (I t-1 , I t , I t+1 ) and input into the VGG19 neural network. The style encoder and content encoder based on the VGG19 neural network extract the style features F s and content feature F c ;

[0052] S20: The style feature F s and content feature F c Feature capture is performed based on MBSTS to obtain new style features F sN and new content features F cN And fusion output fusion feature F cSN , the fusion feature F cSN The final stylized feature F is output by splicing along the feature map channel direction cs ;

[0053] S30: The final stylized feature F cs The decoder based on the VGG19 neural network decodes and outputs the video image.

[0054] Preferably, the new style feature F is obtained in S20 sN and new content features F cN include:

[0055] like Figure 2-3 As shown, the MBSTS includes a feature extraction layer and a feature fusion layer;

[0056] The feature extraction layer includes a 1×1 convolution layer, three progressive modules and a pooling layer, and the three progressive modules include PM1, PM2 and PM3;

[0057] The style feature F s and content feature F c Input feature extraction layer to capture features and obtain new style features F sN and new content features F cN , the formula is as follows:

[0058] F cN =Branch_N(F c )

[0059] F sN =Branch_N(F s )

[0060] In the formula, Branch_N() represents the feature extraction layer, N = 1, 2, 3, 4, 5.

[0061] Preferably, the fusion output fusion feature F in S20 cSN , the formula is as follows:

[0062] F csN =CSAM(F cN , F sN )

[0063] Where CSAM() represents the channel space attention module CSA of the feature fusion layer.

[0064] Preferably, the output of the final stylized feature F in S20 cs , the formula is as follows:

[0065] F cs =f 1×1 (cat(F cs1 , …, F csN ))

[0066] In the formula, f 1×1 represents a 1×1 convolutional layer, and cat represents concatenation along the channel direction of the feature map.

[0067] Specifically, PM1, PM2, and PM3, the three progressive modules, have the same structure, including three consecutive dilated convolutional layers, an instantiation layer, and an activation layer.

[0068] In addition, MBSTS is also a multi-layer branch semantic transformation structure. The feature extraction layer is suitable for MPP, which is used for global and local spatial feature extraction. It uses expanded convolution with different expansion rates for feature extraction. Without changing the size of the convolution kernel, a richer receptive field can be obtained to extract richer feature information; the feature fusion layer uses the channel space attention module CSA module, which is stacked and dimensionally reduced to obtain fusion features. N = 1, 2, 3, 4, 5, indicating that the connected function guarantees comprehensive image information from these five branches, and a 1×1 convolution layer is used to increase the number of feature maps while filtering out redundant information.

[0069] Among them, the feature extraction layer is applicable to MPP. In a multi-layer branch structure, there are three parallel progressive modules of content and style features. The three progressive modules have the same structure, consisting of three consecutive dilated convolutional layers, an instantiation layer, and an activation layer. The only difference is the parameter design of the internal dilated convolutional layer. The dilation rates of the three consecutive dilated convolutional layers in PM1 are 2, 3, and 4, respectively. The dilation rates of the three consecutive dilated convolutional layers in PM2 are 5, 6, and 7, respectively. The dilation rates of the three consecutive dilated convolutional layers in PM3 are 8, 9, and 10, respectively. By using three progressive modules with different expansion rates, a richer receptive field and richer information can be obtained without downsampling. The continuous expansion rate in each progressive module is also used to avoid the grid effect. Take the first module (PM1) as an example to expand the description, taking the content feature as an example. Specifically, x is the content feature extracted by the content encoder as the input of the progressive module. The whole process can be expressed as:

[0070] x1=Dconv4(Dconv3(Dconv2(X)))

[0071] x1′=ReLU(InstanceNorm(x1))

[0072] Among them, Relu is the activation function, InstanceNorm is the normalization function, dconv represents the dilated convolution, and the number following it represents the expansion rate.

[0073] As for the feature fusion layer, the CSA module is used.

[0074] The inputs of the channel spatial attention module are style features and content features. First, the input style features are max-pooled and average-pooled in the spatial dimension through the channel attention module, and then passed through the fully connected layer. The features of the output of the fully connected layer are added pixel by pixel to generate the final channel attention feature map. The channel attention feature map is matrix multiplied with the content features after 1×1 convolution, and the output is added pixel by pixel with the content features after 1×1 convolution. The generated features are also used as the input of the spatial attention module. The process can be expressed as:

[0075]

[0076] in, represents the output of the channel attention module, f 1×1 represents 1×1 convolution, F c Represents the content characteristics of the input, represents matrix multiplication, FC represents the fully connected layer, AvgPool s Indicates average pooling of the spatial dimension, F s represents the style features of the input, Indicates pixel addition, MaxPool s Indicates maximum pooling of the spatial dimension.

[0077] The spatial attention module performs maximum pooling and average pooling on the style features in the channel dimension, then concatenates them together and obtains a spatial attention feature map through a 7×7 convolution. The spatial attention feature map is pixel-wise multiplied with the output of the channel attention module, and then the obtained output is pixel-wise multiplied with the content features after the 1×1 convolution.

[0078] The final output is obtained after addition. This process can be expressed as:

[0079]

[0080] Among them, F cs represents the stylized features of the final output, f 7×7 represents 7×7 convolution, cat represents concatenation along the feature channel direction, AvgPool c Indicates average pooling of the channel dimension, MaxPool c Indicates maximum pooling on the channel dimension.

[0081] Preferably, the S30 includes:

[0082] The decoder of the VGG19 neural network includes the main decoder D m and auxiliary decoder D a , the main decoder D m and auxiliary decoder Da Alternate training, shared input fusion feature F cs and style features F s , auxiliary decoder D a Generate output O auxiliary , main decoder D m Output main frame O main and abnormal frame O minor , the formula is as follows:

[0083] [O main , O minor ]=D m (F cs , F s )

[0084] O auxiliary =D a (F cs , F s )

[0085] In the formula, [O main , O minnor ] is the output video image.

[0086] Preferably, the auxiliary decoder D a The loss function of the training process is as follows:

[0087] L auxiliary =λ s L s +λ c L+λ temp L temp +λ SSL L SSL +λ SSRL L SSRL +λ recon L recon +λtL tv

[0088] In the formula, λ s , c , temp , SSL , recon and λ tv is the weight term, L s is the style loss value, L c is the content loss value, L comp is the time domain continuity loss value, L SSL is the self-similarity loss value, L temp is the reconstruction loss value, L tv is the total variation loss value.

[0089] Preferably, the main decoder D mThe loss function of the training process is as follows:

[0090] L main =λ s L s +λ c L c +λ cr L cr .

[0091] Preferably, the progressive multi-branch video style transfer method based on confidence-weighted loss also includes:

[0092] Setting a confidence-weighted loss function on the output video image to select a primary mode for each pixel based on the video image in the plurality of modes and ignoring the secondary mode;

[0093] The setting of the confidence weighted loss function includes calculating a confidence map of the main mode, and updating the confidence weighted loss function based on the confidence map of the main mode.

[0094] Preferably, the confidence map of the main mode is calculated as follows:

[0095]

[0096] In the formula, O main is the main frame, O minor is the outlier frame, O auxiliary is an auxiliary frame, d1 and d2 are respectively main With O auxiliary and O minor With O auxiliary For the distance in relative space, ε is set to 0.1 and 6 is set to 0.01.

[0097] Preferably, the confidence weighted loss function formula is updated based on the confidence map of the main mode, as follows:

[0098] d′1=L data (C t ·O main , C t ·O auxiliary )

[0099] d′2=L data ((1-C t )·O minor , (1-C t )·O auxiliary )

[0100] L cr =d′1+αd′2

[0101] In the formula, d'1 and d'2 represent O main Upper Omain and O minor The distance relationship between projections, L data is the L1 Manhattan distance, and α is the preset value.

[0102] Specifically, there may be multimodal inconsistencies in the generated videos. To this end, we design a confidence weighted calculation mechanism (L cr ). Our method selects one of multiple possible modes to ensure temporal consistency and maintain perceptual quality. In the confidence weighted loss function, the confidence map C t ∈{0,1} H*W*1 We aim to select the primary mode for each pixel based on the content image among multiple modes while ignoring outliers (secondary mode or multiple modes). We select the primary mode by comparing the relative distances between different modes in space.

[0103] For the overall framework training strategy, the network consists of two decoders: a primary decoder and an auxiliary decoder, which are trained alternately. The two decoders share the same input F cs 、F s We increase the number of channels in the main decoder output of the network, for example, adding six channels to two RGB images.

[0104] In addition, the function of the auxiliary decoder is to learn the basic features of the migration image, bear the heavy loss function, and establish the confidence reweighting condition for the main decoder. The goal of the auxiliary decoder loss is to optimize the encoder, MBS module, and auxiliary decoder. The parameters of the main decoder are frozen during the training process. The weight items of the seven losses of the auxiliary decoder include:

[0105] (1) In order to generate an artistic style image, the basic structure of the content image is maintained while also having a stylized image. The style loss L is introduced s and content loss L c for:

[0106]

[0107] Among them, φ i represents the feature extracted from the i-th layer of the pre-trained VGG19, μ represents the feature mean, and σ represents the feature variance. c Represents the content image. cs Represents the result of style transfer.

[0108] (2) In order to ensure the temporal consistency between frames of the stylized video, a temporal loss L is introduced temp for:

[0109]

[0110] Among them, F represents the image processing model, which means The corresponding x n to x m The optical flow of Δ represents local flicker or noise.

[0111] (3) In order to allow the decoder to reconstruct the style and content features of the source image, the reconstruction loss L is introduced recon for:

[0112] L recon =||F(C gray , C rgb )-C rgb || 2 ,

[0113] Among them, F is the image processing model, C gray is a grayscale image of the content map.

[0114] (4) In order to make the image smoother in signal processing, full-variation regularization is used and the full-variation loss L is introduced tv for:

[0115] L tv =||δ x (I cs )-δ y (I cs )||,

[0116] where δ x is the residual in the horizontal direction, δ y is the residual in the vertical direction.

[0117] (5) In order to suppress the amplification of noise in the input video, the self-similarity loss L is introduced SSL for:

[0118]

[0119] Among them, G() means converting the RGB image into a grayscale image, x m and x n denote the nth and mth frames of the video respectively. F denotes the image processing model. Δ denotes local jitter or noise. Represents x n to x m The optical flow of . Parameter μX res and μY res Represents X res and Y res The average value, σX res and σY res Yes X res and Y res The standard deviation ofres Y res Represents X res and Y res The covariance between .

[0120] (6) In order to improve the visual quality of the network reconstruction results of human viewers, content and style losses are introduced

[0121] L SSRL Structural similarity reconstruction of:

[0122] L SSRL =1-SSIM(F(C gray , C rgb ), C rgb ).

[0123] Where SSIM() is defined as self-similarity loss. F() is the image processing model.

[0124] In addition, for the loss of the main decoder, the loss target of the main decoder is the MBS module and the main decoder. The parameters of the auxiliary decoder and encoder are frozen during the training process. Therefore, the auxiliary decoder loss and the main decoder loss are independently constrained. The above three losses constitute the final linear addition total loss L main , which can be expressed as:

[0125] L main =λ s L s +λ c L c +λ cr L cr

[0126] Among them, using O main As the output of the style network, style loss and content loss are calculated.

[0127] The preferred specific embodiments of the present invention are described in detail above. It should be understood that ordinary technicians in the field can make many modifications and changes based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by technicians in the technical field based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A progressive multi-branch video style transfer method based on confidence-weighted loss, characterized in that: The following steps are involved: S10: Receive stylized image I s and content frame (I t-1 , I t , I t+1 ) and input into the VGG19 neural network. The style encoder and content encoder based on the VGG19 neural network extract the style features F s and content feature F c ; S20: The style feature F s and content feature F c Feature capture is performed based on MBSTS to obtain new style features F sN and new content features F cN And fusion output fusion feature F csN , the fusion feature F csN The final stylized feature F is output by splicing along the feature map channel direction cs ; S30: The final stylized feature F cs The decoder based on the VGG19 neural network decodes and outputs the video image; The MBSTS includes a feature extraction layer and a feature fusion layer; The feature extraction layer includes a 1×1 convolution layer, three progressive modules and a pooling layer, and the three progressive modules include PM1, PM2 and PM3; The S30 includes: The decoder of the VGG19 neural network includes the main decoder D m and auxiliary decoder D a , the main decoder D m and auxiliary decoder D a Alternate training, shared input fusion feature F cs and style features F s , auxiliary decoder D a Generate output O auxiliary , main decoder D m Output main frame O main and abnormal frame O minor , the formula is as follows: [O main ,O minor ]=D m (F cs ,F s ) O auxiliary =D a (F cs ,F s ) In the formula, [O main ,O minor ] is the output video image; The main decoder D m The loss function of the training process is as follows: L main =λ s L s +λ c L c +λ cr L cr ; In the formula, λ s , c is the weight term, L s is the style loss value, L c is the content loss value; The method further comprises: Setting a confidence-weighted loss function on the output video image to select a primary mode for each pixel based on the video image in the plurality of modes and ignoring the secondary mode; The setting of the confidence weighted loss function includes calculating a confidence map of the main mode, and updating the confidence weighted loss function based on the confidence map of the main mode.

2. The method for progressive multi-branch video style transfer based on confidence-weighted loss according to claim 1, characterized in that: The new style feature F is obtained in S20 sN and new content features F cN include: The style feature F s and content feature F c Input feature extraction layer to capture features and obtain new style features F sN and new content features F cN , the formula is as follows: F cN =Branch_N(F c ) F sN =Branch_N(F s ) Where Branch_N() represents the feature extraction layer, N = 1, 2, 3, 4, 5.

3. The method for progressive multi-branch video style transfer based on confidence-weighted loss according to claim 2, characterized in that: The fusion output fusion feature F in S20 csN , the formula is as follows: F csN =CSAM(F cN F sN ) Where CSAM() represents the channel space attention module CSA of the feature fusion layer.

4. The method for progressive multi-branch video style transfer based on confidence-weighted loss according to claim 3, characterized in that: The output of S20 is the final stylized feature F cs , the formula is as follows: F cs =f 1×1 (cat(F cs1 ,…,F csN )) In the formula, f 1×1 represents a 1×1 convolutional layer, and cat represents concatenation along the channel direction of the feature map.

5. The method for progressive multi-branch video style transfer based on confidence-weighted loss according to claim 1, characterized in that: The auxiliary decoder D a The loss function of the training process is as follows: L auxiliary =λ s L s +λ c L c +λ temp L temp +λ SSL L SSL +λ SSRL L SSRL +λ recon L recon +λ tv L tv In the formula, λ s , c , temp , SSL , recon and λ tv is the weight term, L s is the style loss value, L c is the content loss value, L recon is the time domain continuity loss value, L SSL is the self-similarity loss value, L temp is the reconstruction loss value, L tv is the total variation loss value.

6. The method for progressive multi-branch video style transfer based on confidence-weighted loss according to claim 1, characterized in that: The confidence map of the main mode is calculated as follows: In the formula, O main is the main frame, O minor is an abnormal frame, O auxiliary is an auxiliary frame, d1 and d2 are respectively main With O auxiliary and O minor With O auxiliary For the distance in relative space, ε is set to 0.1 and δ is set to 0.

01.

7. The method for progressive multi-branch video style transfer based on confidence-weighted loss according to claim 6, characterized in that: The confidence map based on the main mode updates the confidence weighted loss function formula as follows: d′1=L data (C t ·O main ,C t ·O auxiliary ) d′2=L data ((1-C t )·O minor ,(1-C t )·O auxiliary ) L cr =d′1+αd′2 Where d′1 is the confidence graph C below t Weighted O main and O auxiliary The distance between them, d′2 is the distance between them in the confidence graph 1-C t Weighted O minor and O auxiliary The distance between data is the L1 Manhattan distance, and α is the preset value.

Images