Image style migration method based on two-way vision Transform

Through the combination of dual-channel visual Transformer and style parameter extractor, the problems of insufficient rendering and style deviation in image style transfer are solved, and high-quality stylized image generation is achieved, combining content fidelity and style restoration.

CN119068100APending Publication Date: 2024-12-03NANJING UNIV
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202310646537.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-06-02
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing image style transfer methods have problems such as insufficient rendering, bias in style representation, unreasonable rendering intensity, difficult to maintain the balanced relationship between content semantics retention and style presentation, and coarse feature granularity, resulting in poor migration effect.

Method used

The image style transfer method based on dual-channel visual Transformer is adopted, and the global features of content images and style images are extracted using the Transformer encoder, and the style encoding is discreteed into feature vectors through the Transformer style parameter extractor, combining with the conditional instance normalization technology , realize the weighted fusion of content coding and style coding and the prediction of style feature distribution parameters, adaptively learn the target style feature distribution, and generate high-quality stylized images.

Benefits of technology

It effectively improves the richness and accuracy of style representation, achieves the balance between content fidelity and style restoration, solves the problems of insufficient rendering and style deviation, and the generated image stylization results are more detailed and balanced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068100A_ABST
    Figure CN119068100A_ABST
Patent Text Reader

Abstract

The invention discloses an image style migration method based on double-path visual Transform, which comprises the following steps of: respectively configuring a path of visual Transform encoder for a content image and a style image to obtain a content code and a style code; representing the style code as a style parameter vector by a Transform style parameter extractor; and carrying out weighted fusion on the content code and the style code through a visual Transform image decoder, predicting a feature distribution parameter of a target style based on a style parameter vector to obtain a stylized code, and finally decoding, outputting and generating a stylized image. According to the method, on the basis of a visual Transform architecture of double-channel coding-decoding, coding differences among different image domains are considered, the accuracy of style representation is improved, style distribution measurement is introduced, the style expression ability is further improved, the color distribution and texture texture of image rendering are highly matched with a real style, and the image rendering efficiency is improved. Meanwhile, original content semantics can be completely reserved, and the problem of distortion caused by application of local style textures is avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of computer image technology and relates to a style transfer method for an image, in particular to an image style transfer method based on a dual-path visual Transformer. Background Art

[0002] Image style transfer aims to separate, decouple and reassemble image content and image style, and then fuse the semantic concept of the content image and the texture style of the style image to generate a stylized image. Among them, the image with style texture is called the content image; the image that provides the target texture style is called the style image; and the generated image is called the stylized image. Image style transfer technology can meet people's basic needs for editing and rendering existing images, and create synthetic images with artistic expression and visual appeal.

[0003] Thanks to the powerful nonlinear representation learning ability of Convolutional Neural Network (CNN), image style transfer technology has flourished. CNN-based image style transfer methods usually divide the network logic according to the encoder-decoder architecture, in which the encoder is mostly based on a pre-trained image classification network, and its representation ability directly determines the quality of style presentation. This type of method requires stacking a large number of convolutional pooling layers to obtain sufficient receptive fields, thereby reorganizing low-level visual features into high-level abstract semantics. At the same time, the network scale and complexity are simultaneously improved, and the feature resolution is continuously reduced, which makes it impossible to accurately depict the content semantics and texture style. In addition, this type of method does not consider the encoding differences between different image domains, and directly applies the encoder trained based on the natural image dataset to the encoded style image, thereby introducing representation errors. As described in references [2,3], CNN also has obvious texture bias, which is not conducive to preserving the original semantic structure of the content image.

[0004] With the successful application of Transformer in the field of computer vision, the image style transfer task has been revitalized. Unlike CNN, the visual Transformer relies on the attention mechanism to capture the global information of the image at the shallow level of the network, and models the long-distance dependency between image blocks, which is similar to the visual perception characteristics of humans who recognize objects by shape. However, the visual Transformer cannot effectively represent the pixel correlation within each image block, that is, the local feature characterization ability of the image is poor and there is a shape bias. The existing representative methods roughly follow the original visual Transformer structure design, that is, the image is first encoded based on the self-attention mechanism, and then the feature fusion is achieved by calculating the cross-attention between the content encoding and the style encoding, so as to map the content image to the target style domain. This type of method cannot accurately shape the color distribution and significant style in the style image. When there are fine textures in the reference style, the overall style presentation quality may not be as good as the most advanced CNN method.

[0005] In summary, although the network architecture of image style transfer methods has gradually transitioned from CNN to visual Transformer, the following problems still exist in existing methods: First, the second-order feature statistics are directly used to characterize the image style, and the distribution measurement of style features is not considered. Therefore, the style information cannot be fully extracted, and there is a problem of incomplete and insufficient style representation. The style transfer effect is limited to color distribution and lacks significant texture; second, the prediction of style feature distribution parameters is biased, and it is impossible to accurately shape and restore the target style; third, it is difficult to effectively maintain the balanced relationship between content semantic retention and style presentation, and there is a problem of unreasonable rendering intensity, resulting in a single rendering bias of emphasizing content over style or emphasizing style over content; fourth, the encoding difference between the content image domain and the style image domain is not considered, and a general encoder trained based on a natural image dataset is used to extract the feature representation of the content image and the style image, which has obvious style representation deviation; fifth, the feature granularity is coarse, and it is difficult to capture fine-grained and small-scale local features, so that the stylized results lack local texture details, and the transfer effect is concentrated on the adjustment of the image color distribution, and there is a shape bias problem.

[0006] References:

[0007] [1]Deng Y,Tang F,Dong W,et al.StyTr 2 :Image Style Transfer withTransformers[C].IEEE Conference on Computer Vision and Pattern Recognition,2022

[0008] [2]Geirhos R,Rubisch P,Michaelis C,et al.ImageNet-trained CNNs areBiased Towards Texture;Increasing Shape Bias Improves Accuracy and Robustness[C].International Conference on Learning Representations,2019

[0009] [3]Wei H,Deng Y,Tang F,et al.A Comparative Study of CNN-andTransformer-Based Visual Style Transfer[J].Journal of Computer Science andTechnology,2022,37(3)

[0010] [4]Vaswani A,Shazeer N,Parmar N,et al.Attention is All You Need[J].arXiv preprint,2017,arXiv:1706.03762

[0011] [5]Zhang C,Yang J,Wang L,et al.S2WAT:Image Style Transfer viaHierarchical Vision Transformer using Strips Window Attention[J].arXivpreprint,2022,arXiv:2210.12381

[0012] [6]Wang J,Yang H,Fu J,et al.Fine-Grained Image Style Transfer withVisual Transformers[C].Asian Conference on Computer Vision,2022

[0013] [7]Ghiasi G,Lee H,Kudlur M,et al.Exploring the Structure of A Real-time Arbitrary Neural Artistic Stylization Network[C].British Machine VisionConference,2017

[0014] [8]Huang X,Belongie S.Arbitrary Style Transfer in Real-Time withAdaptive Instance Normalization[C].IEEE Conference on Computer Vision andPattern Recognition,2017

[0015] [9]Dae Y.Park,Kwang H.Lee.Arbitrary Style Transfer With Style-Attentional Networks[C].IEEE Conference on Computer Vision and PatternRecognition,2019 Summary of the invention

[0016] The problem to be solved by the present invention is to solve the problems of insufficient rendering, deviation from the true style, over-rendering and weak rendering that may occur in the process of image style transfer, so as to achieve high-quality image style transfer.

[0017] The technical solution of the present invention is: an image style transfer method based on a two-way visual Transformer, constructing a visual Transformer network model based on two-way encoding-decoding for image style transfer tasks, firstly configuring a visual Transformer encoder for the content image and the style image respectively, to obtain content encoding and style encoding; then discretizing the style encoding to represent the style as a set of features, to obtain a style parameter vector; then weightedly fusion of the content encoding and the style encoding through a visual Transformer image decoder, and predicting the feature distribution parameters of the target style based on the style parameter vector, introducing a style distribution metric in the image stylization process, adaptively learning the feature distribution parameters of the target style, mapping the content encoding to the target style domain to achieve specific style rendering, obtaining a stylized encoding, and finally decoding the stylized encoding output to generate a stylized image.

[0018] Furthermore, the implementation of the visual Transformer network model based on dual-path encoding-decoding includes the following steps:

[0019] 1) The content image and style image of the input network model are divided and encoded, and the content image block feature sequence and the style image block feature sequence are obtained through pre-trained VGG19 network feature encoding;

[0020] 2) Perform position encoding on each image block to obtain a content image block position encoding sequence and a style image block position encoding sequence;

[0021] 3) The content image block feature sequence and the content image block position encoding sequence are superimposed to obtain the content input sequence, and the style image block feature sequence is used as the style input sequence. For the two input sequences, a visual Transformer encoder is used to extract global features to obtain content encoding and style encoding;

[0022] 4) Construct a Transformer style parameter extractor to predict the style parameter vector based on the style encoding to discretize the style as a set of features. The Transformer style parameter extractor stacks N extraction layers, each of which contains two multi-head cross-attention MCA modules and a multi-layer perception module. The prediction of the style parameter vector is:

[0023] 4.1) Initialize the style parameter vector to all 0s, and get D represents the vector dimension;

[0024] 4.2) Calculate the channel mean of the style image position encoding in step 2) Will and Add them as the query vector input of the Transformer style parameter extractor, transform the dimension and calculate the channel mean of the style code obtained in step 3), and use them as the key vector and value vector of the multi-head cross attention module. Then calculate the multi-head cross attention of the style parameter vector and the style code based on the query vector, key vector and value vector to obtain the updated style parameter vector

[0025] 4.3) Update the style parameter vector Perform dimensionality reduction to reduce redundant features and enhance understandability, and obtain the final style parameter vector

[0026] 5) Visual Transformer decoder weighted fusion content encoding and style encoding, based on the style encoding in step 3) and the style parameter vector in step 4) Predict a set of denormalization parameters to map the content encoding from the source domain to the target style domain, obtaining a stylized encoding where the layer normalization in the visual Transformer encoder is replaced by conditional instance normalization (CIN);

[0027] 6) Define a three-layer CNN decoder, upsample the stylized code, compress its feature channels, remap the intermediate feature map to the image pixel space, and generate a stylized image.

[0028] Furthermore, when training the network model, feature encoding is performed on the content image, style image and stylized image respectively, the content loss is calculated to evaluate the semantic structure similarity between the stylized image and the content image, the style loss is calculated to evaluate the visual appearance similarity between the stylized image and the style image, and the identity loss is calculated to limit unnecessary image rendering, that is, when the input content image is the same as the style image, the stylized image should be the input image itself; the weighted sum of the content loss, style loss and identity loss is used as the total training loss of the image style transfer, and the network model parameters are optimized by minimizing the total training loss.

[0029] The present invention can solve the problems of insufficient rendering, style representation deviation and unreasonable rendering intensity existing in the existing methods. The key lies in the following three designs:

[0030] (1) Considering that the content image and the style image belong to different image domains, the present invention uses the Transformer content image encoder and the Transformer style image encoder to perform global feature extraction on the content image and the style image respectively, instead of using a general Transformer image encoder, thereby improving the encoding accuracy of the style image.

[0031] (2) The design of the present invention introduces a Transformer style parameter extractor, which can discretize the image style into a set of key features, further improving the richness and accuracy of style representation.

[0032] (3) The present invention replaces the layer normalization in the original visual Transformer decoder with conditional instance normalization, and uses the style parameter vector extracted by the Transformer style parameter extractor to predict a pair of style feature distribution parameters. Compared with the existing method, the distribution measurement of style features is explicitly considered, and a specific style rendering is achieved by matching the feature distribution of content encoding to the style feature distribution, thereby effectively preserving the original content semantics and accurately shaping the target style.

[0033] This paper proposes an image style transfer method based on dual-path visual Transformer, which solves the problems of insufficient rendering, style representation deviation and unreasonable rendering intensity in existing methods, and has both high content fidelity and style restoration. By introducing a Transformer style parameter extractor to discretize the image style as a set of key features, the richness and accuracy of style representation are effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is the implementation process of the present invention.

[0035] Figure 2 Schematic diagram of the various parts of the dual-path visual Transformer encoding-decoding network model of the present invention, (a) is the Transformer content image encoder, (b) is the Transformer style image encoder, (c) is the Transformer style parameter extractor, and (d) is the Transformer image decoder.

[0036] Figure 3 The figure shows the effect of image style transfer. (a) is the content image, (b) is the style image, (c) shows the stylized result of an insufficiently rendered image, (d) shows the stylized result of an image that deviates from the true style, (e) shows the stylized result of an over-rendered image, (f) shows the stylized result of a weakly rendered image, and (g) is a high-quality image stylized result.

[0037] Figure 4 This is an example of comparing the image style transfer results of the present invention with those of existing representative methods. DETAILED DESCRIPTION

[0038] The present invention proposes an image style transfer method based on a dual-path visual Transformer. Figure 1 As shown, the visual Transformer network model based on dual-path encoding-decoding is used for image style transfer tasks. First, a visual Transformer encoder is configured for the content image and the style image respectively to obtain content encoding and style encoding; then the style encoding is discretized to represent the style as a set of features to obtain a style parameter vector; then the content encoding and the style encoding are weightedly fused through the visual Transformer image decoder, and the feature distribution parameters of the target style are predicted based on the style parameter vector. In the image stylization process, a style distribution metric is introduced, and the feature distribution parameters of the target style are adaptively learned. The content encoding is mapped to the target style domain to achieve specific style rendering, and the stylized encoding is obtained. Finally, the stylized encoding is decoded and output to generate a stylized image.

[0039] The implementation of the method of the present invention is described in detail below.

[0040] 1) Use the pre-trained VGG19 network to divide and encode the input content image and style image respectively to obtain the content image block feature sequence and the style image block feature sequence.

[0041] 1.1) Extract content image With style image Feature encoding on the relu4_1 layer of the pre-trained VGG19 network to obtain the content image feature encoding and style image feature encoding The feature encoding width and height obtained by the VGG19 network are both 1 / m of the width and height of the original image, thus dividing the content image and the style image into m×m image blocks, where H=256 represents the image height; W=256 represents the image width; m=8 represents the image block size; and C=512 represents the number of feature channels.

[0042] 1.2) Flatten by spatial dimension and Get the content image block feature sequence And style image patch feature sequence in, Indicates the length of the image block sequence.

[0043] 2) The content-aware positional embedding (CAPE) proposed in the literature [1] is used to represent the spatial position information of each content image block:

[0044]

[0045] Among them, AvgPool represents the average pooling operation; n = 18 is the spatial size of the feature map after average pooling; Conv 1×1 Represents point convolution operation; represents the content image position encoding; C = 512 is the number of output channels of the point convolution operation. Finally, bilinear interpolation is used to convert Upsample to ε c The spatial size is the same, and the content image block position encoding sequence is obtained This position encoding method helps to impose consistent style texture on content image blocks with similar semantics, and the position encoding of the same image area does not change with the image scale, so it is more suitable for image style transfer tasks. Similarly, CAPE is used to characterize the spatial position information of each style image block to obtain the style image position encoding Among them, D=768 is the number of output channels of the point convolution operation.

[0046] 3) Construct a visual Transformer encoder for the content image domain and the style image domain respectively, following the design and implementation in the literature [1], which are the Transformer content image encoder and the Transformer style image encoder respectively, so as to perform feature interaction on the content image block and the style image block respectively, capture the long-distance dependency between the image blocks, and thus extract the global features of the content image and the style image to obtain the content encoding and the style encoding.

[0047] 3.1) Transformer content image encoder has N encoding layers stacked together, each layer contains a multi-head self-attention (MSA) module and a multi-layer perception (MLP) module, and its structure is as follows: Figure 2 (a) shown.

[0048] Since the stylized image needs to preserve the semantic structure in the content image, c and Add together to get the content input sequence Secondly, the multi-head self-attention of the content image block feature sequence is calculated:

[0049]

[0050] Where Q = K = E c , V = ε c denote the query vector, key vector, and value vector respectively; h = 8 denotes the number of attention heads; is the parameter matrix corresponding to the i-th attention head; is the output projection matrix; Attn is the attention operation defined in the literature [4]; Concat represents the dimension concatenation operation. Then, the nonlinear relationship between the content image features is further explored through the multi-layer perception module, which defines two fully connected layers to first increase the dimension of the self-attention output to 2048 and then reduce the dimension to 512. In addition, the input and output of each module are residually connected and input to the next module after layer normalization (Layer Normalization, LN), and finally the content encoding is obtained. Therefore, the calculation process of each encoding layer in the Transformer content image encoder can be defined as:

[0051]

[0052]

[0053] 3.2) Transformer style image encoder also stacks N encoding layers, and its structure is the same as Transformer content image encoder. Similarly, the style encoding can be obtained Since the stylized image does not need to retain the semantic structure in the style image, there is no need to add the style image block position encoding sequence to the style input sequence, that is,

[0054] 4) The present invention constructs a Transformer style parameter extractor, which predicts the style parameter vector based on the style encoding, thereby discretizing the image style as a set of key features. The Transformer style parameter extractor stacks N extraction layers, each of which contains two multi-head cross-attention (MCA) modules and a multi-layer perception module. Its structure is as follows: Figure 2 (c) as shown.

[0055] 4.1) Initialize the style parameter vector to all 0s, and get Wherein, D=768 represents the vector dimension.

[0056] 4.2) Since the style parameters need to reflect the structural information of the style image, first calculate the style image position encoding in step 2) The channel mean of Then and Add as the query vector input to the Transformer style parameter extractor, i.e. Secondly, the style encoding Φ obtained in step 3) s Perform dimension transformation and channel mean calculation to obtain Use it as the key vector and value vector of the multi-head cross attention module, that is, K = V = Φ′ s Next, the multi-head cross attention of the style parameter vector and the style encoding is calculated in the same way as the multi-head self-attention. Therefore, the calculation process of each extraction layer in the Transformer style parameter extractor can be defined as:

[0057]

[0058]

[0059]

[0060] in, Represents the extracted style parameter vector.

[0061] 4.3) For the style parameter vector Perform dimensionality reduction and obtain To reduce redundant features and enhance understandability. Among them, U = 100 represents the dimension of the style parameter vector after dimensionality reduction. Here, dimensionality reduction is an empirical parameter setting used to debug the stylization results to accurately present the target style. In addition, dimensionality reduction can also reduce the amount of intermediate calculations of the network model.

[0062] 5) Construct a visual Transformer image decoder based on the style encoding Φ in step 3) s and the style parameter vector in step 4) The content encoding is mapped from the source domain to the target style domain. The Transformer image decoder stacks N decoding layers, each of which contains two multi-head cross-attention modules and a multi-layer perception module. Its structure is as follows Figure 2 (d) In order to utilize the extracted style parameter vector, the present invention replaces the layer normalization in the original architecture of the visual Transformer with Conditional Instance Normalization (CIN).

[0063] 5.1) Based on style parameter vector Predict 3N pairs of style feature distribution parameters for conditional instance normalization operations in each decoding layer. The 3N, 3 refers to each decoding layer ( Figure 2 (d)) has three conditional instance normalization layers, and then the Transformer image decoder stacks N decoding layers in total, which is a total of 3N pairs of style feature distribution parameters. CIN uses style feature distribution parameters and Scale and translate the intermediate embedding to match it to the style feature distribution and achieve a specific style rendering:

[0064]

[0065]

[0066]

[0067] Among them, μ(x) and σ(x) represent the channel mean and channel standard deviation of feature code x respectively; is the projection matrix; b γ , b β is the bias term, and the intermediate embedding refers to the input of the conditional instance normalization module in the Transformer image decoder.

[0068] 5.2) Since the stylized image needs to preserve the semantic structure of the content image, first c and Add as the query vector input of the Transformer image decoder, that is, And set the key vector and value vector to Φ s , that is, K = V = Φ s Therefore, the calculation process of each decoding layer in the Transformer image decoder can be defined as:

[0069]

[0070]

[0071]

[0072] in, Encoding for style.

[0073] 6) This paper defines a three-layer CNN decoder to upsample the Transformer decoded output and compress its feature channels. The structure is shown in Table 1. Finally, the intermediate feature map is remapped to the image pixel space to generate a stylized image.

[0074] Table 1

[0075]

[0076] Wherein, conv3_N represents a convolution operation with a convolution kernel size of 3×3 and an output channel number of N. In the present invention, each convolution layer is followed by a ReLU nonlinear activation function, and the convolution blocks perform two-fold upsampling on the feature map of the intermediate layer through nearest neighbor interpolation.

[0077] 7) Train the dual-path encoding-decoding visual Transformer network model constructed in the above steps and calculate the content loss to evaluate the stylized image I cs With content image I c The semantic structure similarity of the style loss is calculated to evaluate the stylized image I cs With style image I s The visual appearance similarity of is used to calculate the identity loss to limit unnecessary image rendering, that is, when the input content image is the same as the style image, the stylized image should be the input image itself.

[0078] 7.1) Calculate content loss Extract I cs ,I c The feature encoding on the relu4_2 layer in the pre-trained VGG19 network is normalized to obtain and The present invention defines content loss as the mean square error between image feature encodings:

[0079]

[0080] Among them, c ={relu4_2} is the content loss calculation layer; H l , W l Represents the spatial size of the feature map on layer l; Representation feature map The eigenvalue of position (i, j) in ; |·| represents the cardinality of the set.

[0081] 7.2) Calculate style loss Select the relu1_1, relu2_1, relu3_1 and relu4_1 layers in the pre-trained VGG19 network as the style loss calculation layer, and set I cs ,I s Encoded as and The present invention defines the style loss as the sum of the mean square errors of two pairs of feature distribution parameters:

[0082]

[0083] Among them, s ={relu1_1,relu2_1,relu3_1,relu4_1} is the style loss calculation layer; μ(·) and σ(·) are used to calculate the channel mean and channel standard deviation of the feature map, respectively.

[0084] 7.3) Calculate two identical losses and The fixed network model inputs two content images (style images), and generates a styled image I through a dual-path visual Transformer encoding and decoding architecture. cc (I ss ). The present invention will be constant loss Defined as the pixel difference between the stylized image and the input image:

[0085]

[0086] in, Represents image I cc The present invention further calculates the encoding difference between the stylized image and the input image in the feature space as the identity loss

[0087]

[0088] Among them, id={relu1_1,relu2_1,relu3_1,relu4_1} is the constant loss calculation layer.

[0089] 7.4) Take the weighted sum of content loss, style loss and identity loss as the total training loss for image style transfer:

[0090]

[0091] Among them, λ con , sty , and The branch losses are and The weight parameter of .

[0092] 7.5) Minimize the total training loss of image style transfer to optimize and update the network model parameters.

[0093] Figure 3 The figure shows the effect of image style transfer, (a) is the content image, (b) is the style image, (c) shows the stylized result of an insufficiently rendered image, (d) shows the stylized result of an image that deviates from the true style, (e) shows the stylized result of an over-rendered image, (f) shows the stylized result of a weakly rendered image, and (g) is a high-quality image stylized result, which is the expected effect of the present invention.

[0094] The present invention is implemented on a set of images collected from the Internet and compared with existing representative methods. Figure 4 The figure shows a comparison example between the present invention and an existing representative image style transfer method. Figure 4 (a) is the content image, Figure 4 (b) is the style image, Figure 4 (c) is the result of the present invention, and the representative method involved in the comparison ( Figure 4 (d)-(i)) are: StyTr 2 method (reference [1]), S2WAT method (reference [5]), STTR method (reference [6]), Ghiasi method (reference [7]), AdaIN method (reference [8]) and SANet method (reference [9]). The experimental results show that: (1) StyTr 2 The style expression of the method is not as good as the results of the present invention (such as the second column). 2The method tends to transfer texture styles to salient objects, leaving the image background with a weak stylized effect (such as the third and fourth columns); (2) The transfer quality of the S2WAT method varies greatly under different input combinations, and the overall content structure and semantic edges are weakened due to the application of texture styles (such as the first and second columns); (3) The STTR method tends to retain fine contour structures to avoid content distortion as much as possible, and its stylized effect lacks salient textures (such as the second and fourth columns). In addition, the STTR method’s depiction of the target style deviates from the real texture (such as the first column), making its overall stylization quality lag behind other visual Transformer-based methods; (4) The Ghiasi method has the worst style representation ability and is only suitable for transferring simple styles (such as the third and fourth columns); (5) The AdaIN method has the problem of incomplete and insufficient stylization, with a small amount of content artifacts remaining in the stylized image (such as the third and fourth columns), and poor ability to reshape complex textures (such as the second column); (6) The SANet method has the problem of the rationality of texture style application, the target style is non-uniformly transferred to semantically salient objects, resulting in a fragmentation effect (such as the fifth column), and the same texture has semantic heterogeneity (such as the sky texture transferred to the bridge in the second column). In addition, the SANet method cannot accurately transfer complex styles (such as the first column), especially when there are large visual structure differences between the source domain and the target domain; (7) Compared with the method based on visual Transformer, the results of the present invention have achieved comparable or even better stylization effects in terms of style perceptibility and content recognizability. It can accurately depict significant style features and apply uniform and consistent stylization effects. Compared with the CNN-based method, the results of the present invention are more faithful to the original content and reference style, and the transition between semantic textures is smoother and more natural, thanks to the representation ability of the infrastructure.

Claims

1. An image style transfer method based on dual-path visual Transformer, characterized by A visual Transformer network model based on dual-path encoding-decoding is constructed for image style transfer tasks. First, a visual Transformer encoder is configured for the content image and the style image respectively to obtain content encoding and style encoding. Then, the style encoding is discretized to represent the style as a set of features to obtain a style parameter vector. The content encoding and style encoding are weightedly fused through the visual Transformer image decoder, and the feature distribution parameters of the target style are predicted based on the style parameter vector. A style distribution metric is introduced in the image stylization process, and the feature distribution parameters of the target style are adaptively learned. The content encoding is mapped to the target style domain to achieve specific style rendering, and the stylized encoding is obtained. Finally, the stylized encoding output is decoded to generate a stylized image.

2. The image style transfer method based on dual-path visual Transformer according to claim 1 is characterized by The implementation of the visual Transformer network model based on dual-path encoding-decoding includes the following steps: 1) The content image and style image of the input network model are divided and encoded, and the content image block feature sequence and the style image block feature sequence are obtained through pre-trained VGG19 network feature encoding; 2) Perform position encoding on each image block to obtain a content image block position encoding sequence and a style image block position encoding sequence; 3) The content image block feature sequence and the content image block position encoding sequence are superimposed to obtain the content input sequence, and the style image block feature sequence is used as the style input sequence. For the two input sequences, a visual Transformer encoder is used to extract global features to obtain content encoding and style encoding; 4) Construct a Transformer style parameter extractor to predict the style parameter vector based on the style encoding to discretize the style as a set of features. The Transformer style parameter extractor stacks N extraction layers, each of which contains two multi-head cross-attention MCA modules and a multi-layer perception module. The prediction of the style parameter vector is: 4.1) Initialize the style parameter vector to all 0s, and get D represents the vector dimension; 4.2) Calculate the channel mean of the style image position encoding in step 2) Will and Add them as the query vector input of the Transformer style parameter extractor, transform the dimension and calculate the channel mean of the style code obtained in step 3), and use them as the key vector and value vector of the multi-head cross attention module. Then calculate the multi-head cross attention of the style parameter vector and the style code based on the query vector, key vector and value vector to obtain the updated style parameter vector 4.3) Update the style parameter vector Perform dimensionality reduction to reduce redundant features and enhance understandability, and obtain the final style parameter vector 5) Visual Transformer decoder weighted fusion content encoding and style encoding, based on the style encoding in step 3) and the style parameter vector in step 4) Predict a set of denormalization parameters to map the content encoding from the source domain to the target style domain, obtaining a stylized encoding where the layer normalization in the visual Transformer encoder is replaced by conditional instance normalization (CIN); 6) Define a three-layer CNN decoder, upsample the stylized code, compress its feature channels, remap the intermediate feature map to the image pixel space, and generate a stylized image.

3. The image style transfer method based on dual-path visual Transformer according to claim 1 or 2, characterized in that training When the network model is used, feature encoding is performed on the content image, the style image and the stylized image respectively, the content loss is calculated to evaluate the semantic structure similarity between the stylized image and the content image, the style loss is calculated to evaluate the visual appearance similarity between the stylized image and the style image, and the identity loss is calculated to limit unnecessary image rendering, that is, when the input content image is the same as the style image, the stylized image should be the input image itself; the weighted sum of the content loss, the style loss and the identity loss is used as the total training loss of the image style transfer, and the network model parameters are optimized by minimizing the total training loss.

4. The image style transfer method based on dual-path visual Transformer according to claim 1 or 2, characterized in that Step 1) is specifically as follows: 1.1) Extract the feature encoding of the content image and style image on the relu4_1 layer of the pre-trained VGG19 network respectively; 1.2) Flatten the content image feature encoding and the style image feature encoding according to the spatial dimension to obtain a content image block feature sequence and a style image block feature sequence.

5. The image style transfer method based on dual-path visual Transformer according to claim 1 or 2, characterized in that Step 2) Use the content-aware position coding CAPE method to position encode the image blocks of the content image and the image blocks of the style image to obtain a content image block position coding sequence and a style image block position coding sequence.

6. The image style transfer method based on dual-path visual Transformer according to claim 1 or 2, characterized in that In step 3), the dual-path visual Transformer encoder includes a Transformer content image encoder and a Transformer style image encoder, specifically: 3.1) Stylized images need to preserve the semantic structure of the content image. First, the feature sequence of the content image block ε c Sequence of content image block positions Add together to get the content input sequence Then the Transformer content image encoder encodes the content code Φ c ; 3.2) Stylized images do not need to retain the semantic structure in the style image, and there is no need to add the style image block position encoding sequence to the style input sequence, that is, the style input sequence The style code Φ is then encoded by the Transformer style image encoder s .

7. The image style transfer method based on dual-path visual Transformer according to claim 1 or 2, characterized in that The visual Transformer decoder in step 5) maps the content encoding from the source domain to the target style domain, including: 5.1) Based on style parameter vector Predict 3N pairs of style feature distribution parameters for conditional instance normalization operations in each decoding layer of the decoder, where N is the number of decoding layers. Conditional instance normalization CIN uses style feature distribution parameters and Scale and translate the intermediate embeddings of the network model to match them to the style feature distribution and achieve specific style rendering: Among them, μ(x) and σ(x) represent the channel mean and channel standard deviation of feature code x respectively; is the projection matrix; b γ 、b β is the bias term; 5.2) Use the visual Transformer decoder to perform feature interaction between content encoding and style encoding, and convert the content encoding Φ c Sequence of content image block positions Add as the query vector input to the decoder, set the key vector and value vector to the style code Φ s , through conditional instance normalization, the overall distribution of the intermediate embedding is adjusted to match the style feature distribution, resulting in a stylized encoding.

Citation Information

Cited By

  • Cross-device stylized enhancement method and system for cement hydration microstructure image

    CN119251073A

  • DIC speckle image generation method based on deep learning neural network

    CN119904543A