Two-segment feature sensing method for any style migration of image

Abstract

The invention discloses a two-segment perception feature method for any style migration of an image, and aims to create a new artistic work by applying a target artistic style to a given photograph, and the method comprises the following steps: firstly, extracting features of a content image and a style image by using an attention module to enhance representation; according to the method, local style details are enriched through an attention mechanism, style feature distribution is adjusted according to content feature distribution, a semantic relationship in a content image is reserved in a stylization process to realize better content reservation, and then a global statistical information alignment module is used for fine tuning of a global image style. A contrast experiment on a public image data set verifies that the method can realize high-quality image artistic style migration.

Description

Technical Field

[0001] The present invention belongs to the technical fields of computer vision and image classification, and specifically relates to a method for transferring arbitrary artistic styles of images. Background Art

[0002] Style transfer is one of the basic tasks in computer vision. Image arbitrary artistic style transfer aims to create new artistic images by rendering a given photo into a target artistic style. Existing methods simply learn styles based on global statistics or local patches, lacking careful consideration of the painting process in practice. Therefore, the stylized results either fail to capture rich and diverse local style patterns, or contain unexpected semantic information in the style image, deviating from the global style distribution. To address this problem, we propose a two-segment perceptual feature method for image arbitrary style transfer. First, local features are exchanged in a block-by-block manner to enrich local style details, and then global style features are fine-tuned by aligning the global statistics of content and style features. Through these innovations, our method is able to generate more visually appealing and detailed artistic style transfer images. Summary of the invention

[0003] 1. A two-segment perceptual feature method for arbitrary image style transfer, characterized by comprising the following steps:

[0004] Step 1: Obtain a recognized public image dataset and perform rotation and cropping preprocessing operations on the dataset. The dataset includes content images and style images.

[0005] Step 2: The preprocessed content image I c and style image I s Passed into the pre-trained VGG-19 encoder E, the specific calculation formula is as follows:

[0006] F c =E(I c ), F s =E(I s );

[0007] Step 3: Construct a two-segment perceptual feature network model for arbitrary image style transfer, including an attention module and a global statistical information alignment module;

[0008] Step 3.1: Construct the attention module. In this stage, we first transform F c and F s After regularization, convolution operation is performed to obtain F q 、F k 、F v , let F q Make proxy token A, the proxy token is first used as query token F qAgent, from F k and F v Aggregate information and then broadcast it back to F q , the specific calculation formula is as follows:

[0009] O A =Attn s (Q, A, Attn s (A, K, V)),

[0010]

[0011] Where Softmax is the activation function, F attn is the attention score feature map obtained by the proposed attention module;

[0012] Step 3.2: Build a global statistical information alignment module and calculate the weighted statistics of each point. The calculation method is as follows:

[0013]

[0014] F cs = Std·Norm(F c )+Mean;

[0015] Among them, Softmax is the activation function, and Norm represents mean variance normalization;

[0016] Step 4: Construct the decoder. The decoder and encoder are designed symmetrically. cs Feature map for image reconstruction;

[0017] Step 5: Use the pre-trained VGG-19 to extract the features of the generated image and calculate the loss function. The specific formula is as follows:

[0018]

[0019] in is the content loss, is the style loss, is the identity loss, which is calculated as follows:

[0020] Step 5.1: The content loss calculation formula is as follows:

[0021]

[0022] Where φi is the feature extracted from the i-th layer of the pre-trained VGG-19 network;

[0023] Step 5.2: The style loss calculation formula is as follows:

[0024]

[0025] Among them, μ(·) represents the mean value of the feature, φi represents the feature extracted from the i-th layer of the pre-trained VGG-19 network, and σ(·) represents the variance of the feature;

[0026] Step 5.3: The identity loss is calculated as follows:

[0027]

[0028] Among them, I cc and I ss They represent the composite results generated when the input is the same content and style image; φ i refers to the features extracted from the relui_1(i∈[1-5]) layer of the VGG network, and λ id1 and λ id2 is a hyperparameter;

[0029] Step 6: Train the model and generate model weight and decoder weight files based on the loss function;

[0030] Step 7: The trained model receives content images and style images of any resolution and outputs high-quality artistic style transfer photos.

[0031] The present invention proposes a two-segment perceptual feature method for arbitrary image style transfer, and the beneficial effects brought by the provided technical solution include at least:

[0032] In an embodiment of the present invention, a neural network model for image arbitrary artistic style transfer is constructed: an attention module and a global statistical information alignment module. The attention module first extracts the texture, color, and structural information of the content image and the style image. The deep and shallow features are exchanged for local features in a block-by-block manner, and then the content features and style features are subjected to global statistical information alignment adaptive normalization processing so that their local feature statistics match the global feature statistics. The attention module combines the shallow and deep convolutional neural network features of the style image and the content image, and at the same time achieves point-by-point feature statistical matching of the content features and the style features. The global statistical information alignment adjusts the mean and variance of the feature map after the local block exchange stage to achieve global image information fine-tuning.

[0033] In summary, the main advantages of the present invention are:

[0034] 1. A two-segment perceptual feature method is proposed to achieve the style transfer task in two steps.

[0035] 2. An attention module and a global statistical information alignment module are proposed. The attention module focuses on capturing local style features, and the global statistical information alignment module aligns the global statistical information of content and style features, which significantly improves the effect of image stylization. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1 A schematic flow chart of a two-segment perceptual feature method for arbitrary style transfer of an image provided by an embodiment of the present invention.

[0038] Figure 2 A schematic diagram of a two-segment perception feature network structure provided for an embodiment of the present invention.

[0039] Figure 3 The results of our model generating stylized images provided by the embodiments of the present invention. DETAILED DESCRIPTION

[0040] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0041] like Figure 1 As shown, an embodiment of the present invention provides a two-segment perceptual feature method for arbitrary style transfer of an image, comprising:

[0042] 1. A two-segment perceptual feature method for arbitrary image style transfer, characterized in that we designed a new network structure, specifically including the following steps:

[0043] Step 1: Obtain a recognized public image dataset. In the present invention, the Microsoft COCO dataset is used as the content image dataset and the WikiArt dataset is used as the style image dataset. The dataset is pre-processed by rotation and cropping.

[0044] Step 2: Input the content image and style image of the training set, pass the image into the pre-trained VGG-19 encoder, and obtain the content feature map and style feature map;

[0045] Step 3: Construct an arbitrary art style transfer network model for two segments of perceptual features. The model consists of an attention module and a global statistical information alignment module. The attention mechanism in the first stage of the present invention mainly completes the style transfer of the local blocks of the image first. It is different from the traditional linear attention and Softmax attention. Softmax attention needs to query the similarity between two keyword pairs. The linear attention has low complexity but weak expression ability. In the present invention, a small number of tokens are used to aggregate and broadcast global information, which has the characteristics of high expression ability and low complexity. In this way, the local feature information of the content image and the style image is exchanged. The global statistical information alignment module in the second stage of the present invention fine-tunes the overall image color style after the exchange of local features of the image is completed, aiming to align the global statistics of content and style features. Considering that the mean and variance of the image do not carry any semantic information, but only carry the style information of the image, the present invention uses this module to optimize the image.

[0046] Step 4: Construct the decoder. The decoder and encoder are designed symmetrically. cs Feature map for image reconstruction;

[0047] Step 5: Use the pre-trained VGG-19 to extract the features of the generated image and calculate the loss function. The specific formula is as follows:

[0048]

[0049] in is the content loss, is the style loss, is the identity loss, which is calculated as follows:

[0050] Step 5.1: The content loss calculation formula is as follows:

[0051]

[0052] where φ i Features extracted from the i-th layer of the pre-trained VGG-19 network;

[0053] Step 5.2: The style loss calculation formula is as follows:

[0054]

[0055] Among them, μ(·) represents the average value of the feature, φ i represents the features extracted from the i-th layer of the pre-trained VGG-19 network, and σ(·) represents the variance of the features;

[0056] Step 5.3: The identity loss is calculated as follows:

[0057]

[0058] Among them, I cc and I ss They represent the composite results generated when the input is the same content and style image; φ i refers to the features extracted from the relui_1(i∈[1-5]) layer of the VGG network, and λ id1 and λ id2 is a hyperparameter;

[0059] Step 6: Train the model and iteratively update the Adam optimizer through content loss, style loss, and identity loss to obtain the style transfer model weight file and decoder weight file;

[0060] Step 7: The trained model receives content images and style images of any resolution and outputs high-quality artistic style transfer photos.

[0061] It should be understood that parts not elaborated in detail in this specification belong to the prior art.

[0062] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.

Claims

1. A two-segment perceptual feature method for arbitrary image style transfer, characterized in that The specific steps include: Step 1: Obtain a recognized public image dataset and perform rotation and cropping preprocessing operations on the dataset. The dataset includes content images and style images. Step 2: The preprocessed content image I c and style image I s Passed into the pre-trained VGG-19 encoder E, the specific calculation formula is as follows: F c =E(I c ),F s =E(I s ); Step 3: Construct a two-segment perceptual feature network model for arbitrary image style transfer, including an attention module and a global statistical information alignment module; Step 3.1: Construct the attention module. In this stage, we first transform F c and F s After regularization, convolution operation is performed to obtain F q 、F k 、F v , let F q Make proxy token A, the proxy token is first used as query token F q Agent, from F k and F v Aggregate information and then broadcast it back to F q , the specific calculation formula is as follows: O A =Attn S (Q,A,Attn S (A,K,V)), Where Softmax is the activation function, F attn is the attention score feature map obtained by the proposed attention module; Step 3.2: Build a global statistical information alignment module and calculate the weighted statistics of each point. The calculation method is as follows: F cs =Std·Norm(F c )+Mean; Among them, Softmax is the activation function, and Norm represents mean variance normalization; Step 4: Construct the decoder. The decoder and encoder are designed symmetrically. cs Feature map for image reconstruction; Step 5: Use the pre-trained VGG-19 to extract the features of the generated image and calculate the loss function. The specific formula is as follows: in is the content loss, is the style loss, is the identity loss, which is calculated as follows: Step 5.1: The content loss calculation formula is as follows: where φ i Features extracted from the i-th layer of the pre-trained VGG-19 network; Step 5.2: The style loss calculation formula is as follows: Where μ(·) represents the mean value of the feature, φ represents the feature extracted from the i-th layer of the pre-trained VGG-19 network, and σ(·) represents the variance of the feature; Step 5.3: The identity loss is calculated as follows: Among them, I cc and I ss They represent the composite results generated when the input is the same content and style image; φ refers to the features extracted from the relui_1(i∈[1-5]) layer of the VGG network, and λ id1 and λ id2 is a hyperparameter; Step 6: Train the model and generate model weight and decoder weight files based on the loss function; Step 7: The trained model receives content images and style images of any resolution and outputs high-quality artistic style transfer photos.

Images