Image arbitrary artistic style migration method of affinity-enhanced attention network

Abstract

The invention discloses a method for migrating any artistic style of an image based on an affinity-enhanced attention network, and aims to create a brand-new image from a content image according to a target artistic style, so that the brand-new image not only has artistic features of a style image, but also keeps own content texture structure information. According to the method, firstly, a content image feature is extracted and content representation is enhanced by using a content affinity attention enhancement module, a style image feature is extracted by using a style affinity attention enhancement module, and then detail enhancement is performed on the content image feature by using a content image detail enhancement module. A style image detail enhancement module performs detail enhancement on style image features, and then a mixed attention module adjusts style feature distribution according to content feature distribution. A contrast experiment on a public image data set verifies that the method can realize any artistic style migration of high-quality images.

Description

Technical Field

[0001] The present invention belongs to the technical fields of computer vision and image classification, and specifically relates to a method for transferring arbitrary artistic styles of images. Background Art

[0002] Style transfer is one of the basic tasks of computer vision. Scientific research is rational and rigorous, while artistic creation is emotional and flexible. Image arbitrary artistic style transfer is a study that can achieve a perfect balance between scientific research and artistic creation. This study aims to decompose the color and texture features in the style image and transfer the features to the given content image to achieve image style conversion. Under the constraints of achieving style transfer, the features of the content image are retained as much as possible. In recent years, Gatys et al. used pre-trained deep convolutional neural networks for the first time to separate and recombine the content and style features of images, and pioneered the use of Gram matrices to quantify the correlation of style features, so as to capture and transfer the color and texture layout of the target style image. However, Gatys et al.'s method relies on an iterative gradient descent process to optimize the image, and the image generation speed is extremely slow.

[0003] In order to expand the application scenarios of style transfer, some researchers use feedforward network-based methods to efficiently complete this task. However, these methods can only achieve the transfer of a specific style. Another group of researchers are working hard to expand the scope of application of this research and transfer any artistic style of images. Summary of the invention

[0004] 1. A method for transferring arbitrary artistic styles of images using affinity-enhanced attention networks, characterized by designing a new network structure, specifically including the following steps:

[0005] Step 1: Obtain a recognized public image dataset and preprocess the dataset, including rotation, cropping, and regularization operations. The dataset includes a content image dataset and a style image dataset;

[0006] Step 2: The preprocessed content image I c and style image I s Passed into the pre-trained VGG-19 encoder E, the specific calculation formula is as follows:

[0007] F c =E(I c ),F s =E(I s );

[0008] Step 3: Construct an affinity-enhanced attention network, which includes five modules: content affinity-enhanced attention module, style affinity-enhanced attention module, content image detail enhancement module, style image detail enhancement module, and hybrid attention module;

[0009] Step 3.1: Content affinity enhanced attention module for F c Calculate the attention score to get F affc , the specific formula is as follows:

[0010] F cq =Conv(F c ),F ck =Conv(F c ),F cv =Conv(F c ),

[0011]

[0012] Where Conv refers to 1×1 learnable convolution, Softmax is the activation function, and F affc The feature map of the content image is obtained by the newly proposed attention calculation method;

[0013] Step 3.2: Style affinity enhanced attention module for F s Calculate the attention score to get F affs ;

[0014] F sq =Conv(F s ),F sk =Conv(F s ),F sv =Conv(F s ),

[0015]

[0016] Where Conv refers to 1×1 learnable convolution, Softmax is the activation function, and F affs The feature map of the style image is obtained by the newly proposed attention calculation method;

[0017] Step 3.3: The content image detail enhancement module first subtracts F from the attention feature map tensor affc The square of the matrix is ​​passed through the ReLU activation function, and the square root of this matrix is ​​taken; a new matrix weight matrix is ​​generated, and F cv After normalization, it is multiplied by the detail enhancement weight matrix and finally added to the original F affc Get F from the matrix cc , the specific calculation formula is as follows:

[0018]

[0019] Where ReLU is the activation function, Norm represents mean variance normalization, and F affc The feature map of the content image is obtained by the newly proposed attention calculation method;

[0020] Step 3.4: The style image detail enhancement module first subtracts F from the attention feature map tensor affs The square of the matrix is ​​passed through the ReLU activation function, and the square root of this matrix is ​​taken; a new matrix weight matrix is ​​generated, and F sv After normalization, it is multiplied by the detail enhancement weight matrix and finally added to the original F affs Get F from the matrix ss , the specific calculation formula is as follows:

[0021]

[0022] Where ReLU is the activation function, Norm represents mean variance normalization, and F affs The feature map of the style image is obtained by the newly proposed attention calculation method;

[0023] Step 3.5: The hybrid attention module is based on the content feature F cc Distribution to adjust the style feature F ss Get the feature map F cs Finally, we calculate F cs Feature map, the specific formula is as follows:

[0024]

[0025] Step 4: Construct the decoder. The decoder and encoder are designed symmetrically. cs Feature map for image reconstruction;

[0026] Step 5: Use the pre-trained VGG-19 to extract the features of the generated image and calculate the loss function. The specific formula is as follows:

[0027]

[0028] in is the content loss, is the style loss, It’s a loss of identity. is the local dissimilar content loss, is the local dissimilar style loss, which is calculated as follows:

[0029] Step 5.1: The content loss is calculated as follows:

[0030]

[0031] where φ i Features extracted from the i-th layer of the pre-trained VGG-19 network;

[0032] Step 5.2: The style loss is calculated as follows:

[0033]

[0034] Among them, μ(·) represents the average value of the feature, φ i represents the features extracted from the i-th layer of the pre-trained VGG-19 network, and σ(·) represents the variance of the features;

[0035] Step 5.3: The identity loss is calculated as follows:

[0036]

[0037] Among them, I cc and I ss They represent the composite results generated when the input is the same content and style image; φ refers to the features extracted from the relui_1(i∈[1-5]) layer of the VGG network, and λ id1 and λ id2 is a hyperparameter;

[0038] Step 5.4: Local dissimilar content loss is calculated by calculating the content loss between the original features and the randomly permuted features, and comparing the high-order feature maps of the two feature tensors to calculate the local dissimilar content loss. The specific calculation is as follows:

[0039]

[0040] Among them, I cs1 and I cs2 Represents images generated with the same content image but different style images;

[0041] Step 5.5: Local dissimilar style loss Calculating the style loss between the original style features and the arranged style features involves comparing multiple layers of feature maps to obtain the local dissimilar style loss, which is calculated as follows:

[0042]

[0043] Among them, I sc1 and I sc2 Represents images generated from the same style image but different content images;

[0044] Step 6: Train the model and iteratively update the style transfer model weight file through the Adam optimizer;

[0045] Step 7: Input the content image and style image of any resolution into the model weights and output a high-quality photo with image art style transfer.

[0046] The present invention proposes a method for transferring arbitrary artistic styles of images using affinity-enhanced attention networks. The beneficial effects brought by the provided technical solution include at least:

[0047] In an embodiment of the present invention, a neural network model for transferring arbitrary artistic styles of images is constructed; a content affinity enhanced attention module, a style affinity enhanced attention module, a content image detail enhancement module, a style image detail enhancement module and a hybrid attention module. The content affinity enhanced attention module and the style affinity enhanced attention module respectively extract the texture, color and structure information of the content image and the style image, pass the deep and shallow features into the content image detail enhancement module and the style image detail enhancement module to enhance the content representation, and then perform adaptive normalization processing on the content features and the style features so that their local feature statistics are the same as the feature statistics after detail enhancement. The attention module combines the shallow and deep convolutional neural network features of the style image and the content image, and achieves point-by-point feature statistical matching of the content features and the style features. The hybrid attention module adjusts the style features according to the content feature distribution to better capture the mapping relationship between the content and style features.

[0048] In summary, the main advantages of the present invention are:

[0049] 1. A content affinity enhanced attention module, a style affinity enhanced attention module, a content image detail enhancement module, a style image detail enhancement module, and a hybrid attention module are proposed, among which the content affinity enhanced attention module and the style affinity enhanced attention module capture the local feature information of the style image and the content image.

[0050] 2. The content image detail enhancement module and the style image detail enhancement module enhance the detail representation in the content and style images, significantly improving the image stylization effect.

[0051] 3. A local dissimilarity loss function based on affinity-enhanced attention is proposed to obtain better content affinity by retaining the information of content images during style transfer. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0053] Figure 1A flowchart of a method for transferring any artistic style of an image using an affinity-enhanced attention network is provided in an embodiment of the present invention.

[0054] Figure 2 A schematic diagram of the affinity-enhanced attention network structure provided in an embodiment of the present invention.

[0055] Figure 3 The model provided in the embodiment of the present invention generates a stylized image. DETAILED DESCRIPTION

[0056] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0057] like Figure 1 As shown, an embodiment of the present invention provides a method for transferring any artistic style of an image using an affinity-enhanced attention network, including:

[0058] 1. A method for transferring arbitrary artistic styles of images using affinity-enhanced attention networks, characterized by designing a new network structure, specifically including the following steps:

[0059] Step 1: Acquire and preprocess the image dataset. The present invention needs to use the content image dataset Microsoft COCO dataset and the style image dataset WikiArt to preprocess the dataset, including rotating, cropping, and regularizing the image.

[0060] Step 2: Use the pre-trained VGG-19 network as the encoder to extract the deep and shallow features of the content image and style image, and unify the shape of the input image tensor.

[0061] Step 3: Construct an affinity-enhanced attention network model. This paper proposes to construct an affinity-enhanced attention network model, which includes five modules: content affinity-enhanced attention module, style affinity-enhanced attention module, content image detail enhancement module, style image detail enhancement module and hybrid attention module. The content affinity-enhanced attention module first uses attention to enhance the content image representation, and the style affinity-enhanced attention module uses attention to enhance the style image representation. The proposed new attention calculation method, Softmax attention, calculates the similarity between all query keyword pairs, resulting in quadratic complexity. Linear attention uses a mapping function to change the calculation order and reduce the complexity, but the expression ability is insufficient. The attention proposed in the present invention uses a small number of tokens to aggregate and broadcast global information, thereby realizing a clever combination of Softmax and linear attention, and is compatible with the advantages of high expression ability and low computational complexity. Aiming at the problem of image detail loss in style transfer, the present invention designs a detail enhancement module, uses the content image detail enhancement module and the style image detail enhancement module to strengthen the detail features, and generates a weight matrix through the image feature content to solve the problem of detail loss. The hybrid attention module adjusts the style feature distribution according to the content feature distribution, so as to better capture the mapping relationship between content and style features. In addition, the present invention also proposes a local dissimilarity loss based on affinity attention to better maintain the affinity with the content image and the style image.

[0062] Step 4: Construct the decoder. The decoder and encoder are designed symmetrically. cs Feature map for image reconstruction;

[0063] Step 5: Use the pre-trained VGG-19 to extract the features of the generated image and calculate the loss function. The specific formula is as follows:

[0064]

[0065] in is the content loss, is the style loss, It’s a loss of identity. is the local dissimilar content loss, is the local dissimilar style loss, which is calculated as follows:

[0066] Step 5.1: The content loss is calculated as follows:

[0067]

[0068] where φ i Features extracted from the i-th layer of the pre-trained VGG-19 network;

[0069] Step 5.2: The style loss is calculated as follows:

[0070]

[0071] Among them, μ(·) represents the average value of the feature, φ i represents the features extracted from the i-th layer of the pre-trained VGG-19 network, and σ(·) represents the variance of the features;

[0072] Step 5.3: The identity loss is calculated as follows:

[0073]

[0074] Among them, I cc and I ss They represent the composite results generated when the input is the same content and style image; φ refers to the features extracted from the relui_1(i∈[1-5]) layer of the VGG network, and λ id1 and λ id2 is a hyperparameter;

[0075] Step 5.4: Local dissimilar content loss is calculated by calculating the content loss between the original features and the randomly permuted features, and comparing the high-order feature maps of the two feature tensors to calculate the local dissimilar content loss. The specific calculation is as follows:

[0076]

[0077] Among them, I cs1 and I cs2 Represents images generated with the same content image but different style images;

[0078] Step 5.5: Local dissimilar style loss Calculating the style loss between the original style features and the arranged style features involves comparing multiple layers of feature maps to obtain the local dissimilar style loss, which is calculated as follows:

[0079]

[0080] Among them, I sc1 and I sc2 Represents images generated from the same style image but different content images;

[0081] Step 6: Train the model and iteratively update the style transfer model weight file through the Adam optimizer;

[0082] Step 7: Input the content image and style image of any resolution into the model weights and output a high-quality photo with image art style transfer.

[0083] It should be understood that parts not elaborated in detail in this specification belong to the prior art.

[0084] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.

Claims

1. A method for transferring arbitrary artistic styles of images using affinity-enhanced attention networks, characterized in that A new network structure was designed, which includes the following steps: Step 1: Obtain a recognized public image dataset and preprocess the dataset, including rotation, cropping, and regularization operations. The dataset includes a content image dataset and a style image dataset; Step 2: The preprocessed content image I c and style image I s Passed into the pre-trained VGG-19 encoder E, the specific calculation formula is as follows: F c =E(I c ),F s =E(I s ); Step 3: Construct an affinity-enhanced attention network, which includes five modules: content affinity-enhanced attention module, style affinity-enhanced attention module, content image detail enhancement module, style image detail enhancement module, and hybrid attention module; Step 3.1: Content affinity enhanced attention module for F c Calculate the attention score to get F affc , the specific formula is as follows: F cq =Conv(F c ),F ck =Conv(F c ),F cv =Conv(F c ), Where Conv refers to 1×1 learnable convolution, Softmax is the activation function, and F affc The feature map of the content image is obtained by the newly proposed attention calculation method; Step 3.2: Style affinity enhanced attention module for F s Calculate the attention score to get F affs ; F sq =Conv(F s ),F sk =Conv(F s ),F sv =Conv(F S ), Where Conv refers to 1×1 learnable convolution, Softmax is the activation function, and F affs The feature map of the style image is obtained by the newly proposed attention calculation method; Step 3.3: The content image detail enhancement module first subtracts F from the attention feature map tensor affc The square of the matrix is ​​passed through the ReLU activation function, and the square root of this matrix is ​​taken; a new matrix weight matrix is ​​generated, and F cv After normalization, it is multiplied by the detail enhancement weight matrix and finally added to the original F affc Get F from the matrix cc , the specific calculation formula is as follows: Where ReLU is the activation function, Norm represents mean variance normalization, and F affc The feature map of the content image is obtained by the newly proposed attention calculation method; Step 3.4: The style image detail enhancement module first subtracts F from the attention feature map tensor affs The square of the matrix is ​​passed through the ReLU activation function, and the square root of this matrix is ​​taken; a new matrix weight matrix is ​​generated, and F sv After normalization, it is multiplied by the detail enhancement weight matrix and finally added to the original F affs Get F from the matrix ss , the specific calculation formula is as follows: Where ReLU is the activation function, Norm represents mean variance normalization, and F affs The feature map of the style image is obtained by the newly proposed attention calculation method; Step 3.5: The hybrid attention module is based on the content feature F cc Distribution to adjust the style feature F ss Get the feature map F cs Finally, we calculate F cs Feature map, the specific formula is as follows: Step 4: Construct the decoder. The decoder and encoder are designed symmetrically. cs Feature map for image reconstruction; Step 5: Use the pre-trained VGG-19 to extract the features of the generated image and calculate the loss function. The specific formula is as follows: in is the content loss, is the style loss, It’s a loss of identity. is the local dissimilar content loss, is the local dissimilar style loss, which is calculated as follows: Step 5.1: The content loss is calculated as follows: where φ i Features extracted from the i-th layer of the pre-trained VGG-19 network; Step 5.2: The style loss is calculated as follows: Among them, μ(·) represents the average value of the feature, φ i represents the features extracted from the i-th layer of the pre-trained VGG-19 network, and σ(·) represents the variance of the features; Step 5.3: The identity loss is calculated as follows: Among them, I cc and I ss They represent the composite results generated when the input is the same content and style image; φ refers to the features extracted from the relui_1(i∈[1-5]) layer of the VGG network, and λ id1 and λ id2 is a hyperparameter; Step 5.4: Local dissimilar content loss is calculated by calculating the content loss between the original features and the randomly permuted features, and comparing the high-order feature maps of the two feature tensors to calculate the local dissimilar content loss. The specific calculation is as follows: Among them, I cs1 and I cs2 Represents images generated with the same content image but different style images; Step 5.5: Local dissimilar style loss Calculating the style loss between the original style features and the arranged style features involves comparing multiple layers of feature maps to obtain the local dissimilar style loss, which is calculated as follows: Among them, I sc1 and I sc2 Represents images generated from the same style image but different content images; Step 6: Train the model and iteratively update the style transfer model weight file through the Adam optimizer; Step 7: Input the content image and style image of any resolution into the model weights and output a high-quality photo with image art style transfer.

Images