Image style migration method based on semantic alignment

Abstract

The invention discloses an image style migration method and device based on semantic alignment, a medium and equipment, and the method comprises the steps: obtaining a style image and a content image, processing the style image and the content image through a pre-constructed style migration network, and obtaining a stylized image, and the style migration network comprises a coding module, a decoding module, a first style attention module, a second style attention module, a first adaptive decoupling module, a second adaptive decoupling module, a first dynamic style modulation core module and a second dynamic style modulation core module, and outputs the stylized image. The method comprises the following steps: mapping features of a content graph to content features of a style graph through a style attention module to establish a mapping relationship, introducing a content-based dynamic style modulation core module to adjust a mapping matrix, and simultaneously realizing explicit decoupling of content and style by using a self-adaptive decoupling module; therefore, an image style migration effect with more harmonious semantics and styles is realized.

Description

Technical Field

[0001] The present application relates to the field of image migration technology, and in particular to a method, device, medium and equipment for image style migration based on semantic alignment. Background Art

[0002] Image style transfer aims to transfer the style of a style image to the content of a content image to form a unique stylized image. The application areas of style transfer are also very wide, such as digital art creation and design, cultural relics restoration, calligraphy style transfer, etc., and it has broad application prospects and research value. Style transfer must meet two requirements at the same time: maintain the semantic structure of the content image and apply the global and local texture and style features of the style image. However, most existing methods lack comprehensive consideration of the feature information of the original content, such as depth, saliency, structure, and color, resulting in the loss of the original visual effect and emotional expression. Therefore, existing methods cannot produce harmonious and high-quality style transfer effects. Summary of the invention

[0003] The main purpose of this application is to provide a method, device, medium and equipment for image style transfer based on semantic alignment, which aims to adjust the mapping matrix based on the content-based dynamic style modulation kernel module, and use an adaptive decoupling module to achieve explicit decoupling of content and style, thereby achieving an image style transfer effect with more harmonious semantics and style.

[0004] To achieve the above-mentioned purpose, the present application provides an image style transfer method based on semantic alignment, including: obtaining a style image and a content image; using a pre-built style transfer network to process the style image and the content image to obtain a stylized image, wherein the style transfer network includes an encoding module, a decoding module, a first style attention module, a second style attention module, a first adaptive decoupling module, a second adaptive decoupling module, a first dynamic style modulation core module and a second dynamic style modulation core module; using the encoding module to process the style image and the content image, obtaining a first style feature map and a first content feature map in a first intermediate layer of the encoding module, and obtaining a second style feature map and a second content feature map in a second intermediate layer of the encoding module; using the first style attention module to process the first style feature map and the first content feature map to obtain a first attention map, and using the second style attention module to process the second style feature map and the second content feature map to obtain a second attention map; using the first dynamic The style modulation core module and the second dynamic style modulation core module adjust the first attention map and the second attention map respectively, and obtain the first adjustment map and the second adjustment map accordingly; obtain the first embedded style feature map based on the first style feature map and the first adjustment map, and obtain the second embedded style feature map based on the second style feature map and the second adjustment map; use the first adaptive decoupling module and the second adaptive decoupling module to perform de-stylization processing on the first content feature map and the second content feature map respectively, and obtain the first decoupled feature map and the second decoupled feature map accordingly; and obtain the first fusion map according to the addition result of the first decoupled feature map and the first embedded style feature map after convolution processing, and obtain the second fusion map according to the addition result of the second decoupled feature map and the second embedded style feature map after convolution processing; combine the first fusion map and the second fusion map to obtain the image to be decoded; use the decoding module to process the image to be decoded to obtain the stylized image; output the stylized image.

[0005] Optionally, before using the encoding module to process the style image and the content image, obtaining a first style feature map and a first content feature map in a first intermediate layer of the encoding module, and obtaining a second style feature map and a second content feature map in a second intermediate layer of the encoding module, the method further includes: obtaining a VGG-19 network and determining the encoding module based on the VGG-19 network.

[0006] Optionally, the using the first style attention module to process the first style feature map and the first content feature map to obtain a first attention map, and using the second style attention module to process the second style feature map and the second content feature map to obtain a second attention map, includes: using the first style attention module to perform mean square normalization processing on both the first style feature map and the first content feature map, and correspondingly obtain a first normalized style matrix and a first normalized content matrix; using the second style attention module to perform mean square normalization processing on both the second style feature map and the second content feature map, and correspondingly obtain a second normalized style matrix and a second normalized content matrix; using the first style attention module to perform mean square normalization processing on the first normalized style matrix and the first normalized content matrix, respectively Perform a convolution operation to obtain a first style feature space and a first content feature space; use the second style attention module to perform a convolution operation on the second normalized style matrix and the second normalized content matrix respectively to obtain a second style feature space and a second content feature space; use the first expression preset in the first style attention module to calculate the correlation between each point in the first style feature space and each point in the first content feature space to obtain each element of the first attention matrix, and determine a first attention map based on the first attention matrix; use the first expression preset in the first style attention module to calculate the correlation between each point in the second style feature space and each point in the second content feature space to obtain each element of the second attention matrix, and determine a second attention map based on the second attention matrix.

[0007] Optionally, the first dynamic style modulation core module and the second dynamic style modulation core module are used to adjust the first attention map and the second attention map respectively, and the first adjustment map and the second adjustment map are obtained accordingly, and the first embedded style feature map is obtained based on the first style feature map and the first adjustment map, and the second embedded style feature map is obtained based on the second style feature map and the second adjustment map, including: using a preset second expression to process the first attention matrix and the second attention matrix respectively, and the first average matrix and the second average matrix are obtained accordingly; based on the encoding module, a scale parameter and a bias parameter are determined, and based on the first average matrix, the second average matrix, the first attention matrix, The second attention matrix, the scale parameter and the bias parameter construct a first threshold matrix and a second threshold matrix; based on the attention matrix, the first threshold matrix, the second threshold matrix and the Sign function, determine a first dynamic style modulation kernel module and a second dynamic style modulation kernel module; use the first dynamic style modulation kernel module and the second dynamic style modulation kernel module to process the first attention map and the second attention map respectively, and obtain a first adjustment map and a second adjustment map accordingly; perform a dot product operation on the first style feature map and the first adjustment map to obtain the first embedded style feature map, and perform a dot product operation on the second style feature map and the second adjustment map to obtain the second embedded style feature map.

[0008] Optionally, the first adaptive decoupling module and the second adaptive decoupling module are used to perform de-stylization processing on the first content feature map and the second content feature map respectively, and a first decoupled feature map and a second decoupled feature map are correspondingly obtained, and a first fusion map is obtained according to the addition result of the first decoupled feature map and the first embedded style feature map after the convolution processing, and a second fusion map is obtained according to the addition result of the second decoupled feature map and the second embedded style feature map after the convolution processing, including: using the first adaptive decoupling module and the second adaptive decoupling module to perform whitening transformation processing on the first content feature map and the second content feature map respectively, and a first feature map and a second feature map are correspondingly obtained; using the first convolution layer of the first adaptive decoupling module to process the first feature map to obtain a first sub-feature map and a second sub-feature map, processing the first sub-feature map and the second sub-feature map based on a third expression preset in the first adaptive decoupling module to obtain a first content space attention map, using the second convolution layer of the first adaptive decoupling module to process the first content feature map to obtain a third sub-feature map, and processing the third sub-feature map based on a fourth expression preset in the first adaptive decoupling module. The first content space attention map and the first content feature map obtain a first decoupled feature map; the second feature map is processed by the first convolution layer of the second adaptive decoupling module to obtain a fourth sub-feature map and a fifth sub-feature map, the fourth sub-feature map and the fifth sub-feature map are processed based on the third expression preset in the second adaptive decoupling module to obtain a second content space attention map, the second content feature map is processed by the second convolution layer in the second adaptive decoupling module to obtain a sixth sub-feature map, and the sixth sub-feature map, the second content space attention map and the first content feature map are processed based on the fourth expression preset in the second adaptive decoupling module to obtain a second decoupled feature map; the first embedded style feature map and the second embedded style feature map are processed by the third convolution layer of each of the first adaptive decoupling module and the second adaptive decoupling module, respectively, to obtain a first scale feature map and a second scale feature map; the first adaptive decoupling module is used to add the first decoupled feature map and the first scale feature map to obtain the first fusion map; the second adaptive decoupling module is used to add the second decoupled feature map and the second scale feature map to obtain the second fusion map.

[0009] Optionally, the fifth expression is:

[0010]

[0011] Among them, conv 3×3 represents the convolution module, and There are two output feature maps representing the first fusion map and the second fusion map.

[0012] Optionally, the image style transfer method based on semantic alignment further includes: training the style transfer network using a preset loss function to obtain a final style transfer network, wherein the expression of the loss function is:

[0013]

[0014] Among them, L is the total number of convolutional layers, is the content loss function of the i-th convolutional layer of the deep neural network, is the style loss function of the i-th convolutional layer of the deep neural network, and is the disentanglement loss function of the i-th convolutional layer of the deep neural network, λ c , s , dis_c , dis_s They are the corresponding weights for controlling content loss, style loss, content disentanglement loss, and style disentanglement loss respectively.

[0015] On the basis of the above embodiments, the present application also provides an image style transfer method based on semantic alignment, including: an acquisition module, which acquires a style image and a content image; a processing module, which uses a pre-built style transfer network to process the style image and the content image to obtain a stylized image, wherein the style transfer network includes an encoding module, a decoding module, a first style attention module, a second style attention module, a first adaptive decoupling module, a second adaptive decoupling module, a first dynamic style modulation core module, and a second dynamic style modulation core module; the encoding module is used to process the style image and the content image, and a first style feature map and a first content feature map are obtained in a first intermediate layer of the encoding module, and a second style feature map and a second content feature map are obtained in a second intermediate layer of the encoding module; the first style attention module is used to process the first style feature map and the first content feature map to obtain a first attention map, and the second style attention module is used to process the second style feature map and the second content feature map to obtain a second attention map; The first dynamic style modulation core module and the second dynamic style modulation core module adjust the first attention map and the second attention map respectively, and obtain the first adjustment map and the second adjustment map accordingly; obtain the first embedded style feature map based on the first style feature map and the first adjustment map, and obtain the second embedded style feature map based on the second style feature map and the second adjustment map; use the first adaptive decoupling module and the second adaptive decoupling module to perform destylization processing on the first content feature map and the second content feature map respectively, and obtain the first decoupled feature map and the second decoupled feature map accordingly; obtain the first fusion map according to the addition result of the first decoupled feature map and the first embedded style feature map after convolution processing, and obtain the second fusion map according to the addition result of the second decoupled feature map and the second embedded style feature map after convolution processing; combine the first fusion map and the second fusion map to obtain a to-be-decoded image; use the decoding module to process the to-be-decoded image to obtain the stylized image; output module, used to output the stylized image. .

[0016] In addition, to achieve the above objectives, the present application also provides a computer-readable storage medium, which includes instructions, which, when executed on a computer, enables the computer to execute the semantic alignment-based image style transfer method provided in any of the above embodiments.

[0017] In addition, to achieve the above-mentioned purpose, the present application also provides an electronic device, which includes: at least one processor, a memory and an input and output unit; wherein the memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the semantic alignment-based image style transfer method provided in the aforementioned embodiment.

[0018] The embodiments of the present application propose a method, apparatus, medium and device for image style transfer based on semantic alignment, which obtains a style image and a content image; uses a pre-built style transfer network to process the style image and the content image to obtain a stylized image, wherein the style transfer network includes an encoding module, a decoding module, a first style attention module, a second style attention module, a first adaptive decoupling module, a second adaptive decoupling module, a first dynamic style modulation kernel module and a second dynamic style modulation kernel module, outputs the stylized image, maps the features of the content graph to the content features of the style graph through an attention mechanism, establishes a mapping relationship between them, introduces a content-based dynamic style modulation kernel module to adjust the mapping matrix, and uses an adaptive decoupling module to achieve explicit decoupling of content and style, thereby achieving an image style transfer effect with more harmonious semantics and style. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A flowchart diagram of an embodiment of an image style transfer method based on semantic alignment provided in this application;

[0020] Figure 2 A schematic diagram of a style transfer network flow based on semantic alignment provided in an embodiment of an image style transfer method based on semantic alignment of the present application;

[0021] Figure 3 A schematic diagram of a style transfer network processing process based on semantic alignment provided in an embodiment of an image style transfer method based on semantic alignment of the present application;

[0022] Figure 4 A schematic diagram of functional modules provided for an embodiment of an image style transfer method based on semantic alignment of the present application.

[0023] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0024] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0025] The key goal of image style transfer is to meet the application needs of users in social media and digital entertainment. By training an end-to-end forward network, only one forward propagation is used for style transfer during the test phase to meet the real-time requirements. Through social media surveys, it is found that users show a clear preference for semantically matched image style transfer results, that is, the style of the generated image is consistent with the style image at the semantic level. Existing image style transfer methods use the Gram matrix based on the global deep features of the image as the statistical representation of the style, match the image as a whole, and evenly transfer the overall style to the content area that is not related to semantics, resulting in distorted results or incorrect style patterns. Such a mismatch between style and content semantics will destroy the overall beauty of the generated image and affect the visual quality of the generated results.

[0026] The purpose of the present invention is to provide an image style transfer method based on semantic alignment, which establishes a mapping relationship between the features of the content graph and the content features of the style graph through an attention mechanism, introduces a content-based dynamic style modulation kernel module to adjust the mapping matrix, and uses an adaptive decoupling module to achieve explicit decoupling of content and style, thereby achieving an image style transfer effect with more harmonious semantics and style.

[0027] Reference Figure 1 and Figure 2 The first embodiment of the present application provides an image style transfer method based on semantic alignment, which can be executed by a processor in an electronic device. The image style transfer method based on semantic alignment may include:

[0028] S10, obtaining a style image and a content image;

[0029] S20, using a pre-built style transfer network to process the style image and the content image to obtain a stylized image;

[0030] The style transfer network includes an encoding module, a decoding module, a first style attention module, a second style attention module, a first adaptive decoupling module, a second adaptive decoupling module, a first dynamic style modulation kernel module, and a second dynamic style modulation kernel module;

[0031] Processing the style image and the content image using the encoding module, obtaining a first style feature map and a first content feature map in a first intermediate layer of the encoding module, and obtaining a second style feature map and a second content feature map in a second intermediate layer of the encoding module;

[0032] Processing the first style feature map and the first content feature map using a first style attention module to obtain a first attention map, and processing the second style feature map and the second content feature map using a second style attention module to obtain a second attention map;

[0033] Using the first dynamic style modulation kernel module and the second dynamic style modulation kernel module to adjust the first attention map and the second attention map respectively, and obtain the first adjustment map and the second adjustment map respectively, obtain the first embedded style feature map based on the first style feature map and the first adjustment map, and obtain the second embedded style feature map based on the second style feature map and the second adjustment map;

[0034] De-stylizing the first content feature map and the second content feature map using the first adaptive decoupling module and the second adaptive decoupling module, respectively, to obtain a first decoupled feature map and a second decoupled feature map, and obtaining a first fusion map based on the addition result of the first decoupled feature map and the first embedded style feature map after the convolution process, and obtaining a second fusion map based on the addition result of the second decoupled feature map and the second embedded style feature map after the convolution process;

[0035] Combining the first fused image and the second fused image to obtain an image to be decoded;

[0036] The decoding module is used to process the image to be decoded to obtain a stylized image;

[0037] Specifically, the style transfer network proposed in the present invention consists of four modules: the encoding and decoding module (E, D), the style attention module SANet, the adaptive decoupling module Decoupling and the dynamic style modulation kernel module StyKernel. Our style transfer network is based on the content image I c and style image I s As input, use the content image I c The semantic structure and style image I s The features of the synthesized stylized image I cs . refer to Figure 3 , from the content image I c and style image I s Extract the VGG feature map F of each layer of the encoding module c =E(I c ) and F s =E(I s ). In order to combine the advantages of multi-scale feature representation, a multi-level strategy is adopted: the first content feature map extracted by the relu3_1 layer and relu4_1 layer of the VGG-19 network is used First style feature map Second content feature map Second style feature map Two style attention modules are trained separately, and the dynamic style modulation kernel module is used to adjust the attention map, and the adaptive decoupling module is used to adjust the content image feature F c De-stylization enables a better style embedding process for content images, thereby achieving an image style transfer effect with more harmonious semantics and style.

[0038] It should be noted that the first style attention module and the second style attention module have exactly the same structure. Without loss of generality, the following is an example in which a style attention module processes the first intermediate module of an encoding module.

[0039] The processor combines the deep features of the content image and the style image Input into the first style attention module SANet to obtain the first attention map A:

[0040]

[0041] The processor determines the content characteristics F c Further adjust the attention map. c As input, the encoding module E is used to generate the scale parameter λ and the bias parameter β to form a dynamic style modulation kernel module and generate an embedded style feature map F cs :

[0042]

[0043] The processor processes two content images Adaptive decoupling is performed respectively to remove the stylization to obtain the first decoupled feature map and the second decoupled feature map. Then, the two embedded style feature maps are transferred using a 1×1 convolutional layer and are respectively combined with the decoupled first decoupled feature map and the second decoupled feature map. Add the content features and the embedded style feature map F cs Fusion is performed to obtain the first fusion image and the second fusion image:

[0044]

[0045] In order to utilize style features of different scales, the processor combines the first fusion image and the second fusion image of different scales to obtain the image to be decoded F csc :

[0046]

[0047] Finally, the processor will F csc Input the decoder to get the style transfer image:

[0048] I cs =D(F csc )

[0049] In an embodiment of the present application, before processing the style image and the content image using the encoding module, obtaining the first style feature map and the first content feature map in the first intermediate layer of the encoding module, and obtaining the second style feature map and the second content feature map in the second intermediate layer of the encoding module, the method further includes:

[0050] Obtain a VGG-19 network and determine a coding module based on the VGG-19 network;

[0051] A style image and a content image are input into the encoding module, a first style feature map and a first content feature map are correspondingly output in the first intermediate layer of the encoding module, and a second style feature map and a second content feature map are correspondingly output in the second intermediate layer.

[0052] The processor first uses I c and I s The image pair is taken as input, and the encoding module extracts the feature map F at specific layers (the first intermediate layer and the second intermediate layer) respectively. c (first content feature map and second content feature map) and F s (first style feature map and second style feature map),

[0053] In one embodiment of the present application, using a first style attention module to process a first style feature map and a first content feature map to obtain a first attention map, and using a second style attention module to process a second style feature map and a second content feature map to obtain a second attention map, including:

[0054] Using the first style attention module, performing mean square error normalization processing on the first style feature map and the first content feature map, and obtaining a first normalized style matrix and a first normalized content matrix respectively;

[0055] Using the second style attention module, the second style feature map and the second content feature map are both subjected to mean square error normalization processing, and a second normalized style matrix and a second normalized content matrix are correspondingly obtained;

[0056] Using the first style attention module, performing convolution operations on the first normalized style matrix and the first normalized content matrix respectively to obtain a first style feature space and a first content feature space;

[0057] Using a second style attention module, performing convolution operations on the second normalized style matrix and the second normalized content matrix respectively to obtain a second style feature space and a second content feature space;

[0058] Calculate the correlation between each point in the first style feature space and each point in the first content feature space using a first expression preset in the first style attention module to obtain each element of a first attention matrix, and determine a first attention map based on the first attention matrix;

[0059] The first expression preset in the first style attention module is used to calculate the correlation between each point in the second style feature space and each point in the second content feature space, to obtain each element of the second attention matrix, and to determine the second attention map based on the second attention matrix.

[0060] Exemplarily, the processor calculates the content feature F c and style features F s The feature correlation between them obtains style attention.

[0061] The processor normalizes the content and style feature maps using mean variance, expressed as and That is to say, the first style feature map and the first content feature map are both subjected to mean square error normalization processing, and the first normalized style matrix and the first normalized content matrix are obtained accordingly; the second style feature map and the second content feature map are both subjected to mean square error normalization processing, and the second normalized style matrix and the second normalized content matrix are obtained accordingly.

[0062] Taking the first style attention module as an example, the processor can convert the first normalized style matrix and the first normalized content matrix into two feature spaces f and g through a learnable 1×1 convolution layer. In other words, the first normalized style matrix and the first normalized content matrix are converted into the first style feature space and the first content feature space respectively using a 1×1 convolution operation; the second normalized style matrix and the second normalized content matrix are converted into the second style feature space and the second content feature space respectively using a 1×1 convolution operation.

[0063] The processor then obtains the feature correlation matrix of the corresponding position by calculating the inner product between the vectorized feature maps. The feature correlation matrix is ​​of size N l ×N l The feature correlation matrix A (first style attention module)). Each element A in A ij A represents the correlation between the filter response of the content feature at position i and the style feature at position j, ij It can be calculated by the first expression, where the first expression is:

[0064]

[0065] In an embodiment of the present application, the first attention map and the second attention map are adjusted respectively by using the first dynamic style modulation core module and the second dynamic style modulation core module, and the first adjustment map and the second adjustment map are obtained accordingly, and the first embedded style feature map is obtained based on the first style feature map and the first adjustment map, and the second embedded style feature map is obtained based on the second style feature map and the second adjustment map, including:

[0066] The first attention matrix and the second attention matrix are processed respectively using a preset second expression to obtain a first average matrix and a second average matrix respectively;

[0067] Determine a scale parameter and a bias parameter based on the encoding module, and construct a first threshold matrix and a second threshold matrix based on the first average matrix, the second average matrix, the first attention matrix, the second attention matrix, the scale parameter and the bias parameter;

[0068] Determine a first dynamic style modulation kernel module and a second dynamic style modulation kernel module based on the attention matrix, the first threshold matrix, the second threshold matrix and the Sign function;

[0069] Using the first dynamic style modulation kernel module and the second dynamic style modulation kernel module to process the first attention map and the second attention map respectively, and obtain the first adjustment map and the second adjustment map accordingly;

[0070] A dot product operation is performed on the first style feature map and the first adjustment map to obtain a first embedded style feature map, and a dot product operation is performed on the second style feature map and the second adjustment map to obtain a second embedded style feature map.

[0071] It should be noted that the first dynamic style modulation kernel module and the second dynamic style modulation kernel module have exactly the same structure. Without loss of generality, a dynamic style modulation kernel module is taken as an example for introduction. In the dynamic style modulation kernel module StyKernel, the processor first averages the attention matrix A using the second expression to obtain an average matrix, where the second expression can be

[0072] Then, the processor determines the scale parameter and the bias parameter based on the encoding module, and constructs a threshold matrix based on the average matrix, the attention matrix, the scale parameter and the bias parameter. The constructed threshold matrix can be

[0073] Finally, the processor determines the dynamic style modulation kernel module based on the attention matrix, threshold matrix and Sign function The dynamic style modulation kernel module can be expressed as Where Sign is a function that returns 1 if the input is positive, otherwise it returns 0. The processor uses the dynamic style modulation kernel module to create the focus region and filter out irrelevant style features, so that the processor can aggregate features from highly relevant features, which is crucial to maintain the integrity of the semantic region.

[0074] In an embodiment of the present application, a first content feature map and a second content feature map are de-stylized using a first adaptive decoupling module and a second adaptive decoupling module, respectively, to obtain a first decoupled feature map and a second decoupled feature map, and a first fusion map is obtained according to the addition result of the first decoupled feature map and the first embedded style feature map after the convolution process, and a second fusion map is obtained according to the addition result of the second decoupled feature map and the second embedded style feature map after the convolution process, including:

[0075] Using the first adaptive decoupling module and the second adaptive decoupling module to perform whitening transformation processing on the first content feature map and the second content feature map respectively, so as to obtain the first feature map and the second feature map respectively;

[0076] Processing the first feature map using the first convolution layer of the first adaptive decoupling module to obtain a first sub-feature map and a second sub-feature map, processing the first sub-feature map and the second sub-feature map based on a third expression preset in the first adaptive decoupling module to obtain a first content space attention map, processing the first content feature map using the second convolution layer of the first adaptive decoupling module to obtain a third sub-feature map, and processing the third sub-feature map, the first content space attention map, and the first content feature map based on a fourth expression preset in the first adaptive decoupling module to obtain a first decoupled feature map;

[0077] Processing the second feature map using the first convolution layer of the second adaptive decoupling module to obtain a fourth sub-feature map and a fifth sub-feature map, processing the fourth sub-feature map and the fifth sub-feature map based on the third expression preset in the second adaptive decoupling module to obtain a second content space attention map, processing the second content feature map using the second convolution layer in the second adaptive decoupling module to obtain a sixth sub-feature map, and processing the sixth sub-feature map, the second content space attention map, and the second content feature map based on the fourth expression preset in the second adaptive decoupling module to obtain a second decoupled feature map;

[0078] Using the third convolutional layers of the first adaptive decoupling module and the second adaptive decoupling module to process the first embedded style feature map and the second embedded style feature map respectively, and correspondingly obtain a first scale feature map and a second scale feature map;

[0079] Using a first adaptive decoupling module to perform sum processing on the first decoupled feature map and the first scale feature map to obtain a first fusion map;

[0080] The second adaptive decoupling module is used to perform sum processing on the second decoupled feature map and the second scale feature map to obtain a second fusion map.

[0081] Exemplarily, for any first content feature map and second content feature map, they are processed by the first adaptive decoupling module and the second adaptive decoupling module respectively, wherein the first adaptive decoupling module and the second adaptive decoupling module have the same structure, without loss of generality, in the adaptive decoupling module Decoupling, given a content feature map F c The processor uses a whitening transform to remove style-related texture information to obtain That is, two 1×1 convolutional layers are used to generate two new feature maps, namely sub-feature maps and sub-feature map At the same time, the first content feature map F c Feed to another convolutional layer to generate new feature maps, i.e. sub-feature maps Using the third expression Resize and get the content space attention map A c , it should be noted that the third expression is as follows:

[0082]

[0083] in Represents matrix multiplication; Representation feature map and Then, the processor obtains the decoupled feature map after decoupling through matrix multiplication and element addition in the fourth expression: It should be noted that the fourth expression is as follows:

[0084]

[0085] The decoupled feature map after decoupling and the embedded style feature map F cs Fusion is performed to obtain the output feature F of a specific scale csc , which encodes the structure of the content image and is consistent with the style of the style image at the semantic level:

[0086]

[0087] The processor converts the output features of different scales The fifth expression is used to combine the output features F csc Input the decoder to get the style transfer image.

[0088]

[0089] in, and are two output feature maps, The feature dimension is 256×64×64, where 256 is the number of channels and 64 is the width and height of the feature map. The feature dimension of is 512×32×32, where 512 is the number of channels and 32 is the width and height of the feature map. After channel dimension reduction and upsampling, Then, a 3×3 convolutional layer is used to combine the two feature maps.

[0090] Finally, F csc Input the decoder to get the style transfer image:

[0091] I cs =D(F csc ).

[0092] S30: output a stylized image.

[0093] Based on the above embodiments, the present application also provides a training method for a style transfer network, and the processor can train the style transfer network using a preset loss function. Specifically, the processor can use a pre-trained VGG-19 network as an encoding module and a symmetric decoder, and jointly train two style attention modules, an adaptive decoupling module, and a dynamic style modulation kernel module for arbitrary style transfer.

[0094] The final style transfer network is obtained, where the loss function is expressed as:

[0095]

[0096] Among them, L is the total number of convolutional layers, is the content loss function of the i-th convolutional layer of the deep neural network, is the style loss function of the i-th convolutional layer of the deep neural network, and is the disentanglement loss function of the i-th convolutional layer of the deep neural network. c , s , dis_c , dis_s They are the corresponding weights for controlling content loss, style loss, content disentanglement loss, and style disentanglement loss respectively.

[0097] For example, our network includes three loss functions in the training process, namely content-aware loss, style-aware loss, and disentanglement loss. content It is used to minimize the content difference between the generated image and the content image, and the content-aware loss is defined as the Euclidean distance between the deep feature representation of the content image and the deep feature representation of the output result: Style-aware loss L style To minimize the style difference between the generated image and the style image: Among them, φ i (·) represents the feature extracted from the i-th layer in the pre-trained VGG19, μ(·) represents the mean of the feature, and σ(·) represents the variance of the feature. The disentanglement loss consists of the content disentanglement loss and the style disentanglement loss. The content disentanglement loss makes the content features extracted from the stylized results similar, and the style disentanglement loss makes the style features extracted from the stylized results similar:

[0098]

[0099] in, and The results of using the same content image and different style images are: and Results of using the same style image and different content images.

[0100] The expression of the total loss function is:

[0101]

[0102] refer to Figure 4Based on the above embodiments, the present application further provides an image style migration device based on semantic alignment. The image style migration device based on semantic alignment 100 includes an acquisition module 1001, a processing module 1002 and an output module 1003, wherein the acquisition module 1001 acquires a style image and a content image; the processing module 1002 is used to use a pre-built style migration network to process the style image and the content image to obtain a stylized image, wherein the style migration network includes an encoding module, a decoding module, a first style attention module, a second style attention module, a first adaptive decoupling module, a second adaptive decoupling module, a first dynamic style modulation kernel module and a second dynamic style modulation kernel module; the encoding module is used to process the style image and the content image, and a first style feature map and a first content feature map are obtained in a first intermediate layer of the encoding module, and a second style feature map and a second content feature map are obtained in a second intermediate layer of the encoding module; the first style feature map and the first content feature map are processed by the first style attention module to obtain a first attention map, and the second style feature map is processed by the second style attention module to obtain a second attention map. The attention module processes the second style feature map and the second content feature map to obtain a second attention map; the first dynamic style modulation kernel module and the second dynamic style modulation kernel module are used to adjust the first attention map and the second attention map respectively, and the first adjustment map and the second adjustment map are obtained accordingly; the first embedded style feature map is obtained based on the first style feature map and the first adjustment map, and the second embedded style feature map is obtained based on the second style feature map and the second adjustment map; the first content feature map and the second content feature map are de-stylized using the first adaptive decoupling module and the second adaptive decoupling module respectively, and the first decoupled feature map and the second decoupled feature map are obtained accordingly, and the first fusion map is obtained according to the addition result of the first decoupled feature map and the first embedded style feature map after convolution processing, and the second fusion map is obtained according to the addition result of the second decoupled feature map and the second embedded style feature map after convolution processing; the first fusion map and the second fusion map are combined to obtain a to-be-decoded image; the decoding module is used to process the to-be-decoded image to obtain a stylized image; the output module 1003 is used to output the stylized image.

[0103] Based on the above embodiments, the present application also provides a computer-readable storage medium, which includes instructions, which, when executed on a computer, enables the computer to execute the semantic alignment-based image style transfer method provided in any of the above embodiments.

[0104] Based on the above embodiments, the present application also provides an electronic device, which includes: at least one processor, a memory and an input and output unit; wherein the memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the semantic alignment-based image style transfer method provided according to the above embodiments.

[0105] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for image style transfer based on semantic alignment, characterized in that: include: Get style image and content image; Processing the style image and the content image using a pre-built style transfer network to obtain a stylized image, wherein the style transfer network includes an encoding module, a decoding module, a first style attention module, a second style attention module, a first adaptive decoupling module, a second adaptive decoupling module, a first dynamic style modulation kernel module, and a second dynamic style modulation kernel module; Processing the style image and the content image by using the encoding module, obtaining a first style feature map and a first content feature map in a first intermediate layer of the encoding module, and obtaining a second style feature map and a second content feature map in a second intermediate layer of the encoding module; Using the first style attention module to process the first style feature map and the first content feature map to obtain a first attention map, and using the second style attention module to process the second style feature map and the second content feature map to obtain a second attention map; Using the first dynamic style modulation kernel module and the second dynamic style modulation kernel module to adjust the first attention map and the second attention map respectively, and obtain a first adjustment map and a second adjustment map accordingly, obtain a first embedded style feature map based on the first style feature map and the first adjustment map, and obtain a second embedded style feature map based on the second style feature map and the second adjustment map; Using the first adaptive decoupling module and the second adaptive decoupling module to perform destylization processing on the first content feature map and the second content feature map respectively, and obtain a first decoupled feature map and a second decoupled feature map respectively, and obtain a first fusion map according to the addition result of the first decoupled feature map and the first embedded style feature map after the convolution processing, and obtain a second fusion map according to the addition result of the second decoupled feature map and the second embedded style feature map after the convolution processing; Combining the first fused graph and the second fused graph to obtain a to-be-decoded graph; Processing the to-be-decoded image using the decoding module to obtain the stylized image; The stylized image is output.

2. The image style transfer method based on semantic alignment according to claim 1, characterized in that: Before the method of processing the style image and the content image by using the encoding module, obtaining a first style feature map and a first content feature map in a first intermediate layer of the encoding module, and obtaining a second style feature map and a second content feature map in a second intermediate layer of the encoding module, the method further includes: A VGG-19 network is obtained and the encoding module is determined based on the VGG-19 network.

3. The image style transfer method based on semantic alignment according to claim 1, characterized in that: The method of using the first style attention module to process the first style feature map and the first content feature map to obtain a first attention map, and using the second style attention module to process the second style feature map and the second content feature map to obtain a second attention map, includes: Using the first style attention module, performing mean square error normalization processing on the first style feature map and the first content feature map, and correspondingly obtaining a first normalized style matrix and a first normalized content matrix; Using the second style attention module, performing mean square error normalization processing on the second style feature map and the second content feature map, and correspondingly obtaining a second normalized style matrix and a second normalized content matrix; Using the first style attention module, performing convolution operations on the first normalized style matrix and the first normalized content matrix respectively to obtain a first style feature space and a first content feature space; Using the second style attention module, performing convolution operations on the second normalized style matrix and the second normalized content matrix respectively to obtain a second style feature space and a second content feature space; Calculating the correlation between each point in the first style feature space and each point in the first content feature space using a first expression preset in the first style attention module to obtain each element of a first attention matrix, and determining a first attention map based on the first attention matrix; The first expression preset in the first style attention module is used to calculate the correlation between each point in the second style feature space and each point in the second content feature space, to obtain each element of the second attention matrix, and to determine the second attention map based on the second attention matrix.

4. The image style transfer method based on semantic alignment as claimed in claim 3, characterized in that: The method of using the first dynamic style modulation core module and the second dynamic style modulation core module to adjust the first attention map and the second attention map respectively, and correspondingly obtaining a first adjustment map and a second adjustment map, obtaining a first embedded style feature map based on the first style feature map and the first adjustment map, and obtaining a second embedded style feature map based on the second style feature map and the second adjustment map, includes: Using a preset second expression to process the first attention matrix and the second attention matrix respectively, and correspondingly obtaining a first average matrix and a second average matrix; Determine a scale parameter λ and a bias parameter β based on the encoding module, and construct a first threshold matrix and a second threshold matrix based on the first average matrix, the second average matrix, the first attention matrix, the second attention matrix, the scale parameter λ and the bias parameter β; Determine a first dynamic style modulation kernel module and a second dynamic style modulation kernel module based on the attention matrix, the first threshold matrix, the second threshold matrix and the Sign function; Using the first dynamic style modulation kernel module and the second dynamic style modulation kernel module to process the first attention map and the second attention map respectively, and correspondingly obtain a first adjustment map and a second adjustment map; A dot product operation is performed on the first style feature map and the first adjustment map to obtain the first embedded style feature map, and a dot product operation is performed on the second style feature map and the second adjustment map to obtain the second embedded style feature map.

5. The method for image style transfer based on semantic alignment according to claim 1, characterized in that: The method comprises: performing de-stylization processing on the first content feature map and the second content feature map by using the first adaptive decoupling module and the second adaptive decoupling module respectively, obtaining a first decoupled feature map and a second decoupled feature map correspondingly, and obtaining a first fusion map according to the addition result of the first decoupled feature map and the first embedded style feature map after the convolution processing, and obtaining a second fusion map according to the addition result of the second decoupled feature map and the second embedded style feature map after the convolution processing, including: Using the first adaptive decoupling module and the second adaptive decoupling module to perform whitening transformation processing on the first content feature map and the second content feature map respectively, so as to obtain a first feature map and a second feature map respectively; Processing the first feature map using the first convolution layer of the first adaptive decoupling module to obtain a first sub-feature map and a second sub-feature map, processing the first sub-feature map and the second sub-feature map based on a third expression preset in the first adaptive decoupling module to obtain a first content space attention map, processing the first content feature map using the second convolution layer of the first adaptive decoupling module to obtain a third sub-feature map, and processing the third sub-feature map, the first content space attention map, and the first content feature map based on a fourth expression preset in the first adaptive decoupling module to obtain a first decoupled feature map; Processing the second feature map using the first convolution layer of the second adaptive decoupling module to obtain a fourth sub-feature map and a fifth sub-feature map, processing the fourth sub-feature map and the fifth sub-feature map based on a third expression preset in the second adaptive decoupling module to obtain a second content space attention map, processing the second content feature map using the second convolution layer in the second adaptive decoupling module to obtain a sixth sub-feature map, and processing the sixth sub-feature map, the second content space attention map, and the second content feature map based on a fourth expression preset in the second adaptive decoupling module to obtain a second decoupled feature map; Using the third convolutional layers of the first adaptive decoupling module and the second adaptive decoupling module to process the first embedded style feature map and the second embedded style feature map respectively, and correspondingly obtain a first scale feature map and a second scale feature map; Using the first adaptive decoupling module to perform sum processing on the first decoupled feature map and the first scale feature map to obtain the first fusion map; The second adaptive decoupling module is used to perform addition processing on the second decoupled feature map and the second scale feature map to obtain the second fusion map.

6. The method for image style transfer based on semantic alignment according to claim 1, characterized in that: The combining the first fused graph and the second fused graph to obtain a to-be-decoded graph includes: The first fused image and the second fused image are processed by using a preset fifth expression to obtain an image to be decoded, wherein the fifth expression is: Among them, conv 3×3 represents the convolution module, and There are two output feature maps representing the first fusion map and the second fusion map.

7. The image style transfer method based on semantic alignment according to claim 1, characterized in that: Also includes: The style transfer network is trained using a preset loss function to obtain a final style transfer network, wherein the expression of the loss function is: Among them, L is the total number of convolutional layers, is the content loss function of the i-th convolutional layer of the deep neural network, is the style loss function of the i-th convolutional layer of the deep neural network, and is the disentanglement loss function of the i-th convolutional layer of the deep neural network, λ c , s , dis_c , dis_s They are the weights corresponding to controlling content loss, style loss, content disentanglement loss, and style disentanglement loss respectively.

8. A method for image style transfer based on semantic alignment, characterized in that: include: Acquisition module, acquires style image and content image; A processing module, configured to process the style image and the content image using a pre-built style transfer network to obtain a stylized image, wherein the style transfer network includes an encoding module, a decoding module, a first style attention module, a second style attention module, a first adaptive decoupling module, a second adaptive decoupling module, a first dynamic style modulation kernel module, and a second dynamic style modulation kernel module; Processing the style image and the content image by using the encoding module, obtaining a first style feature map and a first content feature map in a first intermediate layer of the encoding module, and obtaining a second style feature map and a second content feature map in a second intermediate layer of the encoding module; Using the first style attention module to process the first style feature map and the first content feature map to obtain a first attention map, and using the second style attention module to process the second style feature map and the second content feature map to obtain a second attention map; Using the first dynamic style modulation kernel module and the second dynamic style modulation kernel module to adjust the first attention map and the second attention map respectively, and obtain a first adjustment map and a second adjustment map accordingly, obtain a first embedded style feature map based on the first style feature map and the first adjustment map, and obtain a second embedded style feature map based on the second style feature map and the second adjustment map; Using the first adaptive decoupling module and the second adaptive decoupling module to perform destylization processing on the first content feature map and the second content feature map respectively, and obtain a first decoupled feature map and a second decoupled feature map respectively, and obtain a first fusion map according to the addition result of the first decoupled feature map and the first embedded style feature map after the convolution processing, and obtain a second fusion map according to the addition result of the second decoupled feature map and the second embedded style feature map after the convolution processing; Combining the first fused graph and the second fused graph to obtain a to-be-decoded graph; Processing the to-be-decoded image using the decoding module to obtain the stylized image; An output module is used to output the stylized image.

9. A computer-readable storage medium, characterized in that: The method comprises instructions, which, when executed on a computer, enable the computer to execute the image style transfer method based on semantic alignment as claimed in any one of claims 1 to 7.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor, memory, and input-output unit; The memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the image style transfer method based on semantic alignment according to any one of claims 1 to 7.

Images