Image style transfer method, system, storage medium and computing device

Abstract

The present invention discloses an image style transfer method, system, storage medium and computing device. The present invention adopts an encoder to encode an image to be transferred, fuses a low-frequency feature map and a high-frequency feature map of a target style image in a feature map A output by the encoder, fuses the low-frequency feature map and the high-frequency feature map of the target style image in a feature map B output by each resolution processing layer of the encoder, introduces the feature map B after the fusion of features into a decoder, so that a generated style transfer image is not easily distorted, and performs feature enhancement on the feature map A output by the encoder, and introduces the feature map A after the feature enhancement into a decoder, so that the generated style transfer image has higher fidelity.

Description

Technical Field

[0001] The present invention relates to an image style migration method, system, storage medium and computing device, and belongs to the field of computer vision. Background Art

[0002] Image style transfer has always been one of the research focuses in the field of computer vision. The purpose of style transfer is to obtain a content image and a style reference image and mix them together so that the final output image is the content image "painted" in the style of the style reference image.

[0003] Traditional style transfer algorithms based on iterative generation of images have the problems of slow iteration speed and inability to perform real-time conversion. With the rapid development of deep learning, the generative models used for image transfer have also made great leaps. However, the current image style transfer has the problems of distortion and low fidelity in the final output image. Summary of the invention

[0004] The present invention provides an image style transfer method, system, storage medium and computing device, which solve the problems disclosed in the background technology.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] A method for image style transfer, comprising:

[0007] Perform feature separation on the target style image to obtain a low-frequency feature map and a high-frequency feature map of the target style image;

[0008] The image to be migrated is encoded by an encoder to obtain a feature map A of the image to be migrated which has been successively reduced in resolution and dimension; wherein, during the encoding process, a low-frequency feature map and a high-frequency feature map are fused in the feature map B output by each resolution processing layer of the encoder, and the fused feature map B is input into the corresponding resolution processing layer of the decoder;

[0009] Performing several layers of feature enhancement on the feature map A of the image to be transferred in sequence, and inputting each layer of enhanced feature map A into the resolution processing layer corresponding to the decoder; wherein the number of feature enhancement layers is consistent with the resolution processing layer of the decoder, and the feature enhancement layers correspond to the resolution processing layers of the decoder one by one, and the output of the previous feature enhancement layer is used as the input of the next feature enhancement layer;

[0010] The low-frequency feature map and the high-frequency feature map are fused in the feature map A of the image to be transferred, and the decoder of the fused feature map B and the feature enhanced feature map A is used to decode it to obtain the image after style transfer.

[0011] The formulas for the low-frequency feature map and high-frequency feature map of the target style image are:

[0012] F L =conv(T L ; W L→L )+conv(pool(T H ,2);W H→L )

[0013] F H =conv(T H ; W H→H )+upsample(conv(T L ; W L→H ),2)

[0014] Among them, F L is the low-frequency feature map, F H is the high-frequency feature map, conv is the convolution operation, T L is the low-frequency part of the target style image feature map, T H is the high-frequency part of the target style image feature map, W L→L is the low-frequency to low-frequency convolution kernel, W H→H is the high-frequency to high-frequency convolution kernel, W H→L is the convolution kernel from high frequency to low frequency, W L→H is the convolution kernel from low frequency to high frequency, pool(T H ,2) is to use 2*2 convolution kernel step size for average pooling, upsample(conv(T L ; W L→H ), 2) is an upsampling operation with a parameter of 2 through nearest neighbor interpolation.

[0015] The encoder includes several resolution processing layers connected in series, and several dimension processing layers are connected in series downstream of the resolution processing layer string. The decoder includes several resolution processing layers connected in series, and the number of decoder resolution processing layers is consistent with the number of encoder resolution processing layers, and resolution processing layers of the same resolution correspond to each other.

[0016] The fused feature map B is transmitted to the decoder through skip connections.

[0017] For the feature map A of the image to be migrated, the low-frequency feature map and the high-frequency feature map are fused once; along the output to input direction of the encoder, for the feature map B output by the nth resolution processing layer, the low-frequency feature map and the n+1th level high-frequency feature map are fused once;

[0018] The formula for fusing a low-frequency feature map and n+1 layers of high-frequency feature maps is:

[0019]

[0020] Among them, i∈[0,n], is the feature map after fusing 1 low-frequency feature map and i high-frequency feature map. It is the feature map after fusing the 1st low-frequency feature map and the i+1th high-frequency feature map. is the original feature map without fusion of low-frequency feature map and high-frequency feature map, inject(·) is the injection operation, represents element-by-element multiplication, σ is the sigmoid function, for After convolution, the feature map for The feature map after injecting i+1 times of high-frequency feature map, conv is the convolution operation, F H is the high-frequency feature map, for The feature map after injecting i high-frequency feature maps, is the low-frequency feature map after convolution, for Sequentially with The feature map after element-wise multiplication of the 1-sigmoid operation result, for Sequentially with The feature map after element-wise multiplication of the 1-sigmoid operation results.

[0021] An image style transfer system, comprising:

[0022] A separation module performs feature separation on the target style image to obtain a low-frequency feature map and a high-frequency feature map of the target style image;

[0023] The encoding module uses an encoder to encode the image to be migrated to obtain a feature map A of the image to be migrated that has undergone resolution reduction and dimension reduction in sequence; wherein, during the encoding process, the low-frequency feature map and the high-frequency feature map are fused in the feature map B output by each resolution processing layer of the encoder, and the fused feature map B is input into the corresponding resolution processing layer of the decoder;

[0024] The feature enhancement module sequentially performs several layers of feature enhancement on the feature map A of the image to be transferred, and inputs the feature map A after each layer of enhancement into the resolution processing layer corresponding to the decoder; wherein the number of feature enhancement layers is consistent with the resolution processing layer of the decoder, and the feature enhancement layers correspond to the resolution processing layers of the decoder one by one, and the output of the previous feature enhancement layer is used as the input of the next feature enhancement layer;

[0025] The decoding module fuses the low-frequency feature map and the high-frequency feature map in the feature map A of the image to be transferred, and decodes it using the decoder of the input fused feature map B and the feature enhanced feature map A to obtain the image after style transfer.

[0026] In the separation module, the formulas for the low-frequency feature map and high-frequency feature map of the target style image are:

[0027] F L =conv(T L ; W L→L )+conv(pool(T H ,2);W H→L )

[0028] F H =conv(T H ; W H→H )+upsample(conv(T L ; W L→H ),2)

[0029] Among them, F L is the low-frequency feature map, F H is the high-frequency feature map, conv is the convolution operation, T L is the low-frequency part of the target style image feature map, T H is the high-frequency part of the target style image feature map, W L→L is the low-frequency to low-frequency convolution kernel, W H→H is the high-frequency to high-frequency convolution kernel, W H→L is the convolution kernel from high frequency to low frequency, W L→H is the convolution kernel from low frequency to high frequency, pool(T H ,2) is to use 2*2 convolution kernel step size for average pooling, upsample(conv(T L ; W L→H ), 2) is an upsampling operation with a parameter of 2 through nearest neighbor interpolation.

[0030] For the feature map A of the image to be migrated, the low-frequency feature map and the high-frequency feature map are fused once; along the output to input direction of the encoder, for the feature map B output by the nth resolution processing layer, the low-frequency feature map and the n+1th level high-frequency feature map are fused once;

[0031] The formula for fusing a low-frequency feature map and n+1 layers of high-frequency feature maps is:

[0032]

[0033] Among them, i∈[0,n], is the feature map after fusing 1 low-frequency feature map and i high-frequency feature map. It is the feature map after fusing the 1st low-frequency feature map and the i+1th high-frequency feature map. is the original feature map without fusion of low-frequency feature map and high-frequency feature map, inject(·) is the injection operation, represents element-by-element multiplication, σ is the sigmoid function, for After convolution, the feature map for The feature map after injecting i+1 times of high-frequency feature map, conv is the convolution operation, F H is the high-frequency feature map, for The feature map after injecting i high-frequency feature maps, is the low-frequency feature map after convolution, for Sequentially with The feature map after element-wise multiplication of the 1-sigmoid operation result, for Sequentially with The feature map after element-wise multiplication of the 1-sigmoid operation results.

[0034] A computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform an image style transfer method.

[0035] A computing device includes one or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing an image style transfer method.

[0036] The beneficial effects achieved by the present invention are as follows: the present invention uses an encoder to encode the image to be transferred, fuses the low-frequency feature map and the high-frequency feature map of the target style image in the feature map A output by the encoder, fuses the low-frequency feature map and the high-frequency feature map of the target style image in the feature map B output by each resolution processing layer of the encoder, introduces the feature map B after the fusion of features into the decoder, so that the generated style transfer image is not easily distorted, and performs feature enhancement on the feature map A output by the encoder, and introduces the feature map A after the feature enhancement into the decoder, so that the generated style transfer image has higher fidelity. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Flowchart of the image style transfer method;

[0038] Figure 2 This is the structural diagram of the image style transfer network;

[0039] Figure 3 This is a framework diagram for feature separation;

[0040] Figure 4 A framework diagram for fusing low-frequency feature maps and high-frequency feature maps. DETAILED DESCRIPTION

[0041] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0042] like Figure 1 As shown, an image style transfer method includes the following steps:

[0043] Step 1: perform feature separation on the target style image to obtain a low-frequency feature map and a high-frequency feature map of the target style image;

[0044] Step 2, using an encoder to encode the image to be migrated to obtain a feature map A of the image to be migrated that has been successively reduced in resolution and dimension; wherein, during the encoding process, a low-frequency feature map and a high-frequency feature map are fused in the feature map B output by each resolution processing layer of the encoder, and the fused feature map B is input into the corresponding resolution processing layer of the decoder;

[0045] Step 3, performing several layers of feature enhancement on the feature map A of the image to be migrated in sequence, and inputting each layer of enhanced feature map A into the resolution processing layer corresponding to the decoder; wherein the number of feature enhancement layers is consistent with the resolution processing layer of the decoder, and the feature enhancement layers correspond to the resolution processing layers of the decoder one by one, and the output of the previous feature enhancement layer is used as the input of the next feature enhancement layer;

[0046] Step 4: Fuse the low-frequency feature map and the high-frequency feature map in the feature map A of the image to be transferred, and decode it using the decoder of the input fused feature map B and the feature enhanced feature map A to obtain the image after style transfer.

[0047] The above method uses an encoder to encode the image to be transferred, fuses the low-frequency feature map and the high-frequency feature map of the target style image in the feature map A output by the encoder, fuses the low-frequency feature map and the high-frequency feature map of the target style image in the feature map B output by each resolution processing layer of the encoder, introduces the feature map B after the fusion of features into the decoder, so that the generated style transfer image is not easily distorted, and performs feature enhancement on the feature map A output by the encoder, and introduces the feature map A after the feature enhancement into the decoder, so that the generated style transfer image has higher fidelity.

[0048] Based on the above method, the corresponding network can be constructed, such as Figure 2 As shown, it mainly includes an encoder and a decoder.

[0049] Figure 2 In the process, the target style image is feature separated to obtain a low-frequency feature map and a high-frequency feature map of the target style image; in the feature separation, the feature map of the target style image is segmented into a feature map containing low-frequency information and a feature map containing high-frequency information, namely, a low-frequency feature map and a high-frequency feature map through octave convolution; the low-frequency feature map can be simply regarded as containing the color and contour features of the target style image, and the high-frequency feature map can be simply regarded as containing the line and texture features of the target style image.

[0050] The segmentation process is as follows Figure 3 As shown, the formulas for the low-frequency feature map and high-frequency feature map of the target style image can be expressed as:

[0051] F L =conv(T L ; W L→L )+conv(pool(T H ,2);W H→L )

[0052] F H =conv(T H ; W H→H )+upsample(conv(T L ; W L→H ),2)

[0053] Among them, F L is the low-frequency feature map, F H is the high-frequency feature map, conv is the convolution operation, T L is the low-frequency part of the target style image feature map, T H is the high-frequency part of the target style image feature map, W L→L is the low-frequency to low-frequency convolution kernel, W H→H is the high-frequency to high-frequency convolution kernel, W H→L is the convolution kernel from high frequency to low frequency, W L→H is the convolution kernel from low frequency to high frequency, pool(T H ,2) is to use 2*2 convolution kernel step size for average pooling, upsample(conv(T L ; W L→H ), 2) is an upsampling operation with a parameter of 2 through nearest neighbor interpolation.

[0054] The encoder includes several resolution processing layers connected in series. Several dimension processing layers are connected in series downstream of the resolution processing layer string. The figure shows 5 resolution processing layers, each of which performs two convolutions and one maximum pooling, and 2 dimension processing layers.

[0055] After the image to be migrated has been processed through 5 layers of resolution, the original 1024 2 The resolution of the feature map becomes 64 2 The feature map of resolution, each dimension processing layer is 64 2 The encoder uses 1×1 convolution to reduce the dimension of the feature map without losing resolution, thus reducing the number of network calculation parameters. 1×1 convolution can also increase the nonlinear characteristics of the network and deepen the network depth.

[0056] The decoder includes several resolution processing layers connected in series. The number of resolution processing layers of the decoder is consistent with the number of resolution processing layers of the encoder, and the resolution processing layers of the same resolution correspond to each other, that is, 1024 in the encoder 2 The maximum pooling layer at the resolution level and 1024 in the decoder 2 The upsampling layers of the resolution level are connected, and the 512 2 The maximum pooling layer at the resolution level and the 512 in the decoder processing layer 2 The upsampling layers of the resolution level are connected, and so on, until the 64 2 The maximum pooling layer at the resolution level and the 64 in the decoder processing layer 2 The upsampling layers of the resolution level are connected.

[0057] By using the above encoder to encode the image to be transferred, a feature map A of the image to be transferred that has been successively reduced in resolution and dimension can be obtained; during the encoding process, the low-frequency feature map and the high-frequency feature map are fused in the feature map B output by each resolution processing layer of the encoder, and the fused feature map B is input into the corresponding resolution processing layer of the decoder through a skip connection. The skip connection not only reduces the possibility of vanishing gradients in the back propagation, but also enables better reconstruction of the details of the style transfer image at the decoder end when the network is very deep.

[0058] The feature map A of the image to be transferred is subjected to several layers of feature enhancement in sequence, and each layer of enhanced feature map A is input into the resolution processing layer corresponding to the decoder; the number of feature enhancement layers is consistent with the resolution processing layer of the decoder, and the feature enhancement layer corresponds to the resolution processing layer of the decoder one by one, and the output of the previous feature enhancement layer is used as the input of the next feature enhancement layer. Feature enhancement is to gradually extract the detail features contained in the feature map A through convolution and upsampling operations to obtain enhanced features. The feature maps extracted in each layer are added to the corresponding decoder resolution processing layer, thereby enhancing the texture details of the final generated image.

[0059] The low-frequency feature map and the high-frequency feature map are fused in the feature map A of the image to be transferred, and the decoder of the fused feature map B and the feature enhanced feature map A is used to decode it to obtain the image after style transfer. The image after style transfer is based on the image to be transferred and contains the style of the target style image.

[0060] The principle of fusing low-frequency feature maps and high-frequency feature maps is as follows Figure 4 As shown, for the feature map A of the image to be migrated, the low-frequency feature map and the high-frequency feature map are fused once; along the output to input direction of the encoder, for the feature map B output by the nth resolution processing layer, the low-frequency feature map and the n+1-level high-frequency feature map are fused once.

[0061] Figure 4 In order to fuse the principle of low-frequency feature maps and high-frequency feature maps, there are two fusion branches in the figure. The high-frequency features enter the high-frequency feature aggregation branch through the injection operation, and the low-frequency features enter the low-frequency feature aggregation branch.

[0062] Assume that the input content feature F in =R H×W×C , H stands for height, W stands for width, C stands for Channel, F stands for inject Represents injection F in The information in the injection operation is represented by inject(·), and the characteristics after injection inject(·) can be expressed as:

[0063]

[0064] in, Represents the three channels of input, γ x,y,z (F inject ), β x,y,z (F inject ) are two variables obtained by convolution, ε is infinitely close to 0 but not equal to 0.

[0065] Through the injection operation, high-frequency features and low-frequency features are mapped to the feature space where the content features are located. The high-frequency features are used to guide the image to be transferred to learn detailed features such as lines and textures from the target style image, and the low-frequency features are used to guide the image to be transferred to learn overall features such as color and contours from the target style image.

[0066] Figure 4 The fusion process can be expressed as follows:

[0067] The formula for fusing a low-frequency feature map and n+1 layers of high-frequency feature maps is:

[0068]

[0069] Among them, i∈[0,n], is the feature map after fusing 1 low-frequency feature map and i high-frequency feature map. It is the feature map after fusing the 1st low-frequency feature map and the i+1th high-frequency feature map. is the original feature map without fusion of low-frequency feature map and high-frequency feature map, inject(·) is the injection operation, represents element-by-element multiplication, σ is the sigmoid function, for After convolution, the feature map for The feature map after injecting i+1 times of high-frequency feature map, conv is the convolution operation, F H is the high-frequency feature map, for The feature map after injecting i high-frequency feature maps, is the low-frequency feature map after convolution, for Sequentially with The feature map after element-wise multiplication of the 1-sigmoid operation result, for Sequentially with The feature map after element-wise multiplication of the 1-sigmoid operation results.

[0070] In the feature reconstruction process, the present invention adopts a feature injection method to inject the high-frequency and low-frequency features of the target style image into the skip connection, and uses the high-frequency features as a reference for the low-frequency features, so that the high-frequency features can guide the injection of the low-frequency features. Compared with other methods, the final generated style transfer image is not prone to distortion, and enhanced features are introduced at the decoder end to enhance the texture details of the reconstructed image, so that the final generated style transfer image has higher fidelity.

[0071] Based on the same technical solution, the present invention also discloses a software system of the above method, an image style transfer system, comprising:

[0072] The separation module performs feature separation on the target style image to obtain a low-frequency feature map and a high-frequency feature map of the target style image.

[0073] In the separation module, the formulas for the low-frequency feature map and high-frequency feature map of the target style image are:

[0074] F L =conv(T L ; W L→L)+conv(pool(T H ,2);W H→L )

[0075] F H =conv(T H ; W H→H )+upsample(conv(T L ; W L→H ),2)

[0076] Among them, F L is the low-frequency feature map, F H is the high-frequency feature map, conv is the convolution operation, T L is the low-frequency part of the target style image feature map, T H is the high-frequency part of the target style image feature map, W L→L is the low-frequency to low-frequency convolution kernel, W H→H is the high-frequency to high-frequency convolution kernel, W H→L is the convolution kernel from high frequency to low frequency, W L→H is the convolution kernel from low frequency to high frequency, pool(T H ,2) is to use 2*2 convolution kernel step size for average pooling, upsample(conv(T L ; W L→H ), 2) is an upsampling operation with a parameter of 2 through nearest neighbor interpolation.

[0077] The encoding module uses an encoder to encode the image to be migrated to obtain a feature map A of the image to be migrated that has undergone resolution reduction and dimension reduction in sequence; wherein, during the encoding process, the low-frequency feature map and the high-frequency feature map are fused in the feature map B output by each resolution processing layer of the encoder, and the fused feature map B is input into the corresponding resolution processing layer of the decoder.

[0078] The feature enhancement module performs several layers of feature enhancement on the feature map A of the image to be migrated in sequence, and inputs the feature map A after each layer of enhancement into the resolution processing layer corresponding to the decoder; wherein, the number of feature enhancement layers is consistent with the resolution processing layer of the decoder, and the feature enhancement layers correspond to the resolution processing layers of the decoder one by one, and the output of the previous feature enhancement layer serves as the input of the next feature enhancement layer.

[0079] The decoding module fuses the low-frequency feature map and the high-frequency feature map in the feature map A of the image to be transferred, and decodes it using the decoder of the input fused feature map B and the feature enhanced feature map A to obtain the image after style transfer.

[0080] For the feature map A of the image to be migrated, the low-frequency feature map and the high-frequency feature map are fused once; along the output to input direction of the encoder, for the feature map B output by the nth resolution processing layer, the low-frequency feature map and the n+1th level high-frequency feature map are fused once;

[0081] The formula for fusing a low-frequency feature map and n+1 layers of high-frequency feature maps is:

[0082]

[0083] Among them, i∈[0,n], is the feature map after fusing 1 low-frequency feature map and i high-frequency feature map. It is the feature map after fusing the 1st low-frequency feature map and the i+1th high-frequency feature map. is the original feature map without fusion of low-frequency feature map and high-frequency feature map, inject(·) is the injection operation, represents element-by-element multiplication, σ is the sigmoid function, for After convolution, the feature map for The feature map after injecting i+1 times of high-frequency feature map, conv is the convolution operation, F H is the high-frequency feature map, for The feature map after injecting i high-frequency feature maps, is the low-frequency feature map after convolution, for Sequentially with The feature map after element-wise multiplication of the 1-sigmoid operation result, for Sequentially with The feature map after element-wise multiplication of the 1-sigmoid operation results.

[0084] Based on the same technical solution, the present invention also discloses a computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by a computing device, the computing device performs an image style transfer method.

[0085] Based on the same technical solution, the present invention also discloses a computing device, comprising one or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing an image style transfer method.

[0086] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0087] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0088] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0089] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0090] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are included in the scope of the claims of the present invention to be approved.

Claims

1. A method for image style transfer, characterized in that: include: Perform feature separation on the target style image to obtain a low-frequency feature map and a high-frequency feature map of the target style image; The image to be migrated is encoded by an encoder to obtain a feature map A of the image to be migrated which has been successively reduced in resolution and dimension; wherein, during the encoding process, a low-frequency feature map and a high-frequency feature map are fused in the feature map B output by each resolution processing layer of the encoder, and the fused feature map B is input into the corresponding resolution processing layer of the decoder; Performing several layers of feature enhancement on the feature map A of the image to be transferred in sequence, and inputting each layer of enhanced feature map A into the resolution processing layer corresponding to the decoder; wherein the number of feature enhancement layers is consistent with the resolution processing layer of the decoder, and the feature enhancement layers correspond to the resolution processing layers of the decoder one by one, and the output of the previous feature enhancement layer is used as the input of the next feature enhancement layer; The low-frequency feature map and the high-frequency feature map are fused in the feature map A of the image to be transferred, and the decoder of the fused feature map B and the feature enhanced feature map A is used to decode it to obtain the image after style transfer.

2. The image style transfer method according to claim 1, characterized in that: The formulas for the low-frequency feature map and high-frequency feature map of the target style image are: F L =conv(T L ;W L→L )+conv(pool(T H ,2);W H→L ) F H =conv(T H ;W H→H )+upsample(conv(T L ;W L→H ),2) Among them, F L is the low-frequency feature map, F H is the high-frequency feature map, conv is the convolution operation, T L is the low-frequency part of the target style image feature map, T H is the high-frequency part of the target style image feature map, W L→L is the low-frequency to low-frequency convolution kernel, W H→H is the high-frequency to high-frequency convolution kernel, W H→L is the convolution kernel from high frequency to low frequency, W L→H is the convolution kernel from low frequency to high frequency, pool(T H ,2) is to use 2*2 convolution kernel step size for average pooling, upsample(conv(T L ; W L→H ), 2) is an upsampling operation with a parameter of 2 through nearest neighbor interpolation.

3. The image style transfer method according to claim 1, characterized in that: The encoder includes several resolution processing layers connected in series, and several dimension processing layers are connected in series downstream of the resolution processing layer string. The decoder includes several resolution processing layers connected in series, and the number of decoder resolution processing layers is consistent with the number of encoder resolution processing layers, and resolution processing layers of the same resolution correspond to each other.

4. The image style transfer method according to claim 1, characterized in that: The fused feature map B is transmitted to the decoder through skip connections.

5. The image style transfer method according to claim 1, characterized in that: For the feature map A of the image to be migrated, the low-frequency feature map and the high-frequency feature map are fused once; along the output to input direction of the encoder, for the feature map B output by the nth resolution processing layer, the low-frequency feature map and the n+1th level high-frequency feature map are fused once; The formula for fusing a low-frequency feature map and n+1 layers of high-frequency feature maps is: Among them, i∈[0,n], is the feature map after fusing 1 low-frequency feature map and i high-frequency feature map. It is the feature map after fusing the 1st low-frequency feature map and the i+1th high-frequency feature map. is the original feature map without fusion of low-frequency feature map and high-frequency feature map, inject(·) is the injection operation, represents element-by-element multiplication, σ is the sigmoid function, for After convolution, the feature map for The feature map after injecting i+1 times of high-frequency feature map, conv is the convolution operation, F H is the high frequency feature map, for The feature map after injecting i high-frequency feature maps, is the low-frequency feature map after convolution, for Sequentially with The feature map after element-wise multiplication of the 1-sigmoid operation result, for Sequentially with The feature map after element-wise multiplication of the 1-sigmoid operation results.

6. An image style transfer system, characterized in that: include: A separation module performs feature separation on the target style image to obtain a low-frequency feature map and a high-frequency feature map of the target style image; The encoding module uses an encoder to encode the image to be migrated to obtain a feature map A of the image to be migrated that has undergone resolution reduction and dimension reduction in sequence; wherein, during the encoding process, the low-frequency feature map and the high-frequency feature map are fused in the feature map B output by each resolution processing layer of the encoder, and the fused feature map B is input into the corresponding resolution processing layer of the decoder; The feature enhancement module sequentially performs several layers of feature enhancement on the feature map A of the image to be transferred, and inputs the feature map A after each layer of enhancement into the resolution processing layer corresponding to the decoder; wherein the number of feature enhancement layers is consistent with the resolution processing layer of the decoder, and the feature enhancement layers correspond to the resolution processing layers of the decoder one by one, and the output of the previous feature enhancement layer is used as the input of the next feature enhancement layer; The decoding module fuses the low-frequency feature map and the high-frequency feature map in the feature map A of the image to be transferred, and decodes it using the decoder of the input fused feature map B and the feature enhanced feature map A to obtain the image after style transfer.

7. The image style transfer system according to claim 6, characterized in that: In the separation module, the formulas for the low-frequency feature map and high-frequency feature map of the target style image are: F L =conv(T L ;W L→L )+conv(pool(T H ,2);W H→L ) F H =conv(T H ;W H→H )+upsample(conv(T L ;W L→H ),2) Among them, F L is the low-frequency feature map, F H is the high-frequency feature map, conv is the convolution operation, T L is the low-frequency part of the target style image feature map, T H is the high-frequency part of the target style image feature map, W L→L is the low-frequency to low-frequency convolution kernel, W H→H is the high-frequency to high-frequency convolution kernel, W H→L is the convolution kernel from high frequency to low frequency, W L→H is the convolution kernel from low frequency to high frequency, pool(T H ,2) is to use 2*2 convolution kernel step size for average pooling, upsample(conv(T L ; W L→H ), 2) is an upsampling operation with a parameter of 2 through nearest neighbor interpolation.

8. The image style transfer system according to claim 6, characterized in that: For the feature map A of the image to be migrated, the low-frequency feature map and the high-frequency feature map are fused once; along the output to input direction of the encoder, for the feature map B output by the nth resolution processing layer, the low-frequency feature map and the n+1th level high-frequency feature map are fused once; The formula for fusing a low-frequency feature map and n+1 layers of high-frequency feature maps is: Among them, i∈[0,n], is the feature map after fusing 1 low-frequency feature map and i high-frequency feature map. It is the feature map after fusing the 1st low-frequency feature map and the i+1th high-frequency feature map. is the original feature map without fusion of low-frequency feature map and high-frequency feature map, inject(·) is the injection operation, represents element-by-element multiplication, σ is the sigmoid function, for After convolution, the feature map for The feature map after injecting i+1 times of high-frequency feature map, conv is the convolution operation, F H is the high frequency feature map, for The feature map after injecting i high-frequency feature maps, is the low-frequency feature map after convolution, for Sequentially with The feature map after element-wise multiplication of the 1-sigmoid operation result, for Sequentially with The feature map after element-wise multiplication of the 1-sigmoid operation results.

9. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions which, when executed by a computing device, cause the computing device to perform any one of the methods according to claims 1 to 5 .

10. A computing device, characterized in that: include: One or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the methods according to claims 1 to 5.

Images