A multi-attention fusion image style transfer method, device and equipment

Abstract

The present invention relates to a method, apparatus, and device for image style transfer using multi-attention fusion. The method comprises: obtaining image data to be processed; performing multi-scale feature extraction on the content image to be processed and the style image to obtain content features and style features; fusing the content features and style features using a multi-attention fusion module to obtain attention features; and decoding the attention features to obtain a style-transferred image. This method fuses the content features and style features using a multi-attention fusion module to obtain attention features, thereby focusing on and emphasizing salient regional information in the content image and style image to be processed, suppressing irrelevant background details, and enhancing the stylized processing effect of style transfer on key image regions. This establishes attention features that connect global regions, effectively controlling the local regions of style transfer, and ensuring clear and complete edge texture transmission of the style-transferred content image.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multi-attention fusion image style transfer method, device and equipment. Background Art

[0002] Style transfer is a powerful technique for artistic creation and image editing that can transfer the style of one image to another; however, arbitrary style transfer trades flexibility for the ability to model local style patterns in arbitrary style transfer networks.

[0003] Existing style transfer techniques include three approaches. The first is neural style transfer, inspired by the use of convolutional neural networks (CNNs) in visual perception tasks. This approach first introduces a deep neural network to separate and reconstruct the content and style of any image. It then proposes extracting content features and style correlations (Gram matrices) along a processing hierarchy of pre-trained network classifiers to achieve style transfer. However, this approach requires a style map and content map as input, making it extremely inefficient. The second approach transfers the global mean and variance of the style image to the feature space of the content image to support images of any input style. However, this approach calculates the mean and variance of the features globally, largely ignoring local details and thus significantly reducing the local stylization effect. The third processing method takes into account both the shallow and deep features of the image in the calculation of the attention score during image processing, and properly normalizes the content features so that the feature statistics are well aligned with the attention-weighted mean and variance map of the style features at each point, thereby improving the local style effect of any style transfer model and achieving a better balance between style pattern transmission and content structure preservation. However, the disadvantage of this processing method is that it does not consider the three-dimensional surface structure of panoramic images, so the effect of style transfer in panoramic images is not satisfactory.

[0004] Therefore, the style transfer technology for image processing does not take into account the complex spatial structural characteristics of the image panorama. In the style transfer of the image, the texture transmission is not clear and complete, and there is a problem of local distortion. Summary of the Invention

[0005] The embodiments of the present invention provide a multi-attention fusion image style transfer method, device and equipment for solving the technical problems of unclear and distorted textures in existing style transfer technologies for image processing.

[0006] In order to achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0007] A multi-attention fusion image style transfer method includes the following steps:

[0008] Acquire image data to be processed, wherein the image data includes a content image to be processed and a style image;

[0009] Performing multi-scale feature extraction on the content image to be processed and the style image, respectively, to obtain content features and style features corresponding to the content image to be processed and the style image;

[0010] Using a multi-attention fusion module to fuse the content features and the style features to obtain attention features;

[0011] The attention feature is decoded to obtain a style transfer image corresponding to the content image to be processed.

[0012] Preferably, the multi-attention fusion image style migration method includes: performing feature extraction on the style migration image to obtain migration features corresponding to the style migration image; using a loss function to compare the migration features with the content features and the style features respectively to obtain a migration loss; adjusting the migration direction and output result of the style migration image according to the migration loss to obtain a panoramic clear image after style migration corresponding to the content image to be processed.

[0013] Preferably, the loss function is:

[0014] L=λ g L gs +λ l L lf

[0015]

[0016] Where L is the migration loss, λ g and λ l are all hyperparameters of the loss term weight, L gs is the penalty parameter for the distance and standard deviation between the content image to be processed and the style image, L lf is the feature mapping parameter of the style migration image, E() is the feature of the VGG encoder, x is the x-th layer index of the AdaAttN module, I cs is the style transfer image, F c is the content feature, F s is the style feature, μ is the average distance between the content image to be processed and the style image, σ is the standard deviation of the distance between the content image to be processed and the style image, and AdaAttN() is the AdaAttN function of the AdaAttN module.

[0017] Preferably, the content features and the style features are recorded as feature data, and a multi-attention fusion module is used to fuse the content features and the style features to obtain the attention features, which includes:

[0018] Using the channel attention submodule of the multi-attention fusion module to serially process the input feature data and obtain first processed data by upsampling the processed data, and adding and fusing the first processed data with the feature data to obtain second processed data;

[0019] Using the coordinate attention submodule of the multi-attention fusion module to parallel process the second processed data to obtain third processed data corresponding to the second processed data, and adding and fusing the second processed data and the third processed data to obtain fourth processed data;

[0020] Using the channel attention submodule of the multi-attention fusion module to serially process the input feature data to obtain fifth processed data; and adding and fusing the fifth processed data with the fourth processed data to obtain sixth processed data;

[0021] The sixth processed data is output through the coordinate attention submodule of the multi-attention fusion module to obtain attention features.

[0022] Preferably, the sixth processed data is output through the coordinate attention submodule of the multi-attention fusion module, and the attention feature is obtained, including: using the mirror filling, coordinate attention submodule, convolution and coordinate attention submodule of the multi-attention fusion module to process the sixth processed data in sequence and output the attention feature.

[0023] The present invention also provides an image style transfer device with multi-attention fusion, comprising a data acquisition module, a first VGG encoder, a multi-attention fusion module, a VGG decoder, and a second VGG encoder;

[0024] The data acquisition module is used to acquire image data to be processed, wherein the image data includes a content image and a style image to be processed;

[0025] The first VGG encoder is configured to perform multi-scale feature extraction on the content image to be processed and the style image, respectively, to obtain content features and style features corresponding to the content image to be processed and the style image;

[0026] The multi-attention fusion module is used to fuse the content features and the style features to obtain attention features;

[0027] The VGG decoder is used to decode the attention feature to obtain a style transfer image corresponding to the content image to be processed;

[0028] The second VGG encoder is used to perform feature extraction on the style-transferred image to obtain a transfer feature corresponding to the style-transferred image; use a loss function to compare the transfer feature with the content feature and the style feature respectively to obtain a transfer loss; and adjust the transfer direction and output result of the style-transferred image according to the transfer loss to obtain a panoramic clear image after style transfer corresponding to the content image to be processed.

[0029] Preferably, the loss function is:

[0030] L=λ g L gs +λ l L lf

[0031]

[0032] Where L is the migration loss, λ g and λ l are all hyperparameters of the loss term weight, L gs is the penalty parameter for the distance and standard deviation between the content image to be processed and the style image, L lf is the feature mapping parameter of the style migration image, E() is the feature of the VGG encoder, x is the x-th layer index of the AdaAttN module, I cs is the style transfer image, F c is the content feature, F s is the style feature, μ is the average distance between the content image to be processed and the style image, σ is the standard deviation of the distance between the content image to be processed and the style image, and AdaAttN() is the AdaAttN function of the AdaAttN module.

[0033] Preferably, the VGG decoder is a decoder of a symmetrical structure VGG-19 network.

[0034] Preferably, the multi-attention fusion module includes a plurality of channel attention sub-modules and coordinate attention sub-modules connected in series and parallel.

[0035] The present invention also provides a terminal device, comprising a processor and a memory;

[0036] The memory is used to store program code and transmit the program code to the processor;

[0037] The processor is used to execute the above-mentioned multi-attention fusion image style transfer method according to the instructions in the program code.

[0038] As can be seen from the above technical solutions, the embodiments of the present invention have the following advantages: the multi-attention fusion image style transfer method, apparatus, and device include: obtaining image data to be processed, the image data including a content image to be processed and a style image to be processed; performing multi-scale feature extraction on the content image to be processed and the style image to be processed, respectively, to obtain content features and style features corresponding to the content image to be processed and the style image to be processed; fusing the content features and style features using a multi-attention fusion module to obtain attention features; and decoding the attention features to obtain a style-transferred image corresponding to the content image to be processed. The multi-attention fusion image style transfer method fuses the content features and style features using a multi-attention fusion module to obtain attention features, thereby focusing on and emphasizing salient regional information in the content image to be processed and the style image to be processed, suppressing irrelevant background details, and enhancing the stylized processing effect of style transfer on key image regions, thereby establishing attention features that connect global regions, effectively controlling the local regions of the style transfer, and ensuring clear and complete edge texture transmission of the content image to be processed after style transfer, thereby solving the technical problems of unclear and distorted textures in existing image processing style transfer technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 This is a flowchart of the steps of the multi-attention fusion image style transfer method according to an embodiment of the present invention;

[0041] Figure 2 This is an architecture diagram of a multi-attention fusion module in the multi-attention fusion image style transfer method according to an embodiment of the present invention;

[0042] Figure 3 This is a framework diagram of the multi-attention fusion image style transfer device described in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, features, and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0044] The embodiment of the present application provides a multi-attention fusion image style transfer method, device and equipment based on the existing style transfer AdaAttN model. By extracting features of the content image to be processed and the style image and fusing the extracted content features and style features, a more reasonable regional attention and weight distribution style transfer image is obtained, so that the content image to be processed can effectively maintain the original significant visual features in the stylization process. Not only can the style texture of the style transfer image be better rendered into each area of the panoramic image corresponding to the content image to be processed, but the edge structure of the content image to be processed can also be more clearly and naturally preserved. The generated style transfer image can also better present and emphasize the original content theme and artistic emotion, which is used to solve the technical problems of unclear and distorted textures in existing style transfer technologies for image processing.

[0045] Example 1:

[0046] Figure 1 This is a flowchart of the steps of the multi-attention fusion image style transfer method according to an embodiment of the present invention. Figure 2 This is an architectural diagram of the multi-attention fusion module in the multi-attention fusion image style transfer method described in an embodiment of the present invention.

[0047] like Figure 1 As shown, an embodiment of the present invention provides an image style transfer method using multi-attention fusion, comprising the following steps:

[0048] S1. Obtain image data to be processed, where the image data includes a content image and a style image to be processed.

[0049] It should be noted that, in step S1, the content image to be processed and the style image of the transfer target are obtained. In this embodiment, the content image to be processed is a panoramic image.

[0050] S2. Perform multi-scale feature extraction on the content image and style image to be processed, respectively, to obtain content features and style features corresponding to the content image and style image to be processed.

[0051] It should be noted that in step S2, the encoder of the VGG-19 pre-trained network can be used to perform multi-scale feature extraction on the content image and style image to be processed, respectively, to obtain multi-scale content features and style features.

[0052] S3. Use the multi-attention fusion module to fuse the content features and style features to obtain attention features.

[0053] It should be noted that in step S3, a multi-attention fusion module (SC-Attention) is used to fuse the content features and style features to obtain attention features that focus on the spatial structure of different regions of the content image to be processed, so as to achieve effective style transfer processing of the content image to be processed. In this embodiment, the attention fusion module is set on the basis of the AdaAttN module. The attention fusion module is used to further focus on the structure of the input content image to be processed. The attention fusion module is mainly composed of multiple SE modules and CA modules in parallel and serial manners, such as Figure 2 As shown. This multi-attention fusion image style transfer method obtains attention features by fusing content features and style features through a multi-attention fusion module. This method focuses on and emphasizes the salient regional information of the content image and style image to be processed, suppresses irrelevant background details, and can enhance the stylized processing effect of style transfer on key areas of the image. It avoids the existing problem of directly using the AdaAttN model to focus on shallow and deep features of the image, lacking channel features and local edge structure features of the image, resulting in image distortion and unclear texture after style transfer. Among them, the AdaAttN module is a relatively mature neural network processing image style transfer module, and will not be elaborated on in detail here.

[0054] In the embodiment of the present invention, Figure 2 As shown in Figure 1, the attention fusion module SC-Attention is formed by fusing two attention submodules: the channel attention submodule SE and the coordinate attention submodule CA. The attention fusion module and the AdaAttN module together form a multi-attention fusion module. This enables the multi-attention module SC-Attention to process low-level and high-level features with attention mechanisms, adaptively shifting feature distributions and roughly maintaining the original salient visual features of the content image being processed during the stylization process. The data output by the multi-attention fusion module is represented by the first formula, which is:

[0055] f c (x) = y c (pad(y c (conv(z c (AdaAttN c (x))+yc (z c (AdaAttN c (x)))))

[0056] Where x is the data of the input module, f c (x) is the output data of the cth channel, pad() is the output data of the mirror padding layer, conv() is the output data of the convolution layer, z c () is the data output by the channel attention submodule at the cth channel, y c () is the data output by the coordinate attention submodule in the cth channel, AdaAttN c () is the data output by AdaAttN attention in the cth channel.

[0057] S4. Decode the attention features to obtain a style transfer image corresponding to the content image to be processed.

[0058] It should be noted that in step S4, the decoder of the symmetrical structure VGG-19 network is used to decode the attention features, and the decoder outputs the style transfer image.

[0059] In an embodiment of the present invention, the multi-attention fusion image style transfer method distributes weights to various parts of the content image to be processed through a multi-attention fusion module composed of a channel attention submodule, a coordinate attention submodule and an AdaAttN module of the original model to establish a connection between the global regions, thereby effectively controlling the local areas of style transfer, so that the edge texture transmission of the content image to be processed after style transfer is clear and complete.

[0060] It should be noted that the multi-attention fusion image style transfer method can clearly reproduce the key areas of the content image to be processed through the multi-attention fusion module, and the key areas of the content image to be processed can be clearly reproduced without various strange distortions. The multi-attention fusion image style transfer method is significantly better than the method described in the background technology in the effect of style transfer on the content image to be processed, which is reflected in the sufficient preservation of the content texture and the sufficient clarity and rationality of the style color texture rendering. The multi-attention fusion image style transfer method can make the areas of important objects on the content image to be processed that are focused on during the style transfer process richer and more complete. Compared with the method described in the background technology, the salient areas of the content image to be processed are more concentrated and critical, and the image obtained after style transfer has a better spatial structure and layering, for example: the structural texture of the image object is clearer.

[0061] The present invention provides a multi-attention fusion image style transfer method, comprising: obtaining image data to be processed, the image data comprising a content image to be processed and a style image to be processed; performing multi-scale feature extraction on the content image to be processed and the style image to be processed, respectively, to obtain content features and style features corresponding to the content image to be processed and the style image to be processed; fusing the content features and style features using a multi-attention fusion module to obtain attention features; and decoding the attention features to obtain a style-transferred image corresponding to the content image to be processed. This multi-attention fusion image style transfer method fuses the content features and style features using a multi-attention fusion module to obtain attention features, thereby focusing on and emphasizing salient regional information in the content image to be processed and the style image to be processed, while suppressing irrelevant background details. This method can enhance the stylized processing effect of style transfer on key image regions, establish global regionally connected attention features, and effectively control the local regions of style transfer, ensuring clear and complete edge texture transmission of the content image to be processed after style transfer, thereby resolving the technical problems of unclear and distorted textures in existing image processing style transfer technologies.

[0062] In one embodiment of the present invention, the multi-attention fusion image style transfer method includes: S5. extracting features from the style-transferred image to obtain transfer features corresponding to the style-transferred image; using a loss function to compare the transfer features with the content features and style features to obtain a transfer loss; and adjusting the transfer direction and output result of the style-transferred image based on the transfer loss to obtain a panoramic clear image corresponding to the content image to be processed after style transfer. The loss function is:

[0063] L=λ g L gs +λ l L lf

[0064]

[0065] Where L is the migration loss, λ g and λ l are all hyperparameters of the loss term weight, L gs is the penalty parameter for the distance and standard deviation between the content image to be processed and the style image, L lf is the feature mapping parameter of the style migration image, E() is the feature of the VGG encoder, x is the x-th layer index of the AdaAttN module, I cs is the style transfer image, F c is the content feature, F sis the style feature, μ is the average distance between the content image to be processed and the style image, σ is the standard deviation of the distance between the content image to be processed and the style image, and AdaAttN() is the AdaAttN function of the AdaAttN module.

[0066] It should be noted that if Figure 2 As shown, in step S5, the multi-attention fusion image style transfer method guides the style transfer image migration direction and output results in the VGG transformer according to the global style loss and local feature loss in the loss function, thereby obtaining a panoramic clear image corresponding to the content image to be processed after style transfer. In this embodiment, the loss function is mainly composed of the weighted sum of the global style loss and the local feature loss. In the loss function, λ g is the loss term weight hyperparameter of the global style loss; l is the loss term weight hyperparameter of local feature loss; L gs is a global style loss that penalizes the distance and standard deviation between the feature maps of the content image and the style image in the VGG-19 network to ensure the global stylization effect; L lf It is the feature map parameter of the style migration image, and is also used to constrain the feature map of the stylized image to be consistent with the transformation result of the AdaAttN function.

[0067] like Figure 2 As shown, in one embodiment of the present invention, the content features and style features are recorded as feature data, and a multi-attention fusion module is used to fuse the content features and style features to obtain the attention features, including:

[0068] The channel attention submodule of the multi-attention fusion module is used to serially process the input feature data and obtain first processed data by upsampling the processed data, and the first processed data is added and fused with the feature data to obtain second processed data;

[0069] The coordinate attention submodule of the multi-attention fusion module is used to parallel process the second processed data to obtain third processed data corresponding to the second processed data, and the second processed data and the third processed data are added and fused to obtain fourth processed data;

[0070] Using the channel attention submodule of the multi-attention fusion module to serially process the input feature data to obtain fifth processed data; and adding and fusing the fifth processed data with the fourth processed data to obtain sixth processed data;

[0071] The sixth processed data is output by the coordinate attention submodule of the multi-attention fusion module to obtain the attention feature;

[0072] Among them, the sixth processed data is output through the coordinate attention submodule of the multi-attention fusion module, and the attention features are obtained, including: using the mirror filling, coordinate attention submodule, convolution and coordinate attention submodule of the multi-attention fusion module to process the sixth processed data in turn and output the attention features.

[0073] It should be noted that the multi-attention fusion image style transfer method is divided into two paths to process the feature data in the process of fusing content features and style features. One path is to process the feature data through the channel attention submodule and the coordinate attention submodule of the multi-attention fusion module to obtain the fourth processed data; the other path is to process the feature data through the channel attention submodule of the multi-attention fusion module to obtain the fifth processed data; then the fourth processed data and the fifth processed data obtained by the two paths are fused to obtain the fused sixth processed data. In this embodiment, it is necessary to output the attention features, and it is also necessary to process the processed sixth processed data in turn through the mirror filling layer, coordinate attention submodule, convolution layer, and coordinate attention submodule of the AdaAttN module to output the attention features after style. Among them, fusion processing refers to the data obtained by adding multiple data.

[0074] In an embodiment of the present application, the attention submodule SE can learn the local spatial connection pattern representing all input channels and generate a feature map of spatial and channel information. The multi-attention fusion image style transfer method uses the attention submodule SE to process content features and style features, so that the stylized network can learn global information and selectively emphasize features with large amounts of information, and suppress less useful features, so that the style image has a good spatial structure and hierarchy. The attention submodule SE includes two steps, compression and excitation, which are used for embedding global information and weighting of adaptive channel relationships, respectively. The first step through the attention submodule SE is a compression operation, the purpose of which is to collect global channel information. That is, given the input data X, the squeeze operation of the cth channel. The expression for compressing the output data through the attention submodule SE is:

[0075]

[0076] Where Z c is the output of the cth channel, H and W are the height and width of the feature map respectively, x c is the input of data X in the cth channel, i and j represent the position coordinates of the two-dimensional plane. The second step of the attention submodule SE is the excitation operation, which aims to capture the dependency between channels. The dependency can be represented by the transformation function, which is Where σ is the Sigmoid function, Generated by the transformation function, T1 and T2 are two learnable linear transformations and are used to capture the importance of each channel.

[0077] In an embodiment of the present application, the coordinate attention submodule CA decomposes the channel attention into two parallel one-dimensional feature codes, and then integrates the spatial coordinate information into the generated attention map. Compared with the attention method on the existing lightweight network, the coordinate attention submodule CA of the present application can not only capture cross-channel information, but also capture direction perception and position perception information, which can enable the image to more accurately locate and identify the target area of interest during the style transfer process. The key areas of the transferred image can be clearly reproduced and visible. The expression of the output data of the coordinate attention submodule CA is: Where x c (i,j) is the input feature map of the cth channel, It is the feature map of the c-th channel embedded in the horizontal direction information, is the feature map of the c-th channel embedded in the vertical direction information, y c (i,j) is the final output obtained by the cth channel.

[0078] It should be noted that the existing AdaAttN module can focus on shallow and deep image features during image processing, but it does not pay attention to channel-wise features. This multi-attention fusion image style transfer method, using the channel-wise attention submodule SE, can compensate for the lack of channel-wise attention in the AdaAttN module. Furthermore, the coordinate attention submodule CA enables AdaAttN to capture long-range dependencies while also focusing on both spatial and channel aspects. This provides a more complete and comprehensive image information for style transfer.

[0079] Example 2:

[0080] Figure 3 This is a framework diagram of the multi-attention fusion image style transfer device described in an embodiment of the present invention.

[0081] like Figure 3 As shown, an embodiment of the present invention further provides an image style transfer device for multi-attention fusion, including a data acquisition module 10, a first VGG encoder 20, a multi-attention fusion module 30, a VGG decoder 40 and a second VGG encoder 50;

[0082] A data acquisition module 10 is used to acquire image data to be processed, where the image data includes a content image and a style image to be processed;

[0083] A first VGG encoder 20 is configured to perform multi-scale feature extraction on the content image and the style image to be processed, respectively, to obtain content features and style features corresponding to the content image and the style image to be processed;

[0084] A multi-attention fusion module 30 is used to fuse content features and style features to obtain attention features;

[0085] A VGG decoder 40 is used to decode the attention features to obtain a style transfer image corresponding to the content image to be processed;

[0086] The second VGG encoder 50 is used to extract features from the style-transferred image to obtain transfer features corresponding to the style-transferred image; use a loss function to compare the transfer features with the content features and the style features respectively to obtain a transfer loss; adjust the transfer direction and output result of the style-transferred image according to the transfer loss to obtain a panoramic clear image after style transfer corresponding to the content image to be processed.

[0087] In the embodiment of the present application, the loss function is:

[0088] L=λ g L gs +λ l L lf

[0089]

[0090] Where L is the migration loss, λ g and λ l are all hyperparameters of the loss term weight, L gs is the penalty parameter for the distance and standard deviation between the content image to be processed and the style image, L lf is the feature mapping parameter of the style migration image, E() is the feature of the VGG encoder, x is the x-th layer index of the AdaAttN module, I cs is the style transfer image, F c is the content feature, F s is the style feature, μ is the average distance between the content image to be processed and the style image, σ is the standard deviation of the distance between the content image to be processed and the style image, and AdaAttN() is the AdaAttN function of the AdaAttN module.

[0091] In an embodiment of the present application, the VGG decoder is a decoder of a symmetrically structured VGG-19 network; the multi-attention fusion module includes multiple channel attention submodules and coordinate attention submodules connected in series and parallel.

[0092] It should be noted that the contents of the modules and devices in the apparatus of Example 2 correspond to the contents of the steps in the method of Example 1. The contents of the multi-attention fusion image style transfer method have been elaborated in detail in Example 1, and the contents of the modules and devices in the apparatus will no longer be elaborated in this Example 2.

[0093] Example 3:

[0094] An embodiment of the present invention provides a terminal device, including a processor and a memory;

[0095] A memory, configured to store program codes and transmit the program codes to a processor;

[0096] A processor is configured to execute the above-mentioned multi-attention fusion image style transfer method according to instructions in the program code.

[0097] It should be noted that the processor is configured to execute the steps in the above-mentioned multi-attention fusion image style transfer method embodiment according to the instructions in the program code. Alternatively, the processor implements the functions of each module / unit in the above-mentioned system / device embodiments when executing the computer program.

[0098] For example, a computer program may be divided into one or more modules / units, one or more of which are stored in a memory and executed by a processor to complete the present application. One or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in a terminal device.

[0099] Terminal devices can be computing devices such as desktop computers, laptops, PDAs, and cloud servers. Terminal devices may include, but are not limited to, processors and memory. Those skilled in the art will appreciate that this does not constitute a limitation on terminal devices and may include more or fewer components than shown, or a combination of certain components, or different components. For example, terminal devices may also include input / output devices, network access devices, buses, and the like.

[0100] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0101] Memory can be an internal storage unit of a terminal device, such as a hard drive or memory. It can also be an external storage device, such as a plug-in hard drive, a SmartMedia Card (SMC), a Secure Digital (SD) card, or a flash memory card. Furthermore, memory can include both internal and external storage units. Memory is used to store computer programs and other programs and data required by the terminal device. Memory can also be used to temporarily store data that has been output or is about to be output.

[0102] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0103] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0104] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0105] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0106] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0107] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-attention fusion image style transfer method, characterized in that: The following steps are involved: Acquire image data to be processed, wherein the image data includes a content image to be processed and a style image; Performing multi-scale feature extraction on the content image to be processed and the style image, respectively, to obtain content features and style features corresponding to the content image to be processed and the style image; Using a multi-attention fusion module to fuse the content features and the style features to obtain attention features; Decoding the attention feature to obtain a style transfer image corresponding to the content image to be processed; The content features and the style features are recorded as feature data, and the content features and the style features are fused using a multi-attention fusion module to obtain attention features, including: Using the channel attention submodule of the multi-attention fusion module to serially process the input feature data and obtain first processed data by upsampling the processed data, and adding and fusing the first processed data with the feature data to obtain second processed data; Using the coordinate attention submodule of the multi-attention fusion module to parallel process the second processed data to obtain third processed data corresponding to the second processed data, and adding and fusing the second processed data and the third processed data to obtain fourth processed data; Using the channel attention submodule of the multi-attention fusion module to serially process the input feature data to obtain fifth processed data; and adding and fusing the fifth processed data with the fourth processed data to obtain sixth processed data; The sixth processed data is output through the coordinate attention submodule of the multi-attention fusion module to obtain attention features.

2. The multi-attention fusion image style transfer method according to claim 1, characterized in that include: Performing feature extraction on the style-transferred image to obtain a transfer feature corresponding to the style-transferred image; using a loss function to compare the transfer feature with the content feature and the style feature to obtain a transfer loss; The migration direction and output result of the style migration image are adjusted according to the migration loss to obtain a panoramic clear image after style migration corresponding to the content image to be processed.

3. The multi-attention fusion image style transfer method according to claim 2, characterized in that The loss function is: L=λ g L gs +λ l L lf Where L is the migration loss, λ g and λ l are all hyperparameters of the loss term weight, L gs is the penalty parameter for the distance and standard deviation between the content image to be processed and the style image, L lf is the feature mapping parameter of the style migration image, E() is the feature of the VGG encoder, x is the x-th layer index of the AdaAttN module, I cs is the style transfer image, F c is the content feature, F s is the style feature, μ is the average distance between the content image to be processed and the style image, σ is the standard deviation of the distance between the content image to be processed and the style image, and AdaAttN() is the AdaAttN function of the AdaAttN module.

4. The multi-attention fusion image style transfer method according to claim 1, characterized in that The sixth processed data is output through the coordinate attention submodule of the multi-attention fusion module, and the attention features are obtained, including: using the mirror filling, coordinate attention submodule, convolution and coordinate attention submodule of the multi-attention fusion module to process the sixth processed data in sequence and output the attention features.

5. A multi-attention fusion image style transfer device, characterized in that: It includes a data acquisition module, a first VGG encoder, a multi-attention fusion module, a VGG decoder, and a second VGG encoder; The data acquisition module is used to acquire image data to be processed, wherein the image data includes a content image and a style image to be processed; The first VGG encoder is configured to perform multi-scale feature extraction on the content image to be processed and the style image, respectively, to obtain content features and style features corresponding to the content image to be processed and the style image; The multi-attention fusion module is used to fuse the content features and the style features to obtain attention features; The VGG decoder is used to decode the attention feature to obtain a style transfer image corresponding to the content image to be processed; The second VGG encoder is used to perform feature extraction on the style-transferred image to obtain a transfer feature corresponding to the style-transferred image; and compare the transfer feature with the content feature and the style feature using a loss function to obtain a transfer loss; Adjusting the migration direction and output result of the style-migrated image according to the migration loss to obtain a panoramic clear image corresponding to the content image to be processed after style migration; The content features and the style features are recorded as feature data, and the content features and the style features are fused using a multi-attention fusion module to obtain attention features, including: Using the channel attention submodule of the multi-attention fusion module to serially process the input feature data and obtain first processed data by upsampling the processed data, and adding and fusing the first processed data with the feature data to obtain second processed data; Using the coordinate attention submodule of the multi-attention fusion module to parallel process the second processed data to obtain third processed data corresponding to the second processed data, and adding and fusing the second processed data and the third processed data to obtain fourth processed data; Using the channel attention submodule of the multi-attention fusion module to serially process the input feature data to obtain fifth processed data; and adding and fusing the fifth processed data with the fourth processed data to obtain sixth processed data; The sixth processed data is output through the coordinate attention submodule of the multi-attention fusion module to obtain attention features.

6. The multi-attention fusion image style transfer device according to claim 5, characterized in that The loss function is: L=λ g L gs +λ l L lf Where L is the migration loss, λ g and λ l are all hyperparameters of the loss term weight, L gs is the penalty parameter for the distance and standard deviation between the content image to be processed and the style image, L lf is the feature mapping parameter of the style migration image, E() is the feature of the VGG encoder, x is the x-th layer index of the AdaAttN module, I cs is the style transfer image, F c is the content feature, F s is the style feature, μ is the average distance between the content image to be processed and the style image, σ is the standard deviation of the distance between the content image to be processed and the style image, and AdaAttN() is the AdaAttN function of the AdaAttN module.

7. The multi-attention fusion image style transfer device according to claim 5, characterized in that The VGG decoder is a decoder of the symmetrical structure VGG-19 network.

8. The multi-attention fusion image style transfer device according to claim 5, characterized in that The multi-attention fusion module includes multiple channel attention sub-modules and coordinate attention sub-modules connected in series and parallel.

9. A terminal device, characterized in that: including a processor and a memory; The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the multi-attention fusion image style transfer method according to any one of claims 1 to 4 according to the instructions in the program code.

Images