Method, device, computer equipment and storage medium for transferring arbitrary aesthetic style of images

Abstract

The present application discloses a method, device, computer equipment and storage medium for transferring arbitrary aesthetic styles of images, wherein the method for transferring arbitrary aesthetic styles of images comprises: step 1, using a pre-trained deep neural network to extract content features F of content images respectively c and the style feature F of the style image s ; Step 2, use the aesthetic discriminator to extract the aesthetic features F of the style image a ; Step 3, use the aesthetic style attention module to process the content feature F c 、Style Features F s and aesthetic features a , and get the output feature F cs ; Step 4, using the decoder to process the output feature F cs , and obtain the image after style transfer. The aesthetically perceptible arbitrary style transfer method provided in this application can generate aesthetically realistic and refined style transfer results for any given content image and style image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, computer equipment and storage medium for transferring arbitrary aesthetic styles of images. Background Art

[0002] Style transfer refers to transferring the artistic style of a reference image to another content image, such as transferring the style of Van Gogh's "Starry Night" to an everyday photo. As the central task of style transfer, arbitrary style transfer mainly achieves the rapid acquisition of good style transfer results for any given content image and style image. According to the differences in the methods of manipulating content features and style features, the current arbitrary style transfer methods can be roughly divided into two categories: the first category is based on the statistical correlation of global features, AdaIN proposed by Huang et al., WCT and LST proposed by Li et al., etc. belong to this category; the other category is based on local feature patches (Patch) methods, AvatarNet proposed by Sheng et al., SANet proposed by Park et al., etc. belong to this category.

[0003] Although current arbitrary style transfer methods have been able to achieve higher quality results than traditional methods and perform well in speed and generalization, they often generate some discordant style patterns and obvious artificial artifacts, making their generated results easily distinguishable from real paintings. This problem is called the aesthetic unrealism problem, which is prevalent in existing style transfer methods.

[0004] Therefore, it is of great significance to study a method for transferring arbitrary aesthetic styles of images, which can help us generate more aesthetically realistic and satisfactory style transfer results, greatly narrowing the gap between paintings generated by artificial intelligence and paintings created by real artists. Summary of the invention

[0005] Based on this, it is necessary to provide an arbitrary aesthetic style transfer method for images, which can generate aesthetically realistic and refined style transfer results for any given content image and style image.

[0006] A method for transferring arbitrary aesthetic styles of images, comprising:

[0007] Step 1: Use the pre-trained deep neural network to extract the content features F of the content image c and the style feature F of the style image s ;

[0008] Step 2: Use the aesthetic discriminator to extract the aesthetic features F of the style imagea ;

[0009] Step 3: Use the aesthetic style attention module to process the content feature F c 、Style Features F s and aesthetic features a , and get the output feature F cs ;

[0010] Step 4: Use the decoder to process the output feature F cs , and get the image after style transfer.

[0011] Several optional methods are also provided below, but they are not intended to be additional limitations on the above-mentioned overall solution, but are merely further supplements or preferences. Under the premise that there are no technical or logical contradictions, each optional method can be combined with the above-mentioned overall solution separately, and multiple optional methods can also be combined.

[0012] Optionally, the aesthetic discriminator comprises:

[0013] Three encoders with the same structure, namely E1, E2, and E3;

[0014] Three classifiers with the same structure, namely C1, C2, and C3;

[0015] The aesthetic discriminator extracts aesthetic features F using the following algorithm a :

[0016]

[0017] Where:

[0018] F a For the final extracted aesthetic features;

[0019] I s is the input style image;

[0020] It is an element-by-element addition operation;

[0021] ↓ i and ↑ i They represent downsampling and upsampling operations respectively, and i represents the sampling factor.

[0022] Optionally, the aesthetic style attention module includes: a global aesthetics-guided style enhancement module and a local structure-guided style integration module;

[0023] The global aesthetically guided style enhancement module is used to enhance the aesthetic guidance of style features, and specifically includes:

[0024] Step 3-a-1: the global aesthetics-guided style enhancement module performs s and aesthetic features a Perform the following processing to obtain the vectorized features:

[0025]

[0026]

[0027]

[0028] Where:

[0029] Γ represents the feature vectorization operation;

[0030] and f a It is a 1×1 learnable convolutional layer;

[0031] C.H. s and W s are the corresponding style features F s and aesthetic features a The number of channels, height and width;

[0032] Step 3-a-2, calculate features With features Aesthetic attention between:

[0033]

[0034] Where:

[0035] A a is the obtained aesthetic attention map;

[0036] Softmax is the Softmax operation;

[0037] is the matrix multiplication operation;

[0038] T is the matrix transpose operation;

[0039] Step 3-a-3, use the following algorithm to analyze the style feature F s Perform aesthetic guidance enhancement to obtain the aesthetically enhanced style feature F sa :

[0040]

[0041] Where:

[0042] F sa Stylistic features enhanced for aesthetics;

[0043] is the matrix multiplication operation;

[0044] is the matrix addition operation;

[0045] It is the inverse operation of Γ, that is, restoring the feature size to the original size;

[0046] is a 1×1 learnable convolutional layer.

[0047] Optionally, the local structure-guided style integration module is used to perform structure-guided integration of style features, specifically including:

[0048] Step 3-b-1: the local structure-guided style integration module performs c and aesthetically enhanced style features F sa Perform the following processing to obtain the vectorized features:

[0049]

[0050]

[0051]

[0052] Where:

[0053] Γ represents the feature vectorization operation;

[0054] Norm represents the mean and variance normalization operation of the channel dimension;

[0055] f c , and It is a 1×1 learnable convolutional layer;

[0056] C.H. c and W c They are content features F c The number of channels, height and width;

[0057] C.H. s and W s They are the style features F after aesthetic enhancement sa The number of channels, height and width;

[0058] Step 3-b-2, calculate features and Style attention between:

[0059]

[0060] Where: A sis the obtained style attention map;

[0061] Softmax is the Softmax operation;

[0062] is the matrix multiplication operation;

[0063] Step 3-b-3, use the following algorithm to analyze the style feature F sa Perform structure-guided integration to obtain the output feature F cs :

[0064]

[0065] Where: F cs is the final output feature;

[0066] It is the inverse operation of Γ, that is, restoring the feature size to the original size;

[0067] is a 1×1 learnable convolutional layer.

[0068] Optionally, in step 1, the pre-trained deep neural network uses a VGG-19 network trained for classification tasks on the ImageNet dataset.

[0069] Optionally, in step 4, the decoder network is an inverse network of the VGG-19 network, and the pooling layer therein is replaced by an upsampling operation based on nearest neighbor interpolation.

[0070] Optionally, when the aesthetic discriminator, the aesthetic style attention module, and the decoder are trained, content loss, style loss, adversarial loss, and aesthetic regularization loss are used together for training;

[0071] The adversarial loss The definition is as follows:

[0072]

[0073]

[0074] Where:

[0075] is the aesthetic discriminator;

[0076] G is the generator, including the decoder D, the aesthetic style attention module AesSA, and the pre-trained VGG network E with fixed parameters vgg ;

[0077] I c is the content image, from the content image dataset Φ cMedium sampling;

[0078] I s is the style image, from the style image dataset Φ s Medium sampling;

[0079] The aesthetic regularization loss The definition is as follows:

[0080]

[0081] Where:

[0082] I cs|s Indicated by I c As the content image, I s as a style image, and I s The resulting image generated as an aesthetically guided image;

[0083] I cscs|cs Indicated by I cs|s The resulting image is generated as a content image, a style image, and an aesthetic guidance image at the same time;

[0084] μ represents the mean;

[0085] σ represents the standard deviation;

[0086] is an aesthetic discriminator.

[0087] The present application also provides a device for transferring an arbitrary aesthetic style of an image, comprising:

[0088] The first module is used to extract the content features F of the content image using the pre-trained deep neural network. c and the style feature F of the style image s ;

[0089] The second module is used to extract the aesthetic features F of the style image using the aesthetic discriminator a ;

[0090] The third module is used to process the content feature F using the aesthetic style attention module c 、Style Features F s and aesthetic features a , and get the output feature F cs ;

[0091] The fourth module is used to process the output feature F using a decoder cs , and get the image after style transfer.

[0092] The present application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method for transferring arbitrary aesthetic styles of images when executing the computer program.

[0093] The present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method for transferring an arbitrary aesthetic style of an image are implemented.

[0094] The method for transferring an arbitrary aesthetic style of an image provided by the present application can generate an aesthetically authentic and refined style transfer result for any given content image and style image, and its beneficial effects are:

[0095] 1. Compared with the traditional arbitrary style transfer method based on deep learning, the aesthetic information is taken into account to improve the quality of style transfer, effectively solving the aesthetic unrealistic problems such as the disharmony of the style mode after transfer and the introduction of obvious artificial artifacts. The generated painting images are closer to the real paintings created by artists, greatly narrowing the gap between paintings generated by artificial intelligence and paintings created by real artists.

[0096] 2. Compared with other style transfer methods that consider aesthetic information, this method uses a large dataset of paintings created by artists and learns common human-perceivable aesthetic features through adversarial training, and injects these features into the aesthetic style attention module to better guide the style transfer of aesthetic perception. The quality and authenticity of the generated results are more in line with human aesthetic perception and preference. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] Figure 1 This is a flowchart of a method for transferring an arbitrary aesthetic style of an image in one embodiment of the present application;

[0098] Figure 2 This is a network structure diagram of an aesthetic discriminator in one embodiment of the present application;

[0099] Figure 3a A schematic diagram of a style enhancement module guided by global aesthetics in an aesthetic style attention module in one embodiment of the present application;

[0100] Figure 3b A schematic diagram of a local structure-guided style integration module in an aesthetic style attention module in one embodiment of the present application;

[0101] Figure 4 This is a schematic diagram of calculating the training loss of a style transfer network in one embodiment of the present application;

[0102] Figure 5 A schematic diagram of computer equipment. DETAILED DESCRIPTION

[0103] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0104] See also Figure 1 As shown, the image arbitrary aesthetic style transfer method includes:

[0105] Step 1: Use the pre-trained deep neural network to extract the content features F of the content image c and the style feature F of the style image s ;

[0106] Step 2: Use the aesthetic discriminator to extract the aesthetic features F of the style image a ;

[0107] Step 3: Use the aesthetic style attention module to process the content feature F c 、Style Features F s and aesthetic features a , and get the output feature F cs ;

[0108] Step 4: Use the decoder to process the output feature F cs , and get the image after style transfer.

[0109] The image arbitrary style transfer method can generate aesthetically realistic and refined style transfer results for any given content image and style image. The aesthetic discriminator used in it plays two roles in the network: the first role is the discriminator in adversarial training, which enables the generator to generate style transfer results that are more like real paintings in an adversarial way; the second role is the aesthetic feature extractor, which can learn some common human-perceivable aesthetic features of paintings by conducting adversarial training on a large number of real painting images. Therefore, the style image can be input to extract relevant aesthetic features, and these aesthetic features can be injected into the style transfer process to better guide the style transfer and help generate higher quality style transfer results.

[0110] In addition, through the aesthetic style attention module, the style pattern of the style image can be better transferred to the content image, because it not only considers the local structural information guidance of the content image when performing style transfer, but also considers the global aesthetic information guidance of the style image.

[0111] The aesthetic discriminator adopts a multi-scale architecture. In one embodiment, see Figure 2 As shown, the aesthetic discriminator includes:

[0112] Three encoders with the same structure, namely E1, E2, and E3;

[0113] Three classifiers with the same structure, namely C1, C2, and C3;

[0114] The aesthetic discriminator extracts aesthetic features F using the following algorithm a :

[0115]

[0116] Where:

[0117] F a For the final extracted aesthetic features;

[0118] I s is the input style image;

[0119] It is an element-by-element addition operation;

[0120] ↓ i and ↑ i They represent downsampling and upsampling operations respectively, and i represents the sampling factor.

[0121] When training the aesthetic discriminator, the aesthetic style attention module, and the decoder, the aesthetic discriminator is used as a discriminator in adversarial training, and enables the generator to generate style transfer results that are more like real paintings through adversarial means. Specifically, the aesthetic discriminator is used to judge whether the image after style transfer meets the result of target style transfer, that is, whether the style of the image after transfer meets the data distribution of the style of the painting created by the real painter.

[0122] See also Figure 2 As shown, when the aesthetic discriminator performs image discrimination, first, the generated fake painting image and the real painting image are downsampled twice respectively, and each downsampling reduces the length and width of the image to half of the original, so as to obtain images of three scales including the image of the original size, and then for each scale of the image, a pair of encoders and classifiers (such as E1 and C1) are used to distinguish the authenticity of the input image.

[0123] See also Figure 3a , Figure 3b As shown, in one embodiment, the aesthetic style attention module includes: a global aesthetics-guided style enhancement module and a local structure-guided style integration module;

[0124] See also Figure 3aAs shown, the global aesthetically guided style enhancement module is used to perform aesthetically guided enhancement of style features, specifically including:

[0125] Step 3-a-1: the global aesthetics-guided style enhancement module performs s and aesthetic features a Perform the following processing to obtain the vectorized features:

[0126]

[0127]

[0128]

[0129] Where:

[0130] Γ represents the feature vectorization operation;

[0131] and f a It is a 1×1 learnable convolutional layer;

[0132] C.H. s and W s are the corresponding style features F s and aesthetic features a The number of channels, height and width;

[0133] Step 3-a-2, calculate features With features Aesthetic attention between:

[0134]

[0135] Where:

[0136] A a is the obtained aesthetic attention map;

[0137] Softmax is the Softmax operation;

[0138] is the matrix multiplication operation;

[0139] T is the matrix transpose operation;

[0140] Step 3-a-3, use the following algorithm to analyze the style feature F s Perform aesthetic guidance enhancement to obtain the aesthetically enhanced style feature F sa :

[0141]

[0142] Where:

[0143] F sa Stylistic features enhanced for aesthetics;

[0144] is the matrix multiplication operation;

[0145] is the matrix addition operation;

[0146] It is the inverse operation of Γ, that is, restoring the feature size to the original size;

[0147] is a 1×1 learnable convolutional layer.

[0148] See also Figure 3b As shown, in one embodiment, the local structure-guided style integration module is used to perform structure-guided integration of style features, specifically including:

[0149] Step 3-b-1: the local structure-guided style integration module performs c and aesthetically enhanced style features F sa Perform the following processing to obtain the vectorized features:

[0150]

[0151]

[0152]

[0153] Where:

[0154] Γ represents the feature vectorization operation;

[0155] Norm represents the mean and variance normalization operation of the channel dimension;

[0156] f c , and It is a 1×1 learnable convolutional layer;

[0157] C.H. c and W c They are content features F c The number of channels, height and width;

[0158] C.H. s and W s They are the style features F after aesthetic enhancement sa The number of channels, height and width;

[0159] Step 3-b-2, calculate features and Style attention between:

[0160]

[0161] Where: A s is the obtained style attention map;

[0162] Softmax is the Softmax operation;

[0163] is the matrix multiplication operation;

[0164] Step 3-b-3, use the following algorithm to analyze the style feature F sa Perform structure-guided integration to obtain the output feature F cs :

[0165]

[0166] Where: F cs is the final output feature;

[0167] It is the inverse operation of Γ, that is, restoring the feature size to the original size;

[0168] is a 1×1 learnable convolutional layer.

[0169] See also Figure 1 As shown, in one embodiment, in step 1, the pre-trained deep neural network uses a VGG-19 network trained for classification tasks on the ImageNet dataset.

[0170] See also Figure 1 As shown, in one embodiment, in step 4, the decoder network is an inverse network of the VGG-19 network, and the pooling layer therein is replaced by an upsampling operation based on nearest neighbor interpolation.

[0171] In one embodiment, see Figure 4 As shown, when the aesthetic discriminator, the aesthetic style attention module, and the decoder are trained, content loss, style loss, adversarial loss, and aesthetic regularization loss are used together for training.

[0172] Specifically, the content loss The definition is as follows:

[0173]

[0174] Where:

[0175] I cs The final output style migration result image;

[0176] I c for content images;

[0177] φ i A layer of the pre-trained VGG-19 network;

[0178] L c A network layer that uses content loss calculation, such as {Relu4_1, Relu5_1} layer.

[0179] Specifically, the style loss The definition is as follows:

[0180]

[0181] Where:

[0182] μ represents the mean;

[0183] σ represents the standard deviation;

[0184] φ i A layer of the pre-trained VGG-19 network;

[0185] I cs The final output style migration result image;

[0186] I s is the style image;

[0187] L s The network layer that uses style loss calculation, such as {Relu1_1,Relu2_1,Relu3_1,Relu4_1,Relu5_1} layer.

[0188] Specifically, against loss The definition is as follows:

[0189]

[0190]

[0191] Where:

[0192] is the aesthetic discriminator;

[0193] G is the generator, including the decoder D, the aesthetic style attention module AesSA, and the pre-trained VGG network E with fixed parameters vgg ;

[0194] I c is the content image, from the content image dataset Φ c Medium sampling.

[0195] Is is the style image, from the style image dataset Φ s Medium sampling.

[0196] Optionally, MS-COCO can be used as the content image dataset Φ c , WikiArt as a style image dataset Φ s .

[0197] Specifically, the aesthetic regularization loss The definition is as follows:

[0198]

[0199] Where:

[0200] I cs|s Indicated by I c As the content image, I s as a style image, and I s The resulting image generated as an aesthetically guided image;

[0201] I cscs|cs Indicated by I cs|s The resulting image is generated as a content image, a style image, and an aesthetic guidance image at the same time;

[0202] μ represents the mean;

[0203] σ represents the standard deviation;

[0204] is an aesthetic discriminator.

[0205] The present application also provides a device for transferring an arbitrary aesthetic style of an image, comprising:

[0206] The first module is used to extract the content features F of the content image using the pre-trained deep neural network. c and the style feature F of the style image s ;

[0207] The second module is used to extract the aesthetic features F of the style image using the aesthetic discriminator a ;

[0208] The third module is used to process the content feature F using the aesthetic style attention module c 、Style Features F s and aesthetic features a , and get the output feature F cs ;

[0209] The fourth module is used to process the output feature F using a decoder cs , and get the image after style transfer.

[0210] For the functional limitations of each module, please refer to the limitations of the image arbitrary aesthetic style transfer method above, which will not be repeated here. Each module in the above-mentioned image arbitrary aesthetic style transfer device can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0211] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an arbitrary aesthetic style method of an image is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0212] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0213] The present application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method for transferring arbitrary aesthetic styles of images when executing the computer program.

[0214] The present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method for transferring an arbitrary aesthetic style of an image are implemented.

[0215] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0216] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0217] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. The method for transferring arbitrary aesthetic styles of images is characterized by: include: Step 1: Use the pre-trained deep neural network to extract the content features F of the content image c and the style feature F of the style image s ; Step 2: Use the aesthetic discriminator to extract the aesthetic features F of the style image a ; Step 3: Use the aesthetic style attention module to process the content feature F c 、Style Features F s and aesthetic features a , and get the output feature F cs ; Step 4: Use the decoder to process the output feature F cs , get the image after style transfer; The aesthetic discriminator comprises: Three encoders with the same structure, namely E1, E2, and E3; Three classifiers with the same structure, namely C1, C2, and C3; The aesthetic discriminator extracts the aesthetic feature F using the following algorithm a : Where: F a For the final extracted aesthetic features; I s is the input style image; It is an element-by-element addition operation; ↓ i and ↑ i Represent downsampling and upsampling operations respectively, i represents the sampling factor; The aesthetic style attention module includes: a style enhancement module guided by global aesthetics and a style integration module guided by local structure; The global aesthetically guided style enhancement module is used to enhance the aesthetic guidance of style features, and specifically includes: Step 3-a-1: the global aesthetics-guided style enhancement module performs s and aesthetic features a Perform the following processing to obtain the vectorized features: Where: Γ represents the feature vectorization operation; f s 1 、f s 2 and f a It is a 1×1 learnable convolutional layer; C.H. s and W s They are the corresponding style features F s and aesthetic features a The number of channels, height and width; Step 3-a-2, calculate features With features Aesthetic attention between: Where: A a is the obtained aesthetic attention map; Softmax is the Softmax operation; is the matrix multiplication operation; T is the matrix transpose operation; Step 3-a-3, use the following algorithm to analyze the style feature F s Perform aesthetic guidance enhancement to obtain the aesthetically enhanced style feature F sa : Where: F sa Stylistic features enhanced for aesthetics; is the matrix multiplication operation; is the matrix addition operation; It is the inverse operation of Γ, that is, restoring the feature size to the original size; It is a 1×1 learnable convolutional layer; The local structure-guided style integration module is used to perform structure-guided integration of style features, specifically including: Step 3-b-1: the local structure-guided style integration module performs c and aesthetically enhanced style features F sa Perform the following processing to obtain the vectorized features: Where: Γ represents the feature vectorization operation; Norm represents the mean and variance normalization operation of the channel dimension; f c , and It is a 1×1 learnable convolutional layer; C.H. c and W c They are content features F c The number of channels, height and width; C.H. s and W s They are the style features F after aesthetic enhancement sa The number of channels, height and width; Step 3-b-2, calculate features and Style attention between: Where: A s is the obtained style attention map; Softmax is the Softmax operation; is the matrix multiplication operation; Step 3-b-3, use the following algorithm to analyze the style feature F sa Perform structure-guided integration to obtain the output feature F cs : Where: F cs is the final output feature; It is the inverse operation of Γ, that is, restoring the feature size to the original size; is a 1×1 learnable convolutional layer.

2. The method for transferring an arbitrary aesthetic style of an image according to claim 1, characterized in that: In step 1, the pre-trained deep neural network uses the VGG-19 network trained for classification tasks on the ImageNet dataset.

3. The method for transferring arbitrary aesthetic styles of images according to claim 1, characterized in that: In step 4, the decoder network is the inverse network of the VGG-19 network, and the pooling layer therein is replaced by an upsampling operation based on nearest neighbor interpolation.

4. The method for transferring an arbitrary aesthetic style of an image according to any one of claims 1 to 3, characterized in that: When the aesthetic discriminator, the aesthetic style attention module, and the decoder are trained, content loss, style loss, adversarial loss, and aesthetic regularization loss are used together for training; The adversarial loss The definition is as follows: Where: is the aesthetic discriminator; G is the generator, including the decoder D, the aesthetic style attention module AesSA, and the pre-trained VGG network E with fixed parameters vgg ; c is the content image, from the content image dataset Φ c Medium sampling; I s is the style image, from the style image dataset Φ s Medium sampling; The aesthetic regularization loss The definition is as follows: Where: I cs|s Indicated by I c As the content image, I s as a style image, and I s The resulting image generated as an aesthetically guided image; I cscs|cs Indicated by I cs|s The resulting image is generated as a content image, a style image, and an aesthetic guidance image at the same time; μ represents the mean; σ represents standard deviation.

5. The device for transferring arbitrary aesthetic styles of images is characterized by: include: The first module is used to extract the content features F of the content image using the pre-trained deep neural network. c and the style feature F of the style image s ; The second module is used to extract the aesthetic features F of the style image using the aesthetic discriminator a ; The aesthetic discriminator comprises: Three encoders with the same structure, namely E1, E2, and E3; Three classifiers with the same structure, namely C1, C2, and C3; The aesthetic discriminator extracts the aesthetic feature F using the following algorithm a : Where: F a For the final extracted aesthetic features; I s is the input style image; It is an element-by-element addition operation; ↓ i and ↑ i Represent downsampling and upsampling operations respectively, i represents the sampling factor; The third module is used to process the content feature F using the aesthetic style attention module c 、Style Features F s and aesthetic features a , and get the output feature F cs ; The aesthetic style attention module includes: a style enhancement module guided by global aesthetics and a style integration module guided by local structure; The global aesthetically guided style enhancement module is used to enhance the aesthetic guidance of style features, and specifically includes: Step 3-a-1: the global aesthetics-guided style enhancement module performs s and aesthetic features a Perform the following processing to obtain the vectorized features: Where: Γ represents the feature vectorization operation; f s 1 、f s 2 and f a It is a 1×1 learnable convolutional layer; C.H. s and W s They are the corresponding style features F s and aesthetic features a The number of channels, height and width; Step 3-a-2, calculate features With features Aesthetic attention between: Where: A a is the obtained aesthetic attention map; Softmax is the Softmax operation; is the matrix multiplication operation; T is the matrix transpose operation; Step 3-a-3, use the following algorithm to analyze the style feature F s Perform aesthetic guidance enhancement to obtain the aesthetically enhanced style feature F sa : Where: F sa Stylistic features enhanced for aesthetics; is the matrix multiplication operation; is the matrix addition operation; It is the inverse operation of Γ, that is, restoring the feature size to the original size; It is a 1×1 learnable convolutional layer; The local structure-guided style integration module is used to perform structure-guided integration of style features, specifically including: Step 3-b-1: the local structure-guided style integration module performs c and aesthetically enhanced style features F sa Perform the following processing to obtain the vectorized features: Where: Γ represents the feature vectorization operation; Norm represents the mean and variance normalization operation of the channel dimension; f c , and It is a 1×1 learnable convolutional layer; C.H. c and W c They are content features F c The number of channels, height and width; C.H. s and E s They are the style features F after aesthetic enhancement sa The number of channels, height and width; Step 3-b-2, calculate features and Style attention between: Where: A s is the obtained style attention map; Softmax is the Softmax operation; is the matrix multiplication operation; Step 3-b-3, use the following algorithm to analyze the style feature F sa Perform structure-guided integration to obtain the output feature F cs : Where: F cs is the final output feature; It is the inverse operation of Γ, that is, restoring the feature size to the original size; It is a 1×1 learnable convolutional layer; The fourth module is used to process the output feature F using a decoder cs , and get the image after style transfer.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method for transferring an arbitrary aesthetic style of an image according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for transferring an arbitrary aesthetic style of an image according to any one of claims 1 to 4 are implemented.

Images