Image real-time style transfer method, device, computer equipment and storage medium

Abstract

The present application discloses a method, apparatus, computer device and storage medium for real-time image style transfer, wherein the method for real-time image style transfer comprises: step 1, extracting content features f of a content image by using a micro content encoder c ; Step 2, use the micro-style encoder to extract the style features f of the style image s ; Step 3, use the style modulator to transform the style feature f s Converted into style modulation signal m s ; Step 4, using the micro decoder to process the content feature f c and style modulation signal m s , and obtain the image after style transfer. This application can quickly transfer the artistic style of any given style image to other given content images, and is particularly suitable for ultra-high-definition images.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, apparatus, computer device and storage medium for real-time image style transfer. Background Art

[0002] Since Gatys et al. first applied deep learning techniques to the field of image style transfer in 2015, style transfer has begun to attract growing interest from academia and industry. A core problem in this field is to transfer the artistic style of an arbitrary given style image to another target content image, which is called arbitrary style transfer. By leveraging the salient feature expression capabilities of pre-trained deep convolutional neural networks (such as VGG), existing arbitrary style transfer algorithms have been able to achieve satisfactory stylization effects on arbitrary images. However, the high computational cost of large pre-trained deep neural networks has hindered the application of current arbitrary style transfer methods in practical scenarios, such as processing ultra-high resolution / ultra-high definition (such as 4096×2160 pixels, or "4K") images.

[0003] Some methods have made valuable efforts to solve this dilemma. One approach is to compress large pre-trained deep neural network models without losing too much performance. In 2020, Wang et al. proposed using collaborative distillation to reduce the convolution filters of the VGG network and successfully rendered ultra-high-resolution images on a single GPU graphics card with 12GB of video memory. Although the memory consumption is significantly reduced, the compressed model is often not fast enough. Another solution is to divide the ultra-high-resolution image into several small patches (Patch), and then perform style transfer on each patch separately. Although this method can achieve style transfer of arbitrary resolution, it still has efficiency problems. In addition, some methods design lightweight networks for style transfer, however, since their style features are still extracted from large pre-trained VGG, they are inherently difficult to handle ultra-high-resolution images. Therefore, despite the recent rapid progress, the existing arbitrary style transfer methods either cannot handle ultra-high-resolution images or are too slow to process. It is of great significance to study a fast style transfer method for ultra-high-resolution / ultra-high-definition images, both in academic research in academia and in practical applications in industry. Summary of the invention

[0004] Based on this, it is necessary to provide a fast image style transfer method that can quickly transfer the artistic style of any given style image to other given content images, especially for ultra-high-definition images.

[0005] A real-time image style transfer method, comprising:

[0006] Step 1: Use the micro content encoder to extract the content features f of the content image c ;

[0007] Step 2: Use the micro-style encoder to extract the style features f of the style image s ;

[0008] Step 3: Use the style modulator to transform the style feature f s Converted into style modulation signal m s ;

[0009] Step 4: Process the content feature f using a micro decoder c and style modulation signal m s , and get the image after style transfer.

[0010] Several optional methods are also provided below, but they are not intended to be additional limitations on the above-mentioned overall solution, but are merely further supplements or preferences. Under the premise that there are no technical or logical contradictions, each optional method can be combined with the above-mentioned overall solution separately, and multiple optional methods can also be combined.

[0011] Optionally, the micro content encoder comprises:

[0012] A standard convolutional layer with a stride of 1 and a kernel size of 9×9;

[0013] Two depthwise separable convolutional layers with stride 2 and kernel size 3×3;

[0014] Two residual blocks with stride 1;

[0015] The standard convolution layer, the depth-separable convolution layer and the residual block are connected in series, and the output of the last residual block is the extracted content feature.

[0016] Optionally, the micro-style encoder comprises:

[0017] A standard convolutional layer with a stride of 1 and a kernel size of 9×9;

[0018] Two depthwise separable convolutional layers with stride 2 and kernel size 3×3;

[0019] Two residual blocks with stride 1;

[0020] The standard convolution layer, the depth-separable convolution layer, and the residual block are connected in series, and the output of the last residual block is the extracted style feature.

[0021] Optionally, the style modulator comprises: a weight subnetwork ξ w and bias subnetwork ξ b ;

[0022] The weight subnetwork ξ w include:

[0023] A standard convolutional layer with a stride of 1 and a kernel size of 1×1;

[0024] A global pooling layer;

[0025] The bias subnetwork ξ b include:

[0026] A standard convolutional layer with a stride of 1 and a kernel size of 1×1;

[0027] A global pooling layer;

[0028] The style modulator uses the following algorithm to transform the style feature f s Converted into style modulation signal m s :

[0029] m s =(μ(f s ),σ(f s ),ξ w (f s ),ξ b (f s ))

[0030] Where:

[0031] μ(f s ) is the style feature f s The mean of

[0032] σ(f s ) is the style feature f s The standard deviation of

[0033] ξ w (f s ) is to use the weight sub-network ξ w The extracted weight parameters;

[0034] ξ b (f s ) is to use the bias subnetwork ξ b Extracted bias parameters.

[0035] Optionally, the micro decoder comprises:

[0036] Two residual blocks with stride 1;

[0037] Two depthwise separable convolutional layers with upsampling operations and kernel size of 3×3;

[0038] A standard convolutional layer with a stride of 1 and a kernel size of 9×9;

[0039] The residual block, the depth-separable convolutional layer and the standard convolutional layer are connected in series, and the output of the last standard convolutional layer is the image after style transfer.

[0040] Optionally, the micro decoder processes the content feature f using a dual-path modulation method combining feature modulation FeatMod and network filter modulation FilterMod. c and style modulation signal m s ;

[0041] The algorithm of the feature modulation FeatMod is as follows:

[0042]

[0043] Where:

[0044] σ(f s ) is the style feature f s The standard deviation of

[0045] μ(f s ) is the style feature f s The mean of

[0046] σ(f c ) is the content feature f c The standard deviation of

[0047] μ(f c ) is the content feature f c The mean of

[0048] The algorithm of the network filter modulation FilterMod is as follows:

[0049] FilterMod(D,m s )=ResBlock(f c ,(ξ w (f s ),ξ b (f s )))

[0050] =Conv(Relu(Conv(f c ,(ξ w (f s ),ξ b (f s )))),(ξ w (f s ),ξ b (f s )))+f c

[0051] Where:

[0052] D represents the micro decoder;

[0053] ResBlock represents a residual block in the micro-decoder;

[0054] Conv is the convolutional layer in the residual block in the micro-decoder;

[0055] Relu is the activation layer in the residual block in the micro decoder.

[0056] Optionally, when training the micro content encoder, the micro style encoder, the style modulator and the micro decoder, three losses, namely, content loss, style loss and style signal contrast loss, are used together for training;

[0057] The style signal contrast loss The definition is as follows:

[0058]

[0059] Where:

[0060] N represents the number of samples in a training batch;

[0061] is the style modulation signal extracted from the i-th style transfer result image through the micro-style encoder and style modulator;

[0062] is the style modulation signal extracted from the i-th style image through the micro-style encoder and style modulator;

[0063] is the style modulation signal extracted from the j-th style image through the micro-style encoder and style modulator.

[0064] The present application also provides an image real-time style transfer device, comprising:

[0065] The first module is used to extract content features f of the content image using a micro content encoder c ;

[0066] The second module is used to extract the style features f of the style image using a micro-style encoder s ;

[0067] The third module is used to use the style modulator to transform the style feature f s Converted into style modulation signal m s ;

[0068] The fourth module is used to process the content feature f using a micro decoder c and style modulation signal m s , and get the image after style transfer.

[0069] The present application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the image real-time style transfer method when executing the computer program.

[0070] The present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method for real-time image style transfer are implemented.

[0071] The fast ultra-high-definition image style transfer method provided in this application can perform fast style transfer for any given ultra-high-definition style image and ultra-high-definition content image, and its beneficial effects are:

[0072] 1. Compared with traditional deep learning-based style transfer methods, no large pre-trained networks are involved in the testing phase, so it is able to process ultra-high-resolution images (such as 4096×2160 pixels, or "4K") with limited computing resources. At the same time, due to the use of dual-path modulation of style signals and contrastive learning training, the quality of style transfer will not be significantly reduced.

[0073] 2. Compared with other style transfer methods that can handle ultra-high resolution, it uses a miniature neural network structure, which greatly improves the speed of processing ultra-high resolution images and can achieve fast real-time style transfer of ultra-high resolution images. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 This is a flowchart of a method for real-time image style transfer in one embodiment of the present application;

[0075] Figure 2 This is a network structure diagram of a micro content encoder and a micro style encoder in one embodiment of the present application;

[0076] Figure 3 A schematic diagram of the network structure of the style modulator and the micro decoder and the dual-path modulation of the style signal in one embodiment of the present application;

[0077] Figure 4 This is a schematic diagram of calculating the training loss of a micro-style transfer network in one embodiment of the present application;

[0078] Figure 5 A schematic diagram of computer equipment. DETAILED DESCRIPTION

[0079] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0080] See also Figure 1 As shown, a real-time image style transfer method includes:

[0081] Step 1: Use the micro content encoder to extract the content features f of the content image c ;

[0082] Step 2: Use the micro-style encoder to extract the style features f of the style image s ;

[0083] Step 3: Use the style modulator to transform the style feature f s Converted into style modulation signal m s ;

[0084] Step 4: Process the content feature f using a micro decoder c and style modulation signal m s , and get the image after style transfer.

[0085] The real-time image style transfer method provided by the present application can quickly transfer the artistic style of the ultra-high-definition style image to the ultra-high-definition content image for any given ultra-high-definition style image and ultra-high-definition content image. The neural network model used in the whole process is miniature and lightweight, and does not involve any large-scale deep neural network (such as VGG). Therefore, it can not only process ultra-high-definition images (such as 4K) that cannot be processed by previous methods on limited computing resources (such as a GPU graphics card with 8GB of video memory), but also can quickly generate the resulting image after style transfer at a real-time speed.

[0086] See also Figure 2 As shown, in one embodiment, the micro content encoder includes:

[0087] A standard convolutional layer with a stride of 1 and a kernel size of 9×9;

[0088] Two depthwise separable convolutional layers with stride 2 and kernel size 3×3;

[0089] Two residual blocks with stride 1;

[0090] The standard convolution layer, the depth-separable convolution layer and the residual block are connected in series, and the output of the last residual block is the extracted content feature.

[0091] See also Figure 2 As shown, in one embodiment, the micro-style encoder includes:

[0092] A standard convolutional layer with a stride of 1 and a kernel size of 9×9;

[0093] Two depthwise separable convolutional layers with stride 2 and kernel size 3×3;

[0094] Two residual blocks with stride 1;

[0095] The standard convolution layer, the depth-separable convolution layer, and the residual block are connected in series, and the output of the last residual block is the extracted style feature.

[0096] See also Figure 3 As shown, in one embodiment, the style modulator includes: a weight sub-network ξ w and bias subnetwork ξ b ;

[0097] The weight subnetwork ξ w include:

[0098] A standard convolutional layer with a stride of 1 and a kernel size of 1×1;

[0099] A global pooling layer;

[0100] The bias subnetwork ξ b include:

[0101] A standard convolutional layer with a stride of 1 and a kernel size of 1×1;

[0102] A global pooling layer.

[0103] See also Figure 3 As shown, in one embodiment, the style modulator uses the following algorithm to transform the style feature f s Converted into style modulation signal m s :

[0104] m s =(μ(f s ),σ(f s ),ξ w (f s ),ξ b (f s ))

[0105] Where:

[0106] μ(f s ) is the style feature f s The mean of

[0107] σ(f s ) is the style feature f s The standard deviation of

[0108] ξ w (f s ) is to use the weight sub-network ξ w The extracted weight parameters;

[0109] ξ b (f s ) is to use the bias subnetwork ξ b Extracted bias parameters.

[0110] See also Figure 3 As shown, in one embodiment, the micro decoder includes:

[0111] Two residual blocks with stride 1;

[0112] Two depthwise separable convolutional layers with upsampling operations and kernel size of 3×3;

[0113] A standard convolutional layer with a stride of 1 and a kernel size of 9×9;

[0114] The residual block, the depth-separable convolutional layer and the standard convolutional layer are connected in series, and the output of the last standard convolutional layer is the image after style transfer.

[0115] See also Figure 3 As shown, in one embodiment, the micro decoder uses a dual-path modulation method combining feature modulation FeatMod and network filter modulation FilterMod to process the content feature f c and style modulation signal m s ;

[0116] The algorithm of the feature modulation FeatMod is as follows:

[0117]

[0118] Where:

[0119] σ(f s ) is the style feature f s The standard deviation of

[0120] μ(f s ) is the style feature f s The mean of

[0121] σ(f c ) is the content feature f c The standard deviation of

[0122] μ(f c ) is the content feature f c The mean of

[0123] The algorithm of the network filter modulation FilterMod is as follows:

[0124] FilterMod(D,m s )=ResBlock(f c ,(ξ w (f s ),ξ b (f s )))

[0125] =Conv(Relu(Conv(f c ,(ξ w (f s ),ξ b (f s )))),(ξ w (f s ),ξ b (f s )))+f c

[0126] Where:

[0127] D represents the micro decoder;

[0128] ResBlock represents a residual block in the micro-decoder;

[0129] Conv is the convolutional layer in the residual block in the micro-decoder;

[0130] Relu is the activation layer in the residual block in the micro decoder.

[0131] The micro decoder reads the content feature f c and the style modulation signal m s The style modulation signal m is used by combining the feature modulation FeatMod and the network filter modulation FilterMod. s Modulate the content feature f c , the feature modulation FeatMod only uses the style modulation signal m s μ(f s ) and σ(f s ) signal; the network filter modulation FilterMod only uses the style modulation signal m s ξ w (f s ) and ξ b (f s) signal to modulate the filters of the convolutional layers in the residual blocks in the micro-decoder.

[0132] In terms of quality of the style-migrated image obtained by the real-time style migration of the image provided in the present application, due to the use of dual-path modulation of the style signal and contrast learning training, the quality of the style migration will not be significantly reduced.

[0133] Furthermore, since it is easier to operate on features than on network filters on existing deep learning platforms, the following equivalent method can be used to modulate the filter of the convolution layer in the residual block in the micro decoder according to the allocation rate and associative law of convolution:

[0134]

[0135] In the formula, represents a convolution filter, * represents a scalar product operation, Represents a convolution operation.

[0136] When training the micro content encoder, the micro style encoder, the style modulator and the micro decoder, three losses, namely content loss, style loss and style signal contrast loss, are used together for training.

[0137] Specifically, the content loss The definition is as follows:

[0138]

[0139] Where:

[0140] I cs A style transfer result image output by the micro decoder;

[0141] I c is the content image;

[0142] φ i is a layer of the VGG-19 network pre-trained on ImageNet;

[0143] L c A network layer that uses content loss calculation, such as the {Relu4_1} layer.

[0144] Note that although a large pre-trained VGG-19 network is still used when calculating the content loss, this network is discarded after the model training is completed, so no large pre-trained network is involved in the testing phase.

[0145] The style loss The definition is as follows:

[0146]

[0147] Where:

[0148] I cs A style transfer result image output by the micro decoder;

[0149] I s is the style image;

[0150] φ i is a layer of the VGG-19 network pre-trained on ImageNet;

[0151] μ represents the mean;

[0152] σ represents the standard deviation;

[0153] L s The network layer that uses style loss calculation, such as {Relu1_1, Relu2_1, Relu3_1, Relu4_1} layer.

[0154] Note that although a large pre-trained VGG-19 network is still used to calculate the style loss, this network is discarded after the model training is completed, so no large pre-trained network is involved in the testing phase.

[0155] The style signal contrast loss The definition is as follows:

[0156]

[0157] Where:

[0158] N represents the number of samples in a training batch, that is, each training batch has N content images and N style images, which will generate N style transfer result images.

[0159] The i-th style transfer result image is generated by using the i-th content image as the content and the i-th style image as the style. is the style modulation signal extracted from the i-th style transfer result image through the micro-style encoder and style modulator. is the style modulation signal extracted from the i-th style image through the micro-style encoder and style modulator; is the style modulation signal extracted from the j-th style image through the micro-style encoder and style modulator.

[0160] When training the content loss, the style loss and the style signal contrast loss, MS-COCO may be used as a content image dataset and WikiArt may be used as a style image dataset.

[0161] The present application also provides an image real-time style transfer device, comprising:

[0162] The first module is used to extract content features f of the content image using a micro content encoder c ;

[0163] The second module is used to extract the style features f of the style image using a micro-style encoder s ;

[0164] The third module is used to use the style modulator to transform the style feature f s Converted into style modulation signal m s ;

[0165] The fourth module is used to process the content feature f using a micro decoder c and style modulation signal m s , and get the image after style transfer.

[0166] For the functional limitations of each module, please refer to the limitations of the image real-time style transfer method above, which will not be repeated here. Each module in the above-mentioned image real-time style transfer device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0167] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a real-time image style transfer method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0168] Those skilled in the art will understand that Figure 5The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0169] The present application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the image real-time style transfer method when executing the computer program.

[0170] The present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method for real-time image style transfer are implemented.

[0171] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0172] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0173] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A real-time image style transfer method, characterized in that: include: Step 1: Use the micro content encoder to extract the content features f of the content image c ; Step 2: Use the micro-style encoder to extract the style features f of the style image s ; Step 3: Use the style modulator to transform the style feature f s Converted into style modulation signal m s ; The style modulator includes: a weight sub-network ξ w and bias subnetwork ξ b ; The weight subnetwork ξ w include: A standard convolutional layer with a stride of 1 and a kernel size of 1×1; A global pooling layer; The bias subnetwork ξ b include: A standard convolutional layer with a stride of 1 and a kernel size of 1×1; A global pooling layer; The style modulator uses the following algorithm to transform the style feature f s Converted into style modulation signal m s : m s =(μ(f s ),σ(f s ),ξ w (f s ),ξ b (f s )) Where: μ(f s ) is the style feature f s The mean of σ(f s ) is the style feature f s The standard deviation of ξ w (f s ) is to use the weight sub-network ξ w The extracted weight parameters; ξ b (f s ) is to use the bias subnetwork ξ b Extracted bias parameters; Step 4: Process the content feature f using a micro decoder c and style modulation signal m s , get the image after style transfer; The micro decoder processes the content feature f by using a dual-path modulation method combining feature modulation FeatMod and network filter modulation FilterMod. c and style modulation signal m S ; The algorithm of the feature modulation FeatMod is as follows: Where: σ(f s ) is the style feature f s The standard deviation of μ(f s ) is the style feature f s The mean of σ(f c ) is the content feature f c The standard deviation of μ(f c ) is the content feature f c The mean of The algorithm of the network filter modulation FilterMod is as follows: FilterMod(D,ms)=ResBlock(f c ,(ξ w (f s ),ξ b (f s ))) =Conv(Relu(Conv(f c ,(ξ w (f s ),ξ b (f s )))),(ξ w (f s ),ξ b (f s )))+f c Where: D represents the micro decoder; ResBlock represents a residual block in the micro-decoder; Conv is the convolutional layer in the residual block in the micro-decoder; Relu is the activation layer in the residual block in the micro decoder.

2. The method for real-time image style transfer according to claim 1, characterized in that: The micro content encoder comprises: A standard convolutional layer with a stride of 1 and a kernel size of 9×9; Two depthwise separable convolutional layers with stride 2 and kernel size 3×3; Two residual blocks with stride 1; The standard convolution layer, the depth-separable convolution layer and the residual block are connected in series, and the output of the last residual block is the extracted content feature.

3. The method for real-time image style transfer according to claim 1, characterized in that: The micro-style encoder comprises: A standard convolutional layer with a stride of 1 and a kernel size of 9×9; Two depthwise separable convolutional layers with stride 2 and kernel size 3×3; Two residual blocks with stride 1; The standard convolution layer, the depth-separable convolution layer, and the residual block are connected in series, and the output of the last residual block is the extracted style feature.

4. The method for real-time image style transfer according to claim 1, characterized in that: The micro decoder comprises: Two residual blocks with stride 1; Two depthwise separable convolutional layers with upsampling operations and kernel size of 3×3; A standard convolutional layer with a stride of 1 and a kernel size of 9×9; The residual block, the depth-separable convolutional layer and the standard convolutional layer are connected in series, and the output of the last standard convolutional layer is the image after style transfer.

5. The method for real-time image style transfer according to any one of claims 1 to 4, characterized in that: When training the micro content encoder, the micro style encoder, the style modulator and the micro decoder, three losses, namely, content loss, style loss and style signal contrast loss, are used for training; The style signal contrast loss The definition is as follows: Where: N represents the number of samples in a training batch; is the style modulation signal extracted from the i-th style transfer result image through the micro-style encoder and style modulator; is the style modulation signal extracted from the i-th style image through the micro-style encoder and style modulator; is the style modulation signal extracted from the i-th style image through the micro-style encoder and style modulator.

6. A real-time image style transfer device, characterized in that: include: The first module is used to extract content features f of the content image using a micro content encoder c ; The second module is used to extract the style features f of the style image using a micro-style encoder s ; The third module is used to use the style modulator to transform the style feature f s Converted into style modulation signal m S ; The style modulator includes: a weight sub-network ξ w and bias subnetwork ξ b ; The weight subnetwork ξ w include: A standard convolutional layer with a stride of 1 and a kernel size of 1×1; A global pooling layer; The bias subnetwork ξ b include: A standard convolutional layer with a stride of 1 and a kernel size of 1×1; A global pooling layer; The style modulator uses the following algorithm to transform the style feature f s Converted into style modulation signal m s : m s =(μ(f s ),σ(f s ),ξ w (f s ),ξ b (f s )) Where: μ(f s ) is the style feature f s The mean of σ(f s ) is the style feature f s The standard deviation of ξ w (f s ) is to use the weight sub-network ξ w The extracted weight parameters; ξ b (f s ) is to use the bias subnetwork ξ b Extracted bias parameters; The fourth module is used to process the content feature f using a micro decoder c and style modulation signal m s , get the image after style transfer; The micro decoder processes the content feature f by using a dual-path modulation method combining feature modulation FeatMod and network filter modulation FilterMod. c and style modulation signal m s ; The algorithm of the feature modulation FeatMod is as follows: Where: σ(f s ) is the style feature f s The standard deviation of μ(f s ) is the style feature f s The mean of σ(f c ) is the content feature f c The standard deviation of μ(f c ) is the content feature f c The mean of The algorithm of the network filter modulation FilterMod is as follows: FilterMod(D,m s )=ResBlock(f c ,(ξ w (f s ),ξ b (f s ))) =Conv(Relu(Conv(f c ,(ξ w (f s ),ξ b (f s )))),(ξ w (f s ),ξ b (f s )))+f c Where: D represents the micro decoder; ResBlock represents a residual block in the micro-decoder; Conv is the convolutional layer in the residual block in the micro-decoder; Relu is the activation layer in the residual block in the micro decoder.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method for real-time image style transfer according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for real-time image style transfer according to any one of claims 1 to 5 are implemented.

Images