Image style migration method, image style migration device and storage medium
By introducing enhanced autocoding loss and feature fusion network, the problem of high cost and insufficient flexibility of image style transfer in the prior art is solved, and efficient arbitrary style transfer and image quality improvement are achieved.
Patent Information
- Application Number
- CN202310632025.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-30
- Publication Date
- 2024-12-06
AI Technical Summary
The existing image style migration technology has problems such as high computational cost, slow generation speed and insufficient flexibility, and cannot effectively extract high-level semantic features of the image, resulting in low quality of the generated target image.
Using a fast neural network method based on offline model optimization, the image style migration model is trained by introducing enhanced autocoding losses, extracting style features and content features, and generating target migration images through feature fusion networks, and optimizing the decoder's decoding capability and feature restoration ability.
It reduces the calculation cost, improves the target image generation efficiency, realizes image style migration in any style, and improves the image quality of the target migration image, retains content features, and reduces edge glitches.
Smart Images

Figure CN119090703A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of image processing, and in particular to an image style transfer method, an image style transfer device, and a storage medium. Background Art
[0002] Image style transfer is an image processing method that renders the semantic content of an image in different styles. In recent years, with the widespread application of deep neural networks in the field of visual perception (such as object and face recognition), it has also performed well in the field of style transfer. Neural network models have good feature extraction capabilities and can extract high-level features of images, such as semantic information of images, which is the basis of style transfer.
[0003] In related technologies, a neural network model is trained based on the style loss between the style sample image and the transfer sample image, and the content loss between the content sample image and the transfer sample image. The neural network model is used to extract content features and style features of different images, and the extracted content features and style features are fused. Summary of the invention
[0004] In order to overcome the problems existing in the related art, the present disclosure provides an image style transfer method, an image style transfer device and a storage medium.
[0005] According to a first aspect of an embodiment of the present disclosure, a method for image style transfer is provided, comprising: acquiring a first image and a second image, wherein the first image is an image for providing the style of a target transfer image, and the second image is an image for providing the content of a target transfer image; inputting the first image and the second image into a first encoder of an image style transfer model, extracting style features of the first image through the first encoder, and extracting content features of the second image; fusing the style features and the content features through a feature fusion network of the image style transfer model to obtain fused data, decoding the fused data through a decoder of the image style transfer model to obtain a target transfer image, and extracting transfer features of the target transfer image through a second encoder; the image style transfer model is pre-trained based on a target loss; the target loss is determined based on an enhanced autoencoder loss, and the enhanced autoencoder loss is determined based on a content loss generated by decoding a second training image by the decoder, and the second training image is an image for providing predicted transfer image content when training the image style transfer model.
[0006] In one embodiment, the content loss generated by the decoding process of the second training image by the decoder is: a loss determined based on the difference between the second training image and a decoded prediction image, and the decoded prediction image is an image obtained by decoding the second training image by the first encoder.
[0007] In one embodiment, the image style transfer model is pre-trained based on a target loss in the following manner: a second training image is input into a first encoder of the image style transfer model, and content features of the second training image are extracted by the first encoder; the content features of the second training image are input into a decoder of the image style transfer model, and the decoder decodes the content features of the second training image to obtain a decoded prediction image; based on the difference between the decoded prediction image and the second training image, an enhanced autoencoder loss is determined; based on the enhanced autoencoder loss, a target loss is determined, and the image style transfer model is trained based on the target loss.
[0008] In one embodiment, the method further includes: inputting a first training image into a first encoder of an image style transfer model, extracting style features of the first training image through the first encoder; fusing the style features of the first training image and the content features of the second training image through a feature fusion network of the image style transfer model to obtain predicted transfer features, and decoding the predicted transfer features through a decoder of the image style transfer model to obtain a predicted transfer image; determining a style loss based on the predicted transfer features and the style features of the first training image, and determining a content loss based on the predicted transfer features and the content features of the second training image; determining a target loss based on the enhanced autoencoder loss includes: determining a target loss based on the style loss, the content loss, and the enhanced autoencoder loss.
[0009] In one embodiment, the target loss is determined based on the style loss, the content loss and the enhanced autoencoder loss, including: weighting the style loss based on a first parameter to obtain a weighted style loss, weighting the content loss based on a second parameter to obtain a weighted content loss, and weighting the enhanced autoencoder loss based on a third parameter to obtain a weighted enhanced autoencoder loss; and determining the sum of the weighted style loss, the weighted content loss and the weighted enhanced autoencoder loss as the target loss.
[0010] In one embodiment, determining the style loss based on the predicted transfer features and the style features of the first training image includes: determining a first mean square error between the style features of the first training image and the predicted transfer features; and determining the first mean square error as the style loss of the style transfer model.
[0011] In one embodiment, determining the content loss based on the predicted transfer feature and the content feature of the second training image includes: aligning the predicted transfer feature with the content feature of the second training image; determining a second mean square error between features at the same level after the predicted transfer feature is aligned with the content feature of the second training image; and determining the second mean square error as the content loss of the style transfer model.
[0012] According to a second aspect of an embodiment of the present disclosure, an image style transfer device is provided, comprising: an acquisition unit, configured to acquire a first image and a second image, wherein the first image is an image for providing a style of a target transfer image, and the second image is an image for providing content of a target transfer image; an extraction unit, configured to input the first image and the second image into a first encoder of an image style transfer model, extract style features of the first image through the first encoder, and extract content features of the second image; a fusion unit, configured to fuse the style features and the content features through a feature fusion network of the image style transfer model to obtain fused data, decode the fused data through a decoder of the image style transfer model to obtain a target transfer image, and extract transfer features of the target transfer image through a second encoder; a training unit, configured to pre-train the image style transfer model based on a target loss; the target loss is determined based on an enhanced autoencoder loss, and the enhanced autoencoder loss is determined based on a content loss generated by decoding a second training image through the decoder, and the second training image is an image for providing predicted transfer image content when training the image style transfer model.
[0013] In one embodiment, the content loss generated by the decoding process of the second training image by the decoder is: a loss determined based on the difference between the second training image and a decoded prediction image, and the decoded prediction image is an image obtained by decoding the second training image by the first encoder.
[0014] In one embodiment, the image style transfer model is pre-trained based on a target loss by a training unit in the following manner: a second training image is input into a first encoder of the image style transfer model, and content features of the second training image are extracted by the first encoder; the content features of the second training image are input into a decoder of the image style transfer model, and the decoder decodes the content features of the second training image to obtain a decoded prediction image; based on the difference between the decoded prediction image and the second training image, an enhanced autoencoder loss is determined; based on the enhanced autoencoder loss, a target loss is determined, and the image style transfer model is trained based on the target loss.
[0015] In one embodiment, the training unit is also used to: input a first training image into a first encoder of an image style transfer model, and extract style features of the first training image through the first encoder; fuse the style features of the first training image and the content features of the second training image through a feature fusion network of the image style transfer model to obtain predicted transfer features, and decode the predicted transfer features through a decoder of the image style transfer model to obtain a predicted transfer image; determine a style loss based on the predicted transfer features and the style features of the first training image, and determine a content loss based on the predicted transfer features and the content features of the second training image; determining a target loss based on the enhanced autoencoder loss includes: determining a target loss based on the style loss, the content loss, and the enhanced autoencoder loss.
[0016] In one embodiment, the training unit determines the target loss based on the style loss, the content loss and the enhanced autoencoder loss in the following manner: weighting the style loss based on a first parameter to obtain a weighted style loss, weighting the content loss based on a second parameter to obtain a weighted content loss, and weighting the enhanced autoencoder loss based on a third parameter to obtain a weighted enhanced autoencoder loss; and determining the sum of the weighted style loss, the weighted content loss and the weighted enhanced autoencoder loss as the target loss.
[0017] In one embodiment, the training unit determines the style loss based on the predicted transfer features and the style features of the first training image in the following manner: determining a first mean square error between the style features of the first training image and the predicted transfer features; and determining the first mean square error as the style loss of the style transfer model.
[0018] In one embodiment, the training unit determines the content loss based on the predicted transfer features and the content features of the second training image in the following manner: aligning the predicted transfer features with the content features of the second training image; determining a second mean square error between features at the same level after the predicted transfer features are aligned with the content features of the second training image; and determining the second mean square error as the content loss of the style transfer model.
[0019] According to a third aspect of an embodiment of the present disclosure, there is provided an image style transfer apparatus, comprising: a processor; and a memory for storing processor executable instructions; wherein the processor is configured to: execute the image style transfer method described in the first aspect or any one of the embodiments of the first aspect.
[0020] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions in the storage medium are executed by a processor, the processor is enabled to execute the image style transfer method described in the first aspect or any one of the embodiments of the first aspect.
[0021] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: determining an enhanced autoencoder loss based on the content loss generated by the decoding process of the content image used for training the model after the decoder, determining a loss function based on the enhanced autoencoder loss, and training a style transfer model based on the loss function. The style features and content features are fused through the style transfer model to obtain a target transfer image. Through the present disclosure, an enhanced autoencoder loss is introduced in the training stage of the style transfer model to improve the decoding ability and feature restoration ability of the model decoder, and to improve the image quality of the target transfer image.
[0022] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0024] Figure 1 It is a schematic diagram of an application scenario of an image style transfer method according to an exemplary embodiment of the present disclosure.
[0025] Figure 2 It is a schematic diagram showing the application effect of an image style transfer method according to an exemplary embodiment of the present disclosure.
[0026] Figure 3 The figure is a flowchart of a method for image style transfer according to an exemplary embodiment.
[0027] Figure 4 The figure is a flowchart of a method for training an image style transfer model according to an exemplary embodiment.
[0028] Figure 5 The invention is a flow chart showing a method for determining target loss according to an exemplary embodiment.
[0029] Figure 6 The present invention is a flow chart showing a method for determining target loss according to yet another exemplary embodiment.
[0030] Figure 7 The invention is a flowchart showing a method for determining style loss according to an exemplary embodiment.
[0031] Figure 8 The invention is a flowchart showing a method for determining content loss according to an exemplary embodiment.
[0032] Fig. 9 It is a schematic diagram showing the application effect of an image style transfer method according to another exemplary embodiment of the present disclosure.
[0033] Fig.10 The figure is a block diagram of an image style transfer method according to an exemplary embodiment.
[0034] Fig.11 The figure is a block diagram of an image style transfer apparatus according to an exemplary embodiment.
[0035] Fig.12 The invention is a block diagram of a device for image style transfer according to an exemplary embodiment. DETAILED DESCRIPTION
[0036] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.
[0037] The image style transfer method provided by the embodiment of the present disclosure is applied to fuse the style features and content features of different images to obtain a scene of a style-transferred image.
[0038] Image style transfer is an image processing method that uses different styles to render the semantic content of an image. Traditional non-parametric style transfer methods can only perform texture synthesis by extracting low-level features of the image (such as the color, texture, and shape of the image), but cannot extract high-level features of the image (such as the semantic information of the image). They can only perform simple rendering of the image and cannot change the artistic style of the image itself. Figure 1 Schematic diagram and application scenario of image style transfer method in Figure 2 As shown in the schematic diagram of the application effect of the image style transfer method, image style transfer can respectively extract the content features (deep features) of content image A and the style features (shallow features) of style image B, and obtain the style transfer image C after feature fusion. While retaining the image content and the original character features in the image, it changes the lines and texture of the image and transforms the image into a different artistic style.
[0039] In related technologies, image style transfer models are based on neural network training. At present, with the widespread application of deep neural networks in other visual perception fields (such as object and face recognition), deep neural networks have good feature extraction capabilities and can extract rich semantic information. Semantic information containing image content features is the basis of style transfer, so neural network models also perform well in the field of style transfer.
[0040] The current neural style transfer methods are mainly divided into two categories according to the model method: the slow neural network method based on online image optimization and the fast neural network method based on model optimization. The following describes the slow neural network method based on online image optimization and the fast neural network method based on model optimization respectively:
[0041] 1. Slow neural network based on online image optimization: The basic idea of online image optimization is to extract content features and style features from the content image and style image respectively, and recombining the content features and style features into a target image, i.e., a style transfer image, and then reconstructing the target image online. The basis for online iteration and image reconstruction is the difference between the target image and the content image, and the difference between the target image and the style image. The functional relationship that characterizes the above differences is as follows:
[0042] L total (I c ,I s ,I o )=αL c (I c ,I o )+βL s (I s ,I o )
[0043] Among them, I o is the target image, I c is the content image, I s is the style image L c Represents the content loss between the target image and the content image, characterizing the difference between the target image and the content image; L s Represents the style loss between the target image and the style image, characterizing the difference between the target image and the style image, L total is the loss function used for online iteration and reconstruction of images. α and β are hyperparameters for adjusting the balance between content features and style features. In general, the content loss is defined as the Euclidean distance between the last layer of features extracted from the content image and the target image through the VGG network. The style loss is defined as the Euclidean distance of the Gram matrix between the features extracted from the style image and the target image through the VGG network.
[0044] For a deep network, the shallow network extracts shallow features such as color, texture, etc., while the deep network extracts deep features such as semantic content information. Therefore, style loss often compares shallow network features, while content loss compares deep network features. The neural network used for style transfer generally uses the VGG-19 network. When using the VGG-19 network to extract style features and content features, it is generally chosen to take the first-layer feature map of the 1st to 5th stage as the content loss, and the second-layer feature map of the 4th stage as the style loss.
[0045] It is understandable that the slow neural network based on online image optimization needs to reconstruct the target image through online calculation and online iteration, which requires a long time of calculation and consumes a lot of computing cost. It has the problems of high computing cost and slow generation speed.
[0046] 2. Fast neural network method based on offline model optimization
[0047] The basic idea of offline model optimization is to use fast image reconstruction based on offline model optimization to reconstruct the stylized result to solve the problem of target image speed and iterative computation cost. That is, for one or more style images, through a set of content images I c A feedforward network is optimized, and the target image is directly generated through the feedforward network. According to the number of styles that the feedforward network can generate, the fast neural network method based on offline model optimization can be divided into single model single style (Per-Style-Per-Model), single model multiple styles (Multiple-Style-Per-Model) and single model multiple styles (Multiple-Style-Per-Model):
[0048] (1) Single model single style: The single model single style approach can only generate target images of one style through one model. To obtain target images of different styles, a separate generative network must be trained for each specific style image. However, in reality, different art paintings also have similar painting strokes that differ only in the color palette. Therefore, it is redundant to train a separate network for each style. The single model single style approach has the problem of insufficient flexibility.
[0049] (2) Single model, multiple styles: Single model, multiple styles improves the flexibility of the model by further merging multiple styles into a single model. There are two main processing methods: One is to bind a small number of parameters in the network to a single specific style to produce a more applicable model, but as the number of learned styles increases, the size of the model usually increases. The second is to use a single network, but the parameter input becomes a combination of style and content. The limitations of the single model network are solved by fully exploring the capabilities of a single network and combining content and style into the network for style recognition. Single model, multiple styles still have the problem of insufficient flexibility. The styles supported by the single model, multiple style method are still limited, and it is impossible to achieve style transfer of arbitrary image styles.
[0050] (3) Single model arbitrary style: The single model arbitrary style approach aims to extract a universal style transfer network. The input is an arbitrary content image and an arbitrary style map. The style features and content features of the sample image are extracted on a pre-trained network such as VGG-19. The style transfer model is obtained through training. The style transfer model generates a target style image based on the extracted style features and content features.
[0051] In view of the high computational cost and slow generation speed of slow neural networks based on online image optimization, as well as the lack of flexibility and limited supported styles for migration of single-model single style and single-model multiple styles. The present disclosure proposes an image style transfer method, which adopts a single-model arbitrary style method in a fast neural network method based on offline model optimization, which can reduce computational costs, improve target image generation efficiency, and realize image style transfer of arbitrary styles. The enhanced autoencoder loss is determined by the difference between the training image and the decoded image, and the style transfer model is trained by the enhanced autoencoder loss, and then the target migration image is obtained by the style transfer model. By introducing a strong autoencoder loss, the feature restoration capability of the encoder is optimized, and the overall quality of the target migration image is improved.
[0052] Figure 3 is a flowchart of an image style transfer method according to an exemplary embodiment. Figure 3 As shown, the method includes steps S101 to S103.
[0053] In step S101 , a first image and a second image are acquired.
[0054] The first image is an image used to provide the style of the target migration image, and the second image is an image used to provide the content of the target migration image.
[0055] In step S102, the first image and the second image are input into a first encoder of an image style transfer model, and the style features of the first image are extracted by the first encoder, and the content features of the second image are extracted.
[0056] In step S103, the style features and content features are fused through the feature fusion network of the image style transfer model to obtain fused data, the fused data is decoded by the decoder of the image style transfer model to obtain a target transfer image, and the transfer features of the target transfer image are extracted by the second encoder.
[0057] The image style transfer model is pre-trained based on a target loss. The target loss is determined based on an enhanced autoencoder loss, and the enhanced autoencoder loss is determined based on a content loss generated by a second training image after being decoded by a decoder. The second training image is an image used to provide predicted content of a transfer image when training an image style transfer model.
[0058] In the disclosed embodiment, the style transfer model includes an encoder for extracting style features and content features, a feature fusion network for fusing style features and content features, and a decoder for decoding fused data. There is a difference in content between the decoded target transfer image and the content image. The difference is caused by the content loss caused by feature fusion on the one hand and the content loss caused by the decoder decoding process on the other hand. By enhancing the autoencoder loss, the content loss caused by the decoder decoding can be reflected. By training the image style transfer model with enhanced autoencoder loss, the content feature restoration capability of the decoder in the image style transfer model can be optimized, thereby improving the image quality of the target transfer image obtained by the image style transfer model.
[0059] In the embodiment of the present disclosure, the second decoder is a decoder different from the first decoder, and is used to obtain the migration feature of the target migration image.
[0060] In the disclosed embodiment, a decoded image is obtained through a decoder, and an enhanced autoencoder loss is determined by the difference between the content image and the decoded image. A style transfer model is trained by enhancing the autoencoder loss. The style features and style features of the style image and the content image are extracted by the style transfer model, and the style features and the transfer features are fused to obtain the transfer features, and the target transfer image is obtained after decoding. Among them, the enhanced autoencoder loss can reflect the feature restoration capability of the encoder, and training the style transfer model based on the autoencoder loss can improve the feature restoration capability of the encoder. In addition, the target transfer image obtained based on the style transfer model retains more content features, enhances the lines of the target transfer image, reduces the burrs of the target transfer image, and improves the quality of the target transfer image.
[0061] In the embodiment of the present disclosure, the target migration image contains both the features of the style image and the features of the content image. Based on the target migration image and the content image, it is impossible to obtain the content loss generated by the decoding process of the training image by the decoder. Therefore, the decoder needs to decode the content features separately, and determine the content loss generated by the decoding process of the training image by the decoder based on the decoding result and the original content image. The following embodiment of the present disclosure describes a method for determining the content loss generated by the decoding process of the training image by the decoder.
[0062] In one implementation of the disclosed embodiment, the content loss generated by the second training image after decoding by the decoder is: a loss determined based on the difference between the second training image and a decoded prediction image, where the decoded prediction image is an image obtained by decoding the second training image by the first encoder.
[0063] In the disclosed embodiment, after the decoder extracts the content features of the content image for training, while inputting the content features into the feature fusion network, a data branch is also derived to transmit the style features to the decoder of the image style transfer model to generate a self-decoded image of the content image. The content loss generated by the decoder decoding the training image is determined based on the difference between the content image and the self-decoded image.
[0064] In the embodiment of the present disclosure, a part of the VGG19 network structure is used to construct an encoder to extract image features. A corresponding decoder is constructed to restore image features. The decoder structure of the style transfer model in the present disclosure is shown in the following table:
[0065]
[0066] Among them, "conv, 3, 1, 64, relu, reflect_pad" correspond to "convolution operation, convolution kernel size is 3, step size is 1, number of convolution kernels is 64, activation function is ReLU, padding method is reverse mapping padding", if it is None, the corresponding part will not be operated. And maxpool corresponds to maximum pooling.
[0067] The decoder structure of the style transfer model in this disclosure is shown in the following table:
[0068]
[0069] Among them, "conv, 3, 1, 64, relu, reflect_pad" correspond to "convolution operation, convolution kernel size is 3, step size is 1, number of convolution kernels is 64, activation function is ReLU, padding method is reverse mapping padding". Upsample is upsampling, and nearest is the nearest neighbor difference.
[0070] In the embodiments of the present disclosure, the enhanced autoencoder loss is a parameter that reflects the content feature restoration capability of the decoder of the style transfer model, so it is necessary to directly encode and decode the content image, and determine the enhanced autoencoder loss based on the original content image and the content image after encoding and decoding. In addition, the enhanced autoencoder loss can only reflect the content difference between the content image before and after decoding. When training the model, it is also necessary to consider the difference between the target transfer image and the style image, as well as the difference between the target transfer image and the content image, and determine the target loss based on the above differences, and then train the model. The following embodiments of the present disclosure illustrate the method for determining the enhanced autoencoder loss and then determining the target loss for model training.
[0071] Figure 4 is a flowchart of a method for training an image style transfer model according to an exemplary embodiment. Figure 4 As shown, the method includes steps S201 to S204.
[0072] In step S201, the second training image is input into the first encoder of the image style transfer model, and the content features of the second training image are extracted by the first encoder.
[0073] In step S202, the content features of the second training image are input into a decoder of the image style transfer model, and the decoder decodes the content features of the second training image to obtain a decoded prediction image.
[0074] In the disclosed embodiment, the image autoencoding loss is the image content loss generated by the decoder of the style transfer model before and after decoding. Therefore, in the process of determining the enhanced autoencoding loss, after the content features of the content image used for training, i.e., the second training image, are extracted by the encoder, a new data tributary is introduced, so that the content features flow directly into the decoder, are decoded by the decoder, and a self-decoded image, i.e., a decoded predicted image, is generated. The autoencoding loss can be determined by the self-decoded image and the content image used for training.
[0075] In step S203, an enhanced autoencoding loss is determined based on the difference between the decoded prediction image and the second training image.
[0076] In the embodiment of the present disclosure, after the decoded prediction image is obtained, the content features of the decoded prediction image are extracted, and the enhanced autoencoder loss is determined based on the content features of the content image used for training and the decoded prediction image. The enhanced autoencoder loss is the Euclidean distance between the above two content features. The enhanced autoencoder loss is defined as:
[0077] L e =||d(c)-c||2
[0078] Among them, L eTo enhance the autoencoding loss, c is the content feature extracted by the encoder, d is the decoding operation, and d(c) decodes the content feature of the predicted image.
[0079] In step S204, a target loss is determined based on the enhanced autoencoder loss, and the image style transfer model is trained based on the target loss.
[0080] In the disclosed embodiment, the extracted content features are decoded by a decoder to obtain a decoded prediction image, and an enhanced autoencoder loss is determined based on the content features of the decoded prediction image and the content features of the training content image. Then, a target loss is determined based on the strong autoencoder loss, and a style transfer model is trained based on the target loss. The content feature restoration capability of the decoder of the style transfer model can be improved.
[0081] In the embodiments of the present disclosure, the enhanced autoencoder loss is only a parameter used for style transfer model training, which reflects the content difference between the decoder before and after decoding. In addition, the difference between the target transfer image and the style image, as well as the difference between the target transfer image and the content image, must be considered to determine the corresponding image loss, and determine the target loss including various losses. The following embodiments of the present disclosure illustrate the method for determining the target loss.
[0082] Figure 5 is a flow chart showing a method for determining target loss according to an exemplary embodiment. Figure 5 As shown, the method includes steps S301 to S303.
[0083] In step S301, a first training image is input into a first encoder of an image style transfer model, and style features of the first training image are extracted by the first encoder.
[0084] In step S302, the style features of the first training image and the content features of the second training image are fused through a feature fusion network of the image style transfer model to obtain a predicted transfer feature, and the predicted transfer feature is decoded through a decoder of the image style transfer model to obtain a predicted transfer image.
[0085] In step S303, a style loss is determined based on the predicted transfer feature and the style feature of the first training image, and a content loss is determined based on the predicted transfer feature and the content feature of the second training image.
[0086] Wherein, determining the target loss based on the enhanced autoencoder loss includes: determining the target loss based on the style loss, the content loss and the enhanced autoencoder loss.
[0087] In the disclosed embodiment, a style loss is determined based on the style features of a style training image, i.e., a first training image, and the predicted migration features of a target migration image, and the style difference between the style training image and the target migration image is reflected through the style loss. A content loss is determined based on the content features of a content training image, i.e., a first training image, and the predicted migration features of a target migration image, and the content difference between the content training image and the target migration image is reflected through the content loss. The target loss for model training is determined through the content loss and the style loss, combined with the enhanced autoencoder loss that reflects the difference before and after decoding by the decoder. The image migration model can be trained so that it can well balance the content features and style features of the target migration image during the image style migration process, and improve the content feature restoration capability of the decoder.
[0088] In the embodiment of the present disclosure, based on the influence of content loss, style loss and enhanced autoencoder loss on model training, different weights need to be applied to each loss to determine the target loss. The following is an example of further illustrating the method of determining the target loss.
[0089] Figure 6 is a flow chart of a method for determining target loss according to another exemplary embodiment. Figure 6 As shown, the method includes step S401 to step S402.
[0090] In step S401, the style loss is weighted based on the first parameter to obtain a weighted style loss, the content loss is weighted based on the second parameter to obtain a weighted content loss, and the enhanced autoencoder loss is weighted based on the third parameter to obtain a weighted enhanced autoencoder loss.
[0091] In step S402, the sum of the weighted style loss, the weighted content loss and the weighted enhanced autoencoder loss is determined as the target loss.
[0092] In the embodiment of the present disclosure, by weighting the style loss, enhanced autoencoder loss and content loss, the various losses are balanced, the target loss is determined, and the training effect of the target loss on the style transfer model is guaranteed. The objective function in the present disclosure is defined as:
[0093] L=γL c +α1L s +β1L e
[0094] Among them, L is the target loss, L c is the content loss, L s is the style loss, is the enhanced autoencoder loss, and γ, α1 and β1 are different weights.
[0095] In an exemplary embodiment of the present disclosure, γ is set to 1.0, α1 is set to 10.0, and β1 is set to 100.0. That is, the objective function is defined as:
[0096] L=L c +10L s +100L e
[0097] In the embodiment of the present disclosure, the style loss reflects the style difference between the style image used for training and the predicted transfer image. By obtaining the style loss and training the style transfer model based on the style loss, the style restoration capability of the style transfer model can be improved. The following embodiment of the present disclosure describes the method for determining the style loss.
[0098] Figure 7 is a flowchart of a method for determining style loss according to an exemplary embodiment. Figure 7 As shown, the method includes step S501 to step S502.
[0099] In step S501, a first mean square error between a style feature of a first training image and a predicted transfer feature is determined.
[0100] In step S502, the first mean square error is determined as the style loss of the style transfer model.
[0101] In the embodiment of the present disclosure, the content features and style features are fused through a feature fusion network. The mean and variance of the content features and style features are obtained respectively, and the style features are transferred to the content features using the mean and variance. The formula for feature fusion is as follows:
[0102]
[0103] Among them, AdaIN is a feature fusion algorithm, c is the content feature extracted by the encoder, s is the style feature extracted by the encoder, σ is the standard deviation function, and μ is the mean function.
[0104] In the embodiment of the present disclosure, the predicted migration feature of the predicted migration image is determined by combining the fused content feature and style feature with a trade-off parameter for controlling the degree of stylization. The formula for obtaining the predicted migration feature is as follows:
[0105] t=λAdaIN(c,s)+(1-λ)c
[0106] Among them, t is the predicted migration feature, AdaIN(c,s) is the fused content feature and style feature, and λ is the trade-off parameter.
[0107] In the disclosed embodiment, the style of the transferred image is characterized by the mean and variance corresponding to the transferred features, and the style of the style image is characterized by the mean and variance corresponding to the style features. The mean square error of the means and variances attributed to different features is used as the model style loss. The style loss is defined as:
[0108]
[0109] in, represents the i-th feature output by the encoder, l represents the number of feature layers output by the encoder, d represents the decoding operation, σ is the standard deviation function, μ is the mean function, t is the predicted migration feature, and s is the style feature.
[0110] In the embodiment of the present disclosure, the content loss reflects the content difference between the content image used for training and the predicted migration image. By obtaining the content loss and training the content migration model based on the content loss, the content restoration capability of the content migration model can be improved. The following embodiment of the present disclosure describes the method for determining the content loss.
[0111] Figure 8 is a flow chart showing a method for determining content loss according to an exemplary embodiment. Figure 8 As shown, the method includes steps S601 to S603.
[0112] In step S601, the predicted migration feature is aligned with the content feature of the second training image.
[0113] In step S602, a second mean square error between the features at the same level after the predicted migration features are aligned with the content features of the second training image is determined.
[0114] In step S603, the second mean square error is determined as the content loss of the style transfer model.
[0115] In the disclosed embodiment, the predicted migration features and content features obtained by the decoder are both multi-layer features, and features at different levels have the same or similar content features. Therefore, the migration features and content features are aligned to obtain content features and predicted migration features at the same level, thereby obtaining content loss.
[0116] It is understandable that the content features of the content image are extracted by the decoder, and the extracted content features will lose the style features of the image after passing through the pooling layer (maxpool). Compared with the general level, the deep features in the extracted multi-layer content features obtain the content loss, that is, extracting the content features at the deep level after the pooling layer will have a better effect.
[0117] In an exemplary embodiment of the present disclosure, the mean square error corresponding to the last layer of predicted migration features and content features extracted by the encoder is used as the content loss, and the content loss is defined as:
[0118]
[0119] Among them, c is the content feature, represents the last layer feature of the encoder output, and t is the predicted migration feature.
[0120] In the disclosed embodiment, the target loss is determined by enhancing the autoencoder loss, content loss and style loss, and then the style transfer model is trained based on the target loss. The style feature restoration capability of the decoder in the style transfer model can be improved, such as Fig. 9 The schematic diagram of the application effect of the image style transfer method is shown in FIG. 1 , where part A corresponds to the content image, part B corresponds to the style image, part C corresponds to the style transfer image obtained by the style transfer model in the related art, and part D corresponds to the style transfer image obtained by the style transfer model of the present disclosure. Fig. 9 The comparison between parts C and D shows intuitively that the style transfer model proposed in the present disclosure has a better ability to restore content features, with clearer lines and more detailed features. It can well capture details such as the edges of buildings and the contours of people's eyes, reduce edge burrs, and greatly improve the image quality.
[0121] In the embodiment of the present disclosure, when training the style transfer model, a preset content data set and style data set are used. During training, the image size is adjusted to a preset size, and the cropped image is data enhanced by a random cropping strategy, and a sub-image of the target cropping size is cropped from the original image used for training for training. Following the AdaIN scheme, the encoder part adopts the model parameters of the pre-trained VGG19, follows the data preprocessing of VGG19, and does not update the encoder parameters during the training process. The convolution kernel of the encoder is taken as the output feature of the encoder. During training, the batch size and the number of training times are set, an optimizer with a specific learning rate is used, and an inverse time-limited learning rate decay is used to make the final learning rate the target learning rate. For the test part of the style transfer model, the training network used in the present disclosure is a full convolution structure, and images of any resolution can be used for testing. The content map and style map are input into the encoder to obtain content and style features, and the AdaIN module (feature fusion network / feature interaction module) is used for feature interaction. Finally, the final predicted migration image is obtained through the decoder, and the degree of stylization can be controlled by adjusting the trade-off parameter λ.
[0122] In the present disclosure, Fig.10As shown in the block diagram of the image style transfer method in , when training the style transfer model, the encoder of the style transfer model extracts content features from the training content image and style features from the training style image. The content features and style features are fused through the feature fusion network to obtain fused data, and the target transfer image is obtained after the fused data is decoded by the decoder. The target transfer image is decoded by another decoder to obtain the target transfer features. While the fusion network is fusing the content features and style features, the content features extracted by the encoder are input into the decoder of the style transfer model to obtain a self-decoded image, and the self-decoded features of the self-decoded image are extracted. After obtaining various features, the content loss is determined based on the content features of the training content image and the target transfer features of the target transfer image. The style loss is determined based on the style features of the training style image and the target transfer features of the target transfer image. The enhanced self-encoding loss is determined based on the self-decoding features of the self-decoded image and the content features of the training content image. The target loss is determined based on the above style loss, content loss and enhanced self-encoding loss. The style transfer model is trained based on the target loss, and the style image to be processed and the content image are style transferred based on the trained style transfer model to obtain a style transfer image. Through the present disclosure, an enhanced autoencoding loss is determined based on the self-decoding features of a self-decoding image and the content features of a training content image, and a style transfer model is trained based on the enhanced autoencoding loss. This can improve the style feature restoration capability of the decoder of the style training model, so that the target transfer image obtained based on the style transfer model has more content details, clearer lines and fewer edge glitches, thereby improving the quality of the target transfer image.
[0123] Based on the same concept, the embodiment of the present disclosure also provides an image style transfer device 100.
[0124] It is understandable that in order to achieve the above functions, the image style transfer device 100 provided in the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function. In combination with the units and algorithm steps of the various examples disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.
[0125] Fig.11 is a block diagram of an image style transfer apparatus 100 according to an exemplary embodiment. Fig.11 The device includes a collection unit 101, an extraction unit 102, a fusion unit 103 and a training unit 104.
[0126] The acquisition unit 101 is used to acquire a first image and a second image.
[0127] The first image is used to provide the style of the target migration image, and the second image is used to provide the content of the target migration image.
[0128] The extraction unit 102 is used to input the first image and the second image into a first encoder of the image style transfer model, extract the style features of the first image through the first encoder, and extract the content features of the second image.
[0129] The fusion unit 103 is used to fuse the style features and the content features through the feature fusion network of the image style transfer model to obtain fused data, decode the fused data through the decoder of the image style transfer model to obtain a target transfer image, and extract the transfer features of the target transfer image through the second encoder.
[0130] The training unit 104 is used to obtain an image style transfer model through pre-training based on the target loss.
[0131] The enhanced autoencoder loss is determined based on the content loss generated by the second training image after decoding processing by the decoder, and the second training image is an image used to provide predicted migration image content when training the image style transfer model.
[0132] In one implementation, the content loss generated by the second training image after being decoded by the decoder is: a loss determined based on the difference between the second training image and a decoded prediction image, where the decoded prediction image is an image obtained by decoding the second training image by the first encoder.
[0133] In one embodiment, the image style transfer model is pre-trained based on the target loss by the training unit 104 in the following manner: the second training image is input into the encoder of the image style transfer model, and the content features of the second training image are extracted by the encoder. The content features of the second training image are input into the decoder of the image style transfer model, and the decoder decodes the content features of the second training image to obtain a decoded prediction image. Based on the difference between the decoded prediction image and the second training image, an enhanced autoencoder loss is determined. Based on the enhanced autoencoder loss, a target loss is determined, and the image style transfer model is trained based on the target loss.
[0134] In one embodiment, the training unit 104 is further used to: input the first training image into the encoder of the image style transfer model, and extract the style features of the first training image through the encoder. The style features of the first training image and the content features of the second training image are fused through the feature fusion network of the image style transfer model to obtain a predicted transfer feature, and the predicted transfer feature is decoded through the decoder of the image style transfer model to obtain a predicted transfer image. Based on the predicted transfer feature and the style features of the first training image, a style loss is determined, and based on the predicted transfer feature and the content features of the second training image, a content loss is determined. Based on the enhanced autoencoder loss, a target loss is determined, including: based on the style loss, the content loss and the enhanced autoencoder loss, a target loss is determined.
[0135] In one embodiment, the training unit 104 determines the target loss based on the style loss, the content loss and the enhanced autoencoder loss in the following manner: weighting the style loss based on the first parameter to obtain a weighted style loss, weighting the content loss based on the second parameter to obtain a weighted content loss, and weighting the enhanced autoencoder loss based on the third parameter to obtain a weighted enhanced autoencoder loss; the sum of the weighted style loss, the weighted content loss and the weighted enhanced autoencoder loss is determined as the target loss.
[0136] In one implementation, the training unit 104 determines the style loss based on the predicted transfer features and the style features of the first training image in the following manner: determining a first mean square error between the style features of the first training image and the predicted transfer features, and determining the first mean square error as the style loss of the style transfer model.
[0137] In one embodiment, the training unit 104 determines the content loss based on the predicted transfer features and the content features of the second training image in the following manner: aligning the predicted transfer features with the content features of the second training image; determining a second mean square error between features at the same level after the predicted transfer features are aligned with the content features of the second training image; and determining the second mean square error as the content loss of the style transfer model.
[0138] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0139] Fig.12 1 is a block diagram of a device 200 for image style transfer according to an exemplary embodiment. The device 200 may be provided as a terminal. For example, the device 200 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.
[0140] Reference Fig.12, the device 200 may include one or more of the following components: a processing component 202 , a memory 204 , a power component 206 , a multimedia component 208 , an audio component 210 , an input / output (I / O) interface 212 , a sensor component 214 , and a communication component 216 .
[0141] The processing component 202 generally controls the overall operation of the device 200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 202 may include one or more modules to facilitate interaction between the processing component 202 and other components. For example, the processing component 202 may include a multimedia module to facilitate interaction between the multimedia component 208 and the processing component 202.
[0142] The memory 204 is configured to store various types of data to support operations on the device 200. Examples of such data include instructions for any application or method operating on the device 200, contact data, phone book data, messages, pictures, videos, etc. The memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0143] The power component 206 provides power to the various components of the device 200. The power component 206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 200.
[0144] The multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 208 includes a front camera and / or a rear camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
[0145] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC), and when the device 200 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 204 or sent via the communication component 216. In some embodiments, the audio component 210 also includes a speaker for outputting audio signals.
[0146] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.
[0147] The sensor assembly 214 includes one or more sensors for providing various aspects of the status assessment of the device 200. For example, the sensor assembly 214 can detect the open / closed state of the device 200, the relative positioning of components, such as the display and keypad of the device 200, the sensor assembly 214 can also detect the position change of the device 200 or a component of the device 200, the presence or absence of user contact with the device 200, the orientation or acceleration / deceleration of the device 200 and the temperature change of the device 200. The sensor assembly 214 can include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 214 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 214 can also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.
[0148] The communication component 216 is configured to facilitate wired or wireless communication between the device 200 and other devices. The device 200 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0149] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.
[0150] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, and the instructions can be executed by the processor 220 of the device 200 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0151] It is to be understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms "a", "the" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings.
[0152] It is further understood that the terms "first", "second", etc. are used to describe various information, but such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information.
[0153] It will be further understood that the terms “center”, “longitudinal”, “lateral”, “front”, “back”, “up”, “down”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present embodiment and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation.
[0154] It can be further understood that, unless otherwise specified, “connection” includes a direct connection without other components between the two, and also includes an indirect connection with other components between the two.
[0155] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.
[0156] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modifications, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.
[0157] It should be understood that the present disclosure is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the scope of the appended claims.
Claims
1. A method for image style transfer, characterized in that: include: Acquire a first image and a second image, wherein the first image is an image used to provide a style of a target migration image, and the second image is an image used to provide content of a target migration image; Inputting the first image and the second image into a first encoder of an image style transfer model, extracting style features of the first image through the first encoder, and extracting content features of the second image; fusing the style features and the content features through a feature fusion network of the image style transfer model to obtain fused data, decoding the fused data through a decoder of the image style transfer model to obtain a target transfer image, and extracting transfer features of the target transfer image through a second encoder; The image style transfer model is pre-trained based on target loss; The target loss is determined based on an enhanced autoencoder loss, and the enhanced autoencoder loss is determined based on a content loss generated by a second training image after decoding processing by the decoder, wherein the second training image is an image used to provide predicted transfer image content when training the image style transfer model.
2. The method according to claim 1, characterized in that The content loss generated by the second training image after being decoded by the decoder is: A loss determined based on the difference between the second training image and the decoded prediction image, The decoded prediction image is an image obtained by decoding the second training image by the first encoder.
3. The method according to claim 1, characterized in that The image style transfer model is pre-trained based on the target loss in the following manner: Inputting the second training image into a first encoder of the image style transfer model, and extracting content features of the second training image through the first encoder; Inputting the content features of the second training image into a decoder of the image style transfer model, and decoding the content features of the second training image by the decoder to obtain a decoded prediction image; Determining an enhanced autoencoding loss based on a difference between the decoded prediction image and the second training image; Based on the enhanced autoencoder loss, a target loss is determined, and the image style transfer model is trained based on the target loss.
4. The method according to claim 3, characterized in that The method further comprises: Inputting a first training image into a first encoder of an image style transfer model, and extracting style features of the first training image through the first encoder; fusing the style features of the first training image and the content features of the second training image through a feature fusion network of the image style transfer model to obtain a predicted transfer feature, and decoding the predicted transfer feature through a decoder of the image style transfer model to obtain a predicted transfer image; Determining a style loss based on the predicted migration feature and a style feature of the first training image, and determining a content loss based on the predicted migration feature and a content feature of the second training image; The determining of the target loss based on the enhanced autoencoder loss comprises: A target loss is determined based on the style loss, the content loss, and the enhanced autoencoder loss.
5. The method according to claim 4, characterized in that The determining of the target loss based on the style loss, the content loss and the enhanced autoencoder loss comprises: Weighting the style loss based on the first parameter to obtain a weighted style loss, weighting the content loss based on the second parameter to obtain a weighted content loss, and weighting the enhanced autoencoder loss based on the third parameter to obtain a weighted enhanced autoencoder loss; The sum of the weighted style loss, the weighted content loss, and the weighted enhanced autoencoder loss is determined as the target loss.
6. The method according to claim 4 or 5, characterized in that: The determining of the style loss based on the predicted migration feature and the style feature of the first training image includes: determining a first mean square error between the style feature of the first training image and the predicted transfer feature; The first mean square error is determined as the style loss of the style transfer model.
7. The method according to claim 4 or 5, characterized in that: The determining the content loss based on the predicted migration feature and the content feature of the second training image includes: aligning the predicted migration feature with a content feature of the second training image; Determining a second mean square error between features at the same level after the predicted migration feature is aligned with the content feature of the second training image; The second mean square error is determined as the content loss of the style transfer model.
8. An image style transfer device, characterized in that: include: A collection unit, configured to acquire a first image and a second image, wherein the first image is an image used to provide a style of a target migration image, and the second image is an image used to provide content of a target migration image; an extraction unit, configured to input the first image and the second image into a first encoder of an image style transfer model, extract style features of the first image through the first encoder, and extract content features of the second image; A fusion unit, configured to fuse the style features and the content features through a feature fusion network of the image style transfer model to obtain fused data, decode the fused data through a decoder of the image style transfer model to obtain a target transfer image, and extract transfer features of the target transfer image through a second encoder; A training unit, used for pre-training the image style transfer model based on target loss; The target loss is determined based on an enhanced autoencoder loss, and the enhanced autoencoder loss is determined based on a content loss generated by a second training image after decoding processing by the decoder, wherein the second training image is an image used to provide predicted transfer image content when training the image style transfer model.
9. The device according to claim 8, characterized in that The content loss generated by the second training image after being decoded by the decoder is: A loss determined based on the difference between the second training image and the decoded prediction image, The decoded prediction image is an image obtained by decoding the second training image by the first encoder.
10. The device according to claim 8, characterized in that The image style transfer model is pre-trained based on the target loss by the training unit in the following manner: Inputting the second training image into a first encoder of the image style transfer model, and extracting content features of the second training image through the first encoder; Inputting the content features of the second training image into a decoder of the image style transfer model, and decoding the content features of the second training image by the decoder to obtain a decoded prediction image; Determining an enhanced autoencoding loss based on a difference between the decoded prediction image and the second training image; Based on the enhanced autoencoder loss, a target loss is determined, and the image style transfer model is trained based on the target loss.
11. The device according to claim 10, characterized in that The training unit is also used for: Inputting a first training image into a first encoder of an image style transfer model, and extracting style features of the first training image through the first encoder; fusing the style features of the first training image and the content features of the second training image through a feature fusion network of the image style transfer model to obtain a predicted transfer feature, and decoding the predicted transfer feature through a decoder of the image style transfer model to obtain a predicted transfer image; Determining a style loss based on the predicted migration feature and a style feature of the first training image, and determining a content loss based on the predicted migration feature and a content feature of the second training image; The determining of the target loss based on the enhanced autoencoder loss comprises: A target loss is determined based on the style loss, the content loss, and the enhanced autoencoder loss.
12. The device according to claim 11, characterized in that The training unit determines the target loss based on the style loss, the content loss and the enhanced autoencoder loss in the following manner: Weighting the style loss based on the first parameter to obtain a weighted style loss, weighting the content loss based on the second parameter to obtain a weighted content loss, and weighting the enhanced autoencoder loss based on the third parameter to obtain a weighted enhanced autoencoder loss; The sum of the weighted style loss, the weighted content loss, and the weighted enhanced autoencoder loss is determined as the target loss.
13. The device according to claim 11 or 12, characterized in that The training unit determines the style loss based on the predicted transfer feature and the style feature of the first training image in the following manner: determining a first mean square error between the style feature of the first training image and the predicted transfer feature; The first mean square error is determined as the style loss of the style transfer model.
14. The device according to claim 11 or 12, characterized in that The training unit determines the content loss based on the predicted migration feature and the content feature of the second training image in the following manner: aligning the predicted migration feature with a content feature of the second training image; Determining a second mean square error between features at the same level after the predicted migration feature is aligned with the content feature of the second training image; The second mean square error is determined as the content loss of the style transfer model.
15. An image style transfer device, characterized in that: include: processor: a memory for storing processor-executable instructions; Wherein, the processor is configured to: execute the image style transfer method according to any one of claims 1 to 7.
16. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions in the storage medium are executed by a processor, the processor is enabled to execute the image style transfer method according to any one of claims 1 to 7.
