An Arbitrary Style Transfer Method and System Based on Contrast Learning

Abstract

This invention relates to an arbitrary style transfer method and system based on contrastive learning. The method first trains an arbitrary style transfer network model using a training dataset. After training, the target content image is input into a content encoder to obtain content encoding. A reference artwork is input into the style encoder, with the first convolutional layer and the convolutional layer containing max pooling outputting style encodings respectively. The content encoding and the deepest style encoding are input into a replaceable style converter. The output of the replaceable style converter is input into a decoder, and the mean and variance of the style encodings from other layers are input into each feature adaptation single-sample normalization layer in the decoder. The final style transfer result image or video frame is obtained. This invention can transfer the style representation of any specified reference artwork to any target content image, achieving arbitrary stylization of the target content image and improving subjective visual quality and artistic effect.

Description

Technical Field

[0001] This invention belongs to the field of image stylization technology, specifically relating to an arbitrary style transfer method and system based on contrastive learning. Background Technology

[0002] Arbitrary style transfer aims to transfer the artistic style embodied in any given artwork to another given target image, thereby achieving the batch generation of stylized images. In recent years, style transfer has received increasing attention from the academic community.

[0003] Style transfer methods can be mainly divided into three categories. The first is methods based on global statistics, which achieve style transfer by adjusting global statistics such as the mean, variance, and covariance of features to match the artwork. The second is methods based on block matching, which establish block matching relationships between the reference style image and the target content image, and achieve style transfer through block transfer and fusion. The third is methods based on generative adversarial networks (GANs), which introduce generators and discriminators to generate high-quality style transfer results.

[0004] However, methods based on adjusting global statistics cannot comprehensively measure style information, block matching methods cannot transfer semantic information of style, and generative adversarial networks cannot transfer styles that did not appear during training. None of these three types of methods can meet the needs of practical applications. Summary of the Invention

[0005] To address the aforementioned problems, this invention proposes an arbitrary style transfer method and system based on contrastive learning, which can be applied to various existing stylization network models and achieve improved quality of generated results. This invention can transfer the style representation of any specified reference artwork to any target content image, achieving arbitrary stylization of the target content image and improving subjective visual quality and artistic effect.

[0006] The technical solution adopted in this invention is as follows:

[0007] An arbitrary style transfer method based on contrastive learning includes the following steps:

[0008] Using the training dataset, train an arbitrary style transfer network model; the arbitrary style transfer network model includes a content encoder, a content projector, a style encoder, a texture style projector, a color style projector, a replaceable style converter, and a decoder.

[0009] The target content image and reference artwork are input into a pre-trained arbitrary style transfer network model for the following processing steps:

[0010] The target content image is input into the content encoder, and the content is encoded.

[0011] The reference artwork is input into the style encoder, and the first convolutional layer and the convolutional layer containing max pooling output the style code respectively;

[0012] Input the content encoding and the deepest style encoding into the replaceable style converter;

[0013] The output of the replaceable style converter is input into the decoder, and the mean and variance of the style codes of other layers are input into the single-sample normalization layer of each feature in the decoder. The final style transfer result image or video frame is obtained through processing.

[0014] Furthermore, in the arbitrary style transfer network model, both the content encoder and the style encoder include several consecutive convolutional layers, each followed by a linear rectified function. Max pooling is used for downsampling in the non-adjacent convolutional layers and before the last convolutional layer. The content projector, texture style projector, and color style projector each consist of two fully connected layers. The replaceable style converter can employ network modules from various existing style transfer methods. The decoder includes several alternating feature adaptation single-sample normalization layers and convolutional layers, each followed by a linear rectified function. Nearest neighbor upsampling is used in the non-adjacent convolutional layers and before the last convolutional layer.

[0015] Furthermore, the training dataset includes a content dataset consisting of photographic images and a style dataset consisting of painted images; during training, the content dataset is fed into the content encoder and the style dataset is fed into the style encoder.

[0016] Furthermore, both the content encoder and the style encoder have 9 convolutional layers, with max pooling used for downsampling before the 3rd, 5th, and 9th convolutional layers; the decoder has 3 feature adaptation single-sample normalization layers.

[0017] Furthermore, the decoder has nine convolutional layers, with nearest neighbor upsampling used after the 1st, 5th, and 7th convolutional layers, and feature adaptation single-sample normalization layers used after the 4th, 6th, and 8th convolutional layers.

[0018] Furthermore, the feature adaptation single-sample normalization layer contains two fully connected layers, which take the mean and variance of the style encoding as inputs and output the feature adaptation mean and variance for single-sample normalization.

[0019] An arbitrary style transfer system based on contrastive learning includes an arbitrary style transfer network model applicable to various existing style transfer modules. After training, the model processes a target content image and a reference artwork image as input to obtain a style transfer result image. The arbitrary style transfer network model includes:

[0020] The content encoder consists of several consecutive convolutional layers, each followed by a linear rectified function. Max pooling is used for downsampling in non-adjacent convolutional layers and before the last convolutional layer. It is used to process the target content image to obtain the content encoding.

[0021] The style encoder consists of several consecutive convolutional layers, each followed by a linear rectified function. The non-adjacent convolutional layers and the last convolutional layer are downsampled using max pooling. It is used to process reference artwork images. The first convolutional layer and the convolutional layer containing max pooling output style codes, respectively.

[0022] The decoder consists of several alternating feature adaptation single-sample normalization layers and convolutional layers. Each convolutional layer is followed by a linear rectified function. Nearest neighbor upsampling is used before the non-adjacent convolutional layers and the last convolutional layer. The mean and variance of the style codes from different layers of the style encoder are sequentially input into each feature adaptation single-sample normalization layer to obtain the final style transfer result image.

[0023] This invention achieves high-quality style transfer with only one parameter forward pass by adjusting the distribution of the feature space and channels of the neural network. Compared with existing technologies, this invention achieves better results in preserving the semantic structure of the target content, representing the reference style, and improving the overall visual effect. Attached Figure Description

[0024] Figure 1 This is a framework diagram of the arbitrary style transfer network used in the embodiments of the present invention.

[0025] Figure 2 This is a specific structural diagram of the arbitrary style transfer network used in the embodiments of the present invention.

[0026] Figures 3A-3C These are the target content diagram, reference style diagram, and style transfer result diagram of an embodiment of the present invention. Detailed Implementation

[0027] To make the above-mentioned features and advantages of the present invention more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings. It should be noted that the specific number of layers, modules, functions, and layer settings given in the following embodiments are merely preferred implementations and are not intended to limit the scope. Those skilled in the art can select the appropriate number and set different layers according to actual needs, which should be understood.

[0028] This embodiment discloses an arbitrary style transfer method based on contrastive learning. Taking style transfer of a target content image as an example, the specific explanation is as follows:

[0029] Step 1: Collect a large number of photographic images and a large number of art painting images. The photographic images form a content dataset, and the art painting images form a style dataset.

[0030] Step 2: Build an arbitrary style transfer network model.

[0031] Network structure such as Figure 1 As shown in Figure 2, the model is divided into a content encoder E. c Content Projector H c Style Encoder E s Texture style projector H s Color style projector H color The system consists of six sub-networks: a replaceable style converter M and a decoder D. The content encoder E... c It consists of 9 consecutive convolutional layers, each followed by a Rectified Linear Array (ReLU). Max pooling is used for downsampling before the 3rd, 5th, and 9th convolutional layers. The output of the last layer is the content encoding F. c Style Encoder E s With content encoder E c They have the same network structure, but differ in the output of the last layer (i.e. Figure 2 Based on (④) in the previous section, the results of layers 1, 3, and 5 are also output (i.e. Figure 2 (①, ②, ③ in the text).

[0032] Among them, content projector H c The texture-style projector H is used to project content encoding onto a one-dimensional feature vector. s and color style projector H color This is used to project the style encoding onto a one-dimensional feature vector, which is then used to calculate the subsequent loss function term.

[0033] Texture Style Projector H s and color style projector H color Both consist of two fully connected layers, with the first fully connected layer followed by a linear rectified function (ReLU), and the final output is a one-dimensional feature.

[0034] The style converter M can be replaced by a variety of existing style transfer network modules, making this contrastive learning-based arbitrary style transfer method highly transferable and applicable.

[0035] The decoder D consists of alternating layers of 9 convolutional layers, 3 nearest neighbor upsampling layers, and 3 feature adaptation one-sample normalization layers, with each convolutional layer followed by a Rectified Luminaire (ReLU). The 3 feature adaptation one-sample normalization layers sequentially input the mean and variance of style codes from different layers of the style encoder. The structure of the feature adaptation one-sample normalization layer is as follows... Figure 2As shown, it contains two fully connected layers and a single-sample normalization module (Instance Normalization). Figure 2 In this context, 'n' represents the number of input feature channels, and 'm' represents the number of output feature channels. The final output of the decoder is the style transfer result O.

[0036] Step 3: Pre-trained style encoder E s Texture style projector H s and color style projector H color .

[0037] Training Style Encoder E s and texture style projector H s The model loss function is:

[0038] L = L NCE (H s (E s (S q )),H s (E s (S + )),H s (E s (S - )))

[0039] S q S is a random local block of an art painting image as a reference. + For reference, another random local block of an artistic painting image, N is a set of local blocks containing other artistic paintings, with N set to 4096.

[0040] Fixed style encoder E s and texture style projector H s The parameters after training.

[0041] Training color style projector H color The model loss function is:

[0042]

[0043] S jig , All are reference art painting images S q Obtained by performing a 3x3 random puzzle. For S q It is obtained by performing a 3x3 random puzzle and random color perturbation.

[0044] L NCE To compare the learning loss function terms:

[0045]

[0046] Step 4: Train an arbitrary style transfer network model.

[0047] The total loss function term of the model is:

[0048] L=λ c L c +λ s L s +λ color L color +λ rc L rc +λ rs L rs ,

[0049] In the formula, λ c , λ s , λ color , λ rc and λ rs It is a weight term, usually λ c Set to 1, λ s Set to 20, λ color Set to 20, λ rc Set to 1, λ rs Set it to 50.

[0050] L c Loss function term for comparison learning content:

[0051]

[0052] In the formula, The selected first l layers of the content encoder, for The content projector corresponding to the layer, K is set to 4, and l is usually selected as the content encoder E. c The RELU1_1, RELU2_1, RELU3_1, and RELU4_1 layers are represented, where C is the target content image and O is the generated stylized result image.

[0053] L s For the texture style loss function term:

[0054] L s =1-H s (E s (S)).H s (E s (O))

[0055] In the formula, S is the reference style image and O is the generated result image.

[0056] L color For the color style loss function term:

[0057] L colo r = 1 - H color (E s (S))·H color (E s (O))

[0058] S represents the reference style image, and O represents the generated result image.

[0059] L rc Loss function term for content reconstruction:

[0060]

[0061] In the formula, The selected first l layers of the content encoder, For a two-layer fully connected network, K is set to 4, and l is typically chosen as the content encoder E. c The RELU1_1, RELU2_1, RELU3_1, and RELU4_1 layers are defined, where C is the target content image, and ST(C,C) is the reconstruction result of the style transfer model simultaneously inputting C and C.

[0062] L rs For the style reconstruction loss function term:

[0063]

[0064] In the formula, Φ l For the features of the pre-trained image classification model, l is usually taken from the RELU1_1, RELU2_1, RELU3_1, and RELU4_1 layers of the VGG19 model. S is the reference style image, S′ is obtained by randomly perturbing the color and blurring the details of S, and ST(S′,S) is the reconstruction result of the style transfer model with S′ as the target content image and S as the artistic painting image, respectively.

[0065] Step 4: Inference phase, input the reference style image S (see...) Figure 3B ) and target content images (see Figure 3A Finally, output the desired style transfer result image (see...). Figure 3C ).

[0066] Another embodiment of the present invention provides an arbitrary style transfer system based on contrastive learning, comprising an arbitrary style transfer network model applicable to various existing style transfer modules. After training, the model processes a target content image and a reference artwork image as input to obtain a style transfer result image. The arbitrary style transfer network model includes:

[0067] The content encoder consists of several consecutive convolutional layers, each followed by a linear rectified function. Max pooling is used for downsampling in non-adjacent convolutional layers and before the last convolutional layer. It is used to process the target content image to obtain the content encoding.

[0068] The style encoder consists of several consecutive convolutional layers, each followed by a linear rectified function. The non-adjacent convolutional layers and the last convolutional layer are downsampled using max pooling. It is used to process reference artwork images. The first convolutional layer and the convolutional layer containing max pooling output style codes, respectively.

[0069] The decoder consists of several alternating feature adaptation single-sample normalization layers and convolutional layers. Each convolutional layer is followed by a linear rectified function. Nearest neighbor upsampling is used before the non-adjacent convolutional layers and the last convolutional layer. The mean and variance of the style codes from different layers of the style encoder are sequentially input into each feature adaptation single-sample normalization layer to obtain the final style transfer result image.

[0070] Another embodiment of the present invention provides an electronic device (computer, server, smartphone, etc.) including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the steps of the method of the present invention.

[0071] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk) storing a computer program that, when executed by a computer, implements the various steps of the method of the present invention.

[0072] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention. The scope of protection of the present invention should be determined by the claims.

Claims

1. A contrastive learning based arbitrary style transfer method, characterized in that, The method comprises the following steps: An arbitrary style transfer network model is trained using a training data set; the arbitrary style transfer network model comprises a content encoder, a content projector, a style encoder, a texture style projector, a color style projector, a replaceable style converter, and a decoder; The target content image and the reference artwork are input into the trained arbitrary style transfer network model for the following processing steps: The target content image is input into the content encoder to obtain a content code; The reference artwork is input into the style encoder, and the first convolutional layer and the convolutional layer containing the max pooling output style codes; The content code and the deepest layer of the style code are input into the replaceable style converter; The content output by the replaceable style converter is input into the decoder, and the mean and variance of the style codes of other layers are input into the feature adaptive single-sample normalization layer in each feature adaptive single-sample normalization layer in the decoder to obtain a final style transfer result image or video frame; In the arbitrary style transfer network model, the content encoder and the style encoder each comprise a plurality of consecutive convolutional layers, each convolutional layer is followed by a linear rectifier function, and intermediate non-adjacent partial convolutional layers and the last convolutional layer are down-sampled using max pooling; the content projector, the texture style projector, and the color style projector each comprise two fully connected layers; the decoder comprises a plurality of feature adaptive single-sample normalization layers and convolutional layers that alternate with each other, each convolutional layer is followed by a linear rectifier function, and intermediate non-adjacent partial convolutional layers and the last convolutional layer are up-sampled using nearest neighbor up-sampling. In the process of training the arbitrary style transfer network model, the content projector projects the content code into a one-dimensional feature vector, and the texture style projector and the color style projector project the style code into a one-dimensional feature vector for calculation of a loss function term.

2. The method of claim 1, wherein, The training data set comprises a content data set composed of photographic images and a style data set composed of painted images; during training, the content data set is input into the content encoder, and the style data set is input into the style encoder.

3. The method of claim 1, wherein, The convolutional layers of the content encoder and the style encoder are each 9, and the 3rd, 5th, and 9th convolutional layers are down-sampled using max pooling; the feature adaptive single-sample normalization layers of the decoder are 3.

4. The method of claim 1, wherein, The convolutional layers of the decoder are 9, the 1st, 5th, and 7th convolutional layers are up-sampled using nearest neighbor up-sampling, and the 4th, 6th, and 8th convolutional layers are followed by feature adaptive single-sample normalization layers.

5. The method of claim 1, wherein, The feature adaptive single-sample normalization layer comprises two fully connected layers that input the mean and variance of the style code, respectively, and output feature adaptive mean and variance for single-sample normalization.

6. The method of claim 1, wherein, The training of the arbitrary style transfer network model includes two stages: firstly, pre-training a style encoder E s , a texture style projector H s , and a color style projector H color , and then training the arbitrary style transfer network model; where the training style encoder E s and the texture style projector H s The model loss function is: L = L NCE (H s (E s (S q )), H s (E s (S + )), H s (E s (S - ))) S q S is a random local patch of the reference artistic painting image, + S is another random local patch of the reference artistic painting image, S is a set of local patches of other artistic painting images; Training a color style projector H color The model loss function is: S jig , are obtained by 3*3 random puzzle on the reference artistic painting image S q , are obtained by 3*3 random puzzle and random color disturbance on S q ; L NCE For the contrastive learning loss term: The total loss function term of the arbitrary style transfer network model is: L = λ c L c + λ s L s + λ color L color + λ rc L rc + λ rs L rs , where λ c , λ s , λ color , λ rc and λ rs are weight terms, L c is a contrastive learning content loss function term; L s is a texture style loss function term; L color is a color style loss function term; L rc is a content reconstruction loss function term; and L rs is a style reconstruction loss function term.

7. A contrastive learning based arbitrary style transfer system employing the method of any one of claims 1-6. An arbitrary style transfer network model is trained, and after training, a style transfer result image is obtained by inputting a target content image and a reference artwork image; the arbitrary style transfer network model comprises: The content encoder comprises a plurality of convolutional layers in succession, each convolutional layer followed by a linear rectifier function, and intermediate non-adjacent partial convolutional layers and the last convolutional layer are down-sampled using max-pooling; the content encoder is used for processing a target content image to obtain a content code; The style encoder comprises a plurality of convolutional layers in succession, each convolutional layer followed by a linear rectifier function, and intermediate non-adjacent partial convolutional layers and the last convolutional layer are down-sampled using max-pooling; the first convolutional layer and the convolutional layer containing max-pooling output a style code respectively; The decoder comprises a plurality of feature adaptive single sample normalization layers and convolutional layers alternately, each convolutional layer followed by a linear rectifier function, and intermediate non-adjacent partial convolutional layers and the last convolutional layer are up-sampled using nearest neighbor interpolation; wherein Each feature adaptive single sample normalization layer inputs the mean and variance of the style code of different layers of the style encoder in turn, and processes to obtain a final style transfer result image.

8. An electronic device, comprising: The computer readable storage medium stores a computer program, and the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method of any one of claims 1-6.

Images