Multimodal style transfer method, system and device based on knowledge distillation

Abstract

This paper proposes a multimodal style transfer method, system, and device based on knowledge distillation. By introducing knowledge distillation technology, a teacher-student model is designed. Leveraging the spatial alignment of text and images in CLIP, this method achieves multimodal style transfer with a single training run using an image dataset. Compared to CLIPStyler, this method significantly improves speed and efficiency. Furthermore, compared to LDAST, which both train using fixed models, this method incorporates a spatial expansion mapping module, enabling it to be integrated into all conventional image style transfer networks, creating a plug-and-play, multimodal style transfer network with excellent adaptability.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence, computer vision, and image processing, and specifically to a multimodal style transfer method, system, and device based on knowledge distillation. Background Art

[0002] Style transfer is a research direction in the field of computer vision and image processing. Given a reference style image, such as Vincent van Gogh's "Starry Night", style transfer is used to transfer its artistic style, such as color and texture, to another arbitrary content image, thereby combining the content of one image with the style of another image to generate an image with novel artistic effects. To achieve this goal, the style and content of the style image must first be properly separated and then transferred to another content. This involves multiple aspects such as deep learning, neural networks, and image processing. The challenges of this problem are: (1) how to clearly separate the content and style (CS) of an image; and (2) how to transfer the style to another content.

[0003] In 2015, Gatys et al. pioneered the use of pre-trained deep convolutional neural networks (DCNNs) to address the style transfer problem. This method is a descriptive style transfer approach based on image iteration. Its core concept is to extract features from an image using a pre-trained VGG-19 convolutional neural network model, extracting high-level abstract content and style features. Then, starting with a randomly generated white noise image, the abstract feature representations are fitted by iteratively optimizing the noise image, ultimately synthesizing a stylized image that combines the original content features with the new style features.

[0004] While the aforementioned style transfer methods can generate high-quality stylized images, they require users to input both a content image and a style image, which presents certain limitations. However, obtaining a satisfactory style image is often difficult for users. Consequently, multimodal style transfer tasks, such as text-guided image style transfer, have attracted considerable research interest. This task replaces style images with natural language sentences describing the desired style attributes, making style transfer tasks more imaginative and creative.

[0005] The most representative of these approaches is CLIPStyler, a text-guided style transfer method proposed by Kwon. CLIPStyler employs a novel framework that enables style transfer without a style image, but only with a textual description of the desired style. Using the pretrained text-image embedding model CLIP, CLIPStyler modulates the style of the content image using only a single text. Specifically, CLIPStyler proposes a patch text-image matching loss with multi-view enhancement to achieve realistic texture transfer. However, CLIPstyler requires retraining the model for each text input, which is slow, inefficient, and impractical. Consequently, various improvements have been derived based on CLIPStyler. For example, DiffusionCLIP extends CLIPStyler by using a pretrained diffusion model as an image generator for high-quality image synthesis. Fu et al. proposed Language-Driven Artistic Style Transfer (LDAST), which uses two independent encoders to extract style and content features from text and image data. These two features are then fused using an attention operation to produce a stylized result. However, training their model requires a large amount of additional text-image paired data, which is often difficult to obtain and difficult to ensure that the text labels match the images. For these reasons, the image quality produced by LDAST is not satisfactory.

[0006] In summary, there has been some research on the task of text-guided style transfer, but the following problems still exist:

[0007] 1) Existing style transfer models can only perform style transfer by inputting style images or text alone;

[0008] 2) Training the model requires additional text-image paired data, which is usually difficult to obtain and difficult to ensure that the text labels match the images;

[0009] 3) The quality of stylized images in existing frameworks is low, with severe local distortion and a large number of artifacts. Summary of the Invention

[0010] To address the problems of the existing technology, the present invention proposes a multimodal style transfer method, system, and device based on knowledge distillation. By introducing knowledge distillation technology, a teacher-student model is designed. Leveraging the spatial alignment of text and images in CLIP, this method achieves multimodal guided style transfer with a single training run using an image dataset. Compared to CLIPStyler, the speed and efficiency are significantly improved. Furthermore, compared to LDAST, the present invention does not require large-scale text-image pairing data, saving manpower and resources. Furthermore, compared to CLIPStyler and LDAST, which both train on fixed models, the present invention designs a spatial expansion mapping module, enabling the present invention to be inserted into all traditional image style transfer networks, making it a multimodal style transfer network with plug-and-play functionality and excellent adaptability.

[0011] The technical solution of the present invention is:

[0012] Step 1: Build a multimodal style transfer network:

[0013] Establish a traditional style transfer network, including a style image branch and a content image branch; then replace the style image branch with the teacher-student model to obtain a multimodal style transfer network;

[0014] Step 2: Get sample data and train the model:

[0015] The sample data includes content image dataset and style image dataset;

[0016] After obtaining the sample data, the sample data is input into the multimodal style transfer network for joint training. The content image is input into the content image branch for encoding, and the style image is input into the teacher branch and student branch of the teacher-student model for encoding. The vector encoded by the student branch is input into the content image branch for feature fusion with the content image. The vector is then sent to the decoder to decode the image, and the decoded image is re-encoded, and the overall loss L is calculated.

[0017] During training, the overall loss L is designed as follows

[0018] L=αL ts +βL s +δL c

[0019] L ts Training loss for teachers and students, L s is the style loss of the traditional style transfer network, L c is the content loss of the traditional style transfer network, α, β and σ are hyperparameters;

[0020] Step 3: Model Inference:

[0021] After the model training is completed, the teacher branch and loss calculation part in the multimodal style transfer network are discarded, and the trained student branch and content image branch are used for model inference; the text prompt or style image of the desired style is input into the student branch, and the style input extracted by the student branch and the content image to be style transferred are input into the content image branch to achieve style transfer of the image.

[0022] Furthermore, the traditional style transfer network adopts the AdaIN network, including a VGG Encoder module, an AdaIN module, and a Decoder module.

[0023] Furthermore, the teacher-student model includes a teacher branch and a student branch; the teacher branch is composed of VGG19, which is used to extract style features of the style image as a guide; the student branch is composed of a CLIP encoder, a spatial position expansion module, and a spatial mapping module, and the output of the student model has the same architecture as the teacher model.

[0024] Furthermore, in the student branch, when the input is text, the CLIP encoder uses the CLIP text encoder, and when the input is an image, the CLIP encoder uses the image encoder.

[0025] Furthermore, the space expansion mapping module is composed of a space position expansion module and a space mapping module;

[0026] In the spatial position expansion module, a height relative position code Rh and a width relative position code Rw are designed, and the height relative position code Rh is repeated k times along the height dimension, and the width relative position code Rw is repeated k times along the width dimension. The two codes after being repeated k times are then summed to obtain a relative position code; the input vector EI(Is) of the spatial position expansion module is an image encoded by the ImageEncoder of the CLIP encoder or a text feature encoded by the TextEncoder. The input vector EI(Is) is processed through three 1×1 linear projection layers to form a self-attention SA with Q, K, and V. The relative position code obtained by the sum is multiplied by Q and then participates in the SA process to obtain an expanded feature vector Fs;

[0027] The input of the spatial mapping module is the expanded feature vector Fs output from the spatial position expansion module. After passing through a two-layer cross-attention structure, the style feature Ft is finally output.

[0028] Furthermore, the teacher-student training loss L ts The channel KL divergence loss is calculated by calculating the feature vector output by the teacher branch and the vector output by the student branch:

[0029]

[0030] Where H and W represent the height and width of the output features, and C represents the number of channels.

[0031] In addition, the present invention also provides a computer device and a computer-readable storage medium:

[0032] The computer device includes a processor and a memory, and the processor implements the above-mentioned multimodal style transfer method based on knowledge distillation when executing the computer program stored in the memory.

[0033] The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the multimodal style transfer method based on knowledge distillation is implemented.

[0034] Beneficial effects

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. This paper introduces the concept of knowledge distillation for the first time in style transfer. By designing a teacher-student model and channel-wise KL divergence loss, we separate the style space from the CLIP space, thereby guiding the traditional style transfer network to stylize the content image. This method, with only image-guided training, leverages the spatial alignment of text and image in CLIP to achieve multimodal style transfer, significantly improving speed and efficiency compared to CLIPStyler. Furthermore, compared to LDAST, our method does not require large-scale text-image paired data, saving both human and material resources.

[0037] 2. By designing a space expansion mapping module, the present invention enables the designed teacher-student model to be inserted into all traditional image style transfer networks, making it a multimodal style transfer network that can be plug-and-play.

[0038] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0040] Figure 1 : AdaIN network diagram;

[0041] Figure 2 : Schematic diagram of multimodal style transfer network;

[0042] Figure 3 : Schematic diagram of the teacher-student model framework;

[0043] Figure 4 : Schematic diagram of spatial position expansion module;

[0044] Figure 5 : Schematic diagram of spatial mapping module;

[0045] Figure 6 : Model reasoning diagram. DETAILED DESCRIPTION

[0046] The present invention is described in detail below in conjunction with theoretical principles and specific embodiments:

[0047] This invention introduces the technology of knowledge distillation into the multimodal style transfer framework for the first time. Specifically, since the style space and content space are coupled together in the CLIP space and cannot be directly used to guide style transfer, the present invention designs a teacher-student model, using VGG19 as the "teacher" model to guide the "student" model we designed - the spatial expansion mapping branch, and designs a new loss - the channel KL divergence loss, to separate the style space from the CLIP space, thereby guiding the traditional style transfer network to stylize the content image. Therefore, we only need to undergo image-guided training to utilize the spatial alignment property of CLIP text and image to achieve multimodal style transfer. In order to allow our framework to be inserted into any traditional style transfer network, the present invention further designs a spatial expansion mapping module, which consists of two parts: a spatial position expansion module and a spatial mapping module. The spatial position expansion module expands the one-dimensional (1*512) feature vector encoded by CLIP to a two-dimensional (512*512) feature vector aligned with the style input vector of the traditional style transfer network. This allows our designed teacher-student framework to be integrated into any traditional style transfer network. The spatial mapping module maps the CLIP space to the style space, extracting the style features from the CLIP encoding space and inputting them into the traditional style transfer network for stylization guidance.

[0048] Next, a specific embodiment is used to describe the specific steps of the present invention:

[0049] Step 1: Establish a multimodal style transfer network;

[0050] A traditional style transfer network is established, including a style image branch and a content image branch; then the style image branch is replaced by the teacher-student model designed by the present invention to obtain a multimodal style transfer network.

[0051] In this embodiment, the AdaIN network is used as the traditional style transfer network. Figure 1As shown in Figure 2, it consists of a style image branch (red part) and a content image branch (blue part), including a VGG Encoder module, an AdaIN module, and a Decoder module. The style image branch (red part) is then replaced by the teacher-student model designed by the present invention. Figure 2 shown.

[0052] like Figure 2 and Figure 3 As shown, the teacher-student model consists of a teacher branch (blue VGG branch) and a student branch (green CLIP branch). The teacher branch is composed of VGG19, which is used to extract the style features of the style image as a guide; the student branch is composed of a CLIP encoder (CLIP text encoder is used when the input is text, and image encoder is used when the input is image), a spatial position expansion module, and a spatial mapping module. The following is a detailed description of the teacher-student model.

[0053] (1) Teacher branch design

[0054] In this example, VGG19 is used as the "teacher." VGG19 is a well-trained and well-performing model that has been proven effective in extracting style features from images in many traditional style transfer tasks. ITStyler has also demonstrated that the style features extracted by VGG can also be used for multimodal style transfer. In this example, images from the WikiArt style image dataset are used as input for training. The parameters of the pre-trained VGG19 network model are fixed, and the features extracted by the ReLU4-1 layer are used as standard style features to guide the training of the student model.

[0055] (2) Student branch design

[0056] We want to separate the style space from the CLIP space, so we need the output of the student model to have the same architecture as the teacher model. Therefore, we design a spatial expansion mapping module to ensure that the features extracted by the CLIP branch have the same dimensionality as VGG19, so that the output is as close as possible to the output of the teacher model. The CLIP model unifies the text space and the image space. Therefore, after training with the guidance of the teacher model, the student model has the ability to extract style features from text or images.

[0057] (3) Spatial expansion mapping module design

[0058] As mentioned above, in the student branch design, the features extracted by the CLIP branch have the same dimension as those of VGG19 through the spatial expansion mapping module. The spatial expansion mapping module consists of two parts: the spatial position expansion module and the spatial mapping module. They are introduced below:

[0059] Spatial position expansion module:

[0060] like Figure 4 As shown in the figure, in order to expand the dimension to the same dimension as the style feature vector extracted by the traditional style transfer model, a height relative position code Rh and a width relative position code Rw are designed in the spatial position expansion module. The height relative position code Rh and the width relative position code Rw are two learnable random vectors and are randomly initialized. The height relative position code Rh is repeated k times along the height dimension, and the width relative position code Rw is repeated k times along the width dimension. The two codes after being repeated k times are then summed to obtain the relative position code.

[0061] The input vector EI(Is) of the spatial position expansion module is the image Is encoded by the ImageEncoder or the text feature Ts encoded by the TextEncoder from the CLIP encoder. The input vector EI(Is) is processed by three 1×1 linear projection layers to form a self-attention (SA) with Q, K, and V. The relative position encoding obtained by the above sum is multiplied by Q and then participates in the SA process to finally obtain the expanded feature vector Fs.

[0062] Spatial Mapping Module:

[0063] Since the designed spatial position expansion module only expands the feature dimension, but the model still needs to learn the mapping relationship from CLIP space to style space, in order to increase the nonlinearity of the model and allow the model to focus on the more worthy learning parts of the image, the present invention also designs a spatial mapping module, such as Figure 5 As shown, the input is the expanded feature vector Fs from the spatial position expansion module. After passing through two layers of cross-attention, the final output is the style feature Ft. The three 1x1 conv layers in each attention layer act as nonlinear mappings to extract Q, K, and V, preserving the feature map size and the number of channels from the previous layer. This increases the network depth and enables it to learn more complex feature information. By applying multi-layer attention to these image embeddings, key image information can be quickly obtained.

[0064] In this embodiment, the input during training is an image from the style image dataset WikiArt. The student branch first uses the Image Encoder in the CLIP model to encode the image into a 1*512 feature vector and then sends the encoded vector to the spatial expansion mapping module to expand and map it into a 32*32*512 feature vector to keep it consistent with the feature dimension output by the teacher model VGG-19. This not only facilitates the calculation of the loss function, but also allows the student model to be seamlessly inserted into any traditional style transfer network in the subsequent inference process to replace the input of the style branch.

[0065] Step 2: Get sample data and train the model:

[0066] The sample data includes a content image dataset and a style image dataset. In this embodiment, the content image dataset adopts the coco dataset, and the style image dataset adopts the WikiArt dataset.

[0067] After obtaining the sample data, the sample data is input into the multimodal style transfer network for joint training, such as Figure 2 As shown in the figure, the content image is input into the VGG19 of the content image branch for encoding, and the style image is input into the teacher branch and student branch of the teacher-student model for encoding respectively. The vector encoded by the student branch is input into the content image branch for feature fusion with the content image, and then sent to the decoder to decode the image. Finally, the decoded image is sent to VGG19 for re-encoding for subsequent loss calculation.

[0068] During training, the overall loss L is designed as follows

[0069] L=αL ts +βL s +δL c

[0070] L ts Training loss for teachers and students, L s is the style loss of the traditional style transfer network, L c is the content loss of the traditional style transfer network, and α, β, and σ are hyperparameters.

[0071] Teacher-student training loss L ts The channel KL divergence loss is calculated by calculating the feature vector output by the teacher branch and the vector output by the student branch. The specific analysis is as follows:

[0072] During training, the teacher model transfers its learned knowledge about image style to the student model in some form. Previous literature suggests that the probability distributions of image features of the same style should be as similar as possible, and KL divergence can be used to compare the proximity of two probability distributions. Therefore, to narrow the gap between the student model output and the teacher model output, this paper designs a channel-wise KL divergence loss to ensure that the probability distributions of each channel are close.

[0073] This paper uses the style features output by the teacher network model VGG19 as the standard style features, and calculates the loss with the feature vector output by the student branch. The specific calculation method is shown in the formula:

[0074]

[0075] Here, H and W represent the height and width of the output feature, and C represents the number of channels. P and Q refer to two different probability distributions: P is the actual probability distribution, and Q is an approximation. KL divergence accurately calculates the information lost between one approximation and the other. KL divergence loss is calculated for each channel of the feature vector output by the student branch and each channel of the standard style feature to ensure that the probability distribution of the style feature in each channel is close to the standard. The sum and average of these values ​​are then added and averaged per channel to obtain our channel-wise KL divergence loss.

[0076] Step 3: Model Inference:

[0077] After the model training is completed, we load the trained weights into the model for inference. First, we obtain the content image to be style transferred and the text prompt or style image of the desired style. During inference, we discard the teacher branch and loss calculation part of the multimodal style transfer network and only use our trained student branch and content image branch for model inference, as shown in the following example: Figure 3 As shown in the figure, the text or style image is input into the student branch, and the content image is input into the content image branch. If the input into the student branch is natural language text, the TextEncoder in CLIP is used, and the image is used by ImageEncoder. By utilizing the spatial alignment property of CLIP text and image, multimodal input guidance can be achieved. The style extracted by the student branch is then input into the trained content image branch to achieve style transfer of the image.

[0078] In addition, this embodiment also provides a computer device and a computer-readable storage medium:

[0079] The computer device includes a processor and a memory, and the processor implements the above-mentioned multimodal style transfer method based on knowledge distillation when executing the computer program stored in the memory.

[0080] The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the multimodal style transfer method based on knowledge distillation is implemented.

[0081] From the above description, it can be seen that the present invention introduces the technology of knowledge distillation into the multimodal style transfer framework, which solves the problem that existing models require a large amount of text-image paired datasets, and can realize style transfer with both text and image modalities as input. Specifically, since the style space and content space are coupled together in the CLIP space and cannot be directly used to guide style transfer, a teacher-student model is designed. The trained VGG19 is used as the "teacher" model to guide the designed "student" model - the space expansion mapping branch, and a new loss - the channel KL divergence loss is designed to separate the style space from the CLIP space, thereby guiding the traditional style transfer network to stylize the content image. Therefore, the present invention only needs to undergo image-guided training to utilize the property of CLIP text and image space alignment to achieve multimodal style transfer.

[0082] To allow the proposed framework to be inserted into any traditional style transfer network, the present invention designs a spatial expansion mapping module, which consists of two parts: a spatial position expansion module and a spatial mapping module. The spatial position expansion module expands the 1*512 feature vector encoded by CLIP to a 32*32*512 feature vector, thereby aligning the dimension of the style input vector of the traditional style transfer network. This allows the designed teacher-student framework to be inserted into any traditional style transfer network. The spatial mapping module maps the CLIP space to the style space, extracting the style features from the CLIP encoding space and inputting them into the traditional style transfer network for stylization guidance.

[0083] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention without departing from the principles and purpose of the present invention.

Claims

1. A multimodal style transfer method based on knowledge distillation, characterized by: The following steps are involved: Step 1: Build a multimodal style transfer network: Establish a traditional style transfer network, including style image branch and content image branch; Then the style image branch is replaced with the teacher-student model to obtain a multimodal style transfer network; Step 2: Get sample data and train the model: The sample data includes content image dataset and style image dataset; After obtaining the sample data, the sample data is input into the multimodal style transfer network for joint training; The content image is input into the content image branch for encoding, and the style image is input into the teacher branch and student branch of the teacher-student model for encoding. The vector encoded by the student branch is input into the content image branch for feature fusion with the content image, and then sent to the decoder to decode the image. The decoded image is re-encoded and the overall loss L is calculated. During training, the overall loss L is designed as follows L=αL ts +βL s +δL c L ts Training loss for teachers and students, L s is the style loss of the traditional style transfer network, L c is the content loss of the traditional style transfer network, α, β and σ are hyperparameters; Step 3: Model Inference: After the model training is completed, the teacher branch and loss calculation part in the multimodal style transfer network are discarded, and the trained student branch and content image branch are used for model inference; the text prompt or style image of the desired style is input into the student branch, and the style input extracted by the student branch and the content image to be style transferred are input into the content image branch to achieve style transfer of the image.

2. The multimodal style transfer method based on knowledge distillation according to claim 1, characterized in that: The traditional style transfer network adopts the AdaIN network, including the VGG Encoder module, the AdaIN module, and the Decoder module.

3. The multimodal style transfer method based on knowledge distillation according to claim 1, characterized in that: The teacher-student model includes a teacher branch and a student branch; the teacher branch is composed of VGG19 and is used to extract style features of the style image as a guide; the student branch is composed of a CLIP encoder, a spatial position expansion module, and a spatial mapping module. The output of the student model has the same architecture as the teacher model.

4. The multimodal style transfer method based on knowledge distillation according to claim 3, characterized in that: In the student branch, when the input is text, the CLIP encoder uses the CLIP text encoder, and when the input is an image, the CLIP encoder uses the image encoder.

5. The multimodal style transfer method based on knowledge distillation according to claim 3, characterized in that: The space expansion mapping module consists of a space position expansion module and a space mapping module; In the spatial position expansion module, a height relative position code Rh and a width relative position code Rw are designed, and the height relative position code Rh is repeated k times along the height dimension, and the width relative position code Rw is repeated k times along the width dimension. The two codes after being repeated k times are then summed to obtain a relative position code; the input vector EI(Is) of the spatial position expansion module is an image encoded by the ImageEncoder of the CLIP encoder or a text feature encoded by the TextEncoder. The input vector EI(Is) is processed through three 1×1 linear projection layers to form a self-attention SA with Q, K, and V. The relative position code obtained by the sum is multiplied by Q and then participates in the SA process to obtain an expanded feature vector Fs; The input of the spatial mapping module is the expanded feature vector Fs output from the spatial position expansion module. After passing through a two-layer cross-attention structure, the style feature Ft is finally output.

6. The multimodal style transfer method based on knowledge distillation according to claim 1, characterized in that: Teacher-student training loss L ts The channel KL divergence loss is calculated by calculating the feature vector output by the teacher branch and the vector output by the student branch: Where H and W represent the height and width of the output features, and C represents the number of channels.

7. A computer device, characterized in that: The method comprises a processor and a memory, wherein when the processor executes the computer program stored in the memory, the method implements the multimodal style transfer method based on knowledge distillation according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the multimodal style transfer method based on knowledge distillation according to any one of claims 1 to 6 is implemented.

Images