Multi-modal style migration method, system and equipment based on knowledge distillation

Abstract

The invention provides a multi-modal style migration method, system and equipment based on knowledge distillation, a teacher-student model is designed by introducing a knowledge distillation technology, and multi-modal guided style migration can be realized only by using an image data set to carry out one-time training by using a CLIP text image space alignment property. Compared with CLIPStyler, the speed and efficiency are greatly improved, and compared with LDAST, large-scale text image pairing data is not needed, and manpower and material resources are saved. Besides, compared with the fact that both CLIPStyler and LDAST are trained by designing fixed models, a spatial expansion mapping module is designed in the invention, so that the method can be inserted into all traditional image style migration networks to become a multi-modal style migration network, plug-and-play is realized, and the method has excellent adaptability.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence, computer vision and image processing, and specifically to a multimodal style transfer method, system and device based on knowledge distillation. Background Art

[0002] Style transfer is a research direction in the field of computer vision and image processing. Given a reference style image, such as Vincent Van Gogh's "Starry Night", style transfer is used to transfer its artistic style, such as color and texture, to another arbitrary content image, so as to combine the content of one image with the style of another image and generate an image with novel artistic effects. To achieve this goal, the style of the style image must first be properly separated from the content, and then transferred to another content, which involves deep learning, neural networks, and image processing. The challenges of this problem are: (1) how to clearly separate the content and style (CS) of an image; (2) how to transfer the style to another content.

[0003] In 2015, Gatys et al. first proposed using pre-trained deep convolutional neural networks (DCNNs) to solve the problem of style transfer. This method is a descriptive style transfer method based on image iteration. Its core idea is to use the pre-trained VGG-19 convolutional neural network model to extract features from images, which are used to extract high-level abstract content feature representations and style feature representations, respectively. Then, starting from a randomly generated white noise image, the abstract feature representation is fitted by continuously iteratively optimizing the noise image, and finally a stylized image with original content features and new style features is synthesized.

[0004] Although the above style transfer methods can generate high-quality stylized images, users need to input both content images and style images at the same time, which has certain limitations. However, in general, it is difficult for users to obtain a style image that satisfies them. Therefore, multimodal style transfer tasks represented by text-guided image style transfer have attracted the interest of many researchers. This task replaces the style image with a natural language sentence that describes the required style attributes, making the style transfer task more imaginative and creative.

[0005] The most representative of these is CLIPStyler, a text-guided style transfer method proposed by Kwon. CLIPStyler adopts a new framework that can achieve style transfer in the absence of a style image but only a text description of the desired style. Using the pre-trained text-image embedding model CLIP, CLIPStyler achieves the modulation of the content image style under the condition of only a single text. Specifically, CLIPStyler proposes a patch text-image matching loss with multi-view enhancement to achieve realistic texture transfer. However, for each text input, CLIPstyler needs to retrain the model, which is slow, inefficient and impractical. Therefore, various improved methods have been derived based on CLIPStyler, such as DiffusionCLIP, which extends CLIPStyler by using a pre-trained diffusion model as an image generator for high-quality image synthesis; Fu et al. proposed Language-Driven Artistic Style Transfer (LDAST), which uses two independent encoders to extract style and content features from text and image data. Then, the two features are fused using an attention operation to obtain a stylized result. However, training their model also requires a large amount of additional text-image pairing data, which is usually difficult to obtain and it is difficult to ensure that the text label matches the image. For these reasons, the image quality produced by LDAST is not satisfactory.

[0006] In summary, there has been some research on the task of text-guided style transfer, but the following problems still exist:

[0007] 1) Existing style transfer models can only input style images or texts separately for style transfer;

[0008] 2) Training the model requires additional text-image pairing data, which is usually difficult to obtain and it is also difficult to ensure that the text labels match the images;

[0009] 3) The quality of stylized images in existing frameworks is not high, with serious local distortion and a large number of artifacts. Summary of the invention

[0010] In view of the problems existing in the prior art, the present invention proposes a multimodal style transfer method, system and device based on knowledge distillation. By introducing knowledge distillation technology, a teacher-student model is designed, and the property of spatial alignment of CLIP text images is utilized. Only one training with an image data set is required to realize multimodal guided style transfer. Compared with CLIPStyler, the speed and efficiency are greatly improved, and compared with LDAST, the present invention does not require large-scale text image pairing data, saving manpower and material resources. In addition, compared with CLIPStyler and LDAST, which are trained by designing fixed models, the present invention designs a space expansion mapping module, so that the present invention can be inserted into all traditional image style transfer networks, making it a multimodal style transfer network, realizing plug-and-play, and having excellent adaptability.

[0011] The technical solution of the present invention is:

[0012] Step 1: Build a multimodal style transfer network:

[0013] A traditional style transfer network is established, including a style image branch and a content image branch. The style image branch is then replaced with a teacher-student model to obtain a multimodal style transfer network.

[0014] Step 2: Get sample data and train the model:

[0015] The sample data includes content image dataset and style image dataset;

[0016] After obtaining the sample data, the sample data is input into the multimodal style transfer network for joint training; the content image is input into the content image branch for encoding, and the style image is input into the teacher branch and the student branch of the teacher-student model for encoding, and the vector encoded by the student branch is input into the content image branch for feature fusion with the content image, and then sent to the decoder to decode the image, and the decoded image is re-encoded, and the overall loss L is calculated;

[0017] During training, the overall loss L is designed as follows

[0018] L=αL ts +βL s +δL c

[0019] L ts is the teacher-student training loss, L s is the style loss of the traditional style transfer network, L c is the content loss of the traditional style transfer network, α, β and σ are hyperparameters;

[0020] Step 3: Model Inference:

[0021] After the model training is completed, the teacher branch and loss calculation part in the multimodal style transfer network are discarded, and the trained student branch and content image branch are used for model reasoning; the text prompt or style image of the desired style is input into the student branch, and the style input extracted by the student branch and the content image that needs to be style transferred are input into the content image branch to achieve style transfer of the image.

[0022] Furthermore, the traditional style transfer network adopts an AdaIN network, including a VGG Encoder module, an AdaIN module, and a Decoder module.

[0023] Furthermore, the teacher-student model includes a teacher branch and a student branch; the teacher branch is composed of VGG19, which is used to extract style features of the style image as a guide; the student branch is composed of a CLIP encoder, a spatial position expansion module, and a spatial mapping module, and the output of the student model has the same architecture as the teacher model.

[0024] Furthermore, in the student branch, when the input is text, the CLIP encoder uses the CLIP text encoder, and when the input is an image, the CLIP encoder uses the image encoder.

[0025] Further, the space expansion mapping module is composed of a space position expansion module and a space mapping module;

[0026] In the spatial position expansion module: a height relative position code Rh and a width relative position code Rw are designed, and the height relative position code Rh is repeated k times along the height dimension, and the width relative position code Rw is repeated k times along the width dimension, and then the two codes after being repeated k times are summed to obtain a relative position code; the input vector EI (Is) of the spatial position expansion module is an image encoded by ImageEncoder or a text feature encoded by TextEncoder from the CLIP encoder, and the input vector EI (Is) is processed through three 1×1 linear projection layers to form a self-attention SA with Q, K, and V, and the relative position code obtained by the sum is multiplied by Q, and then participates in the SA process to obtain an expanded feature vector Fs;

[0027] The input of the spatial mapping module is the expanded feature vector Fs output from the spatial position expansion module, which passes through a two-layer cross-attention structure and finally outputs the style feature Ft.

[0028] Furthermore, the teacher-student training loss L ts The channel KL divergence loss is calculated by calculating the feature vector output by the teacher branch and the vector output by the student branch:

[0029]

[0030] Where H and W represent the height and width of the output features, and C represents the number of channels.

[0031] In addition, the present invention also provides a computer device and a computer-readable storage medium:

[0032] The computer device includes a processor and a memory, and the processor implements the above-mentioned multimodal style transfer method based on knowledge distillation when executing the computer program stored in the memory.

[0033] The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the multimodal style transfer method based on knowledge distillation is implemented.

[0034] Beneficial Effects

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. This invention introduces the idea of ​​knowledge distillation in style transfer for the first time. By designing a teacher-student model and channel KL divergence loss, the style space is separated from the CLIP space, thereby guiding the traditional style transfer network to stylize the content image. This invention only needs image-guided training to utilize the properties of CLIP text and image space alignment to achieve multimodal style transfer, which greatly improves the speed and efficiency compared to CLIPStyler. And compared to LDAST, our method does not require large-scale text-image pairing data, saving manpower and material resources.

[0037] 2. The present invention designs a space expansion mapping module so that the designed teacher-student model can be inserted into all traditional image style transfer networks, making it a multimodal style transfer network that can be plug-and-play.

[0038] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0040] Figure 1 :AdaIN network diagram;

[0041] Figure 2 : Schematic diagram of multimodal style transfer network;

[0042] Figure 3 : Schematic diagram of the teacher-student model framework;

[0043] Figure 4 : Schematic diagram of spatial position expansion module;

[0044] Figure 5 : Schematic diagram of spatial mapping module;

[0045] Figure 6 : Schematic diagram of model reasoning. DETAILED DESCRIPTION

[0046] The present invention is described in detail below in conjunction with theoretical principles and specific embodiments:

[0047] The present invention introduces the technology of knowledge distillation into the multimodal style transfer framework for the first time. Specifically, since the style space and content space are coupled together in the CLIP space and cannot be directly used to guide style transfer, the present invention designs a teacher-student model, using VGG19 as the "teacher" model to guide the "student" model we designed---the spatial extension mapping branch, and designs a new loss--channel KL divergence loss to separate the style space from the CLIP space, thereby guiding the traditional style transfer network to stylize the content image. Therefore, we only need to undergo image-guided training to utilize the property of CLIP text and image space alignment to achieve multimodal style transfer. And in order to allow our framework to be inserted into any traditional style transfer network, the present invention further designs a spatial extension mapping module, which consists of two parts: a spatial position expansion module and a spatial mapping module. The spatial position expansion module expands the one-dimensional (1*512) feature vector encoded by CLIP to a two-dimensional (512*512) feature vector that is aligned with the style input vector dimension of the traditional style transfer network, so that the teacher-student framework we designed can be inserted into any traditional style transfer network. The spatial mapping module maps the CLIP space to the style space, extracts the style features from the CLIP encoding space, and inputs them into the traditional style transfer network for stylization guidance.

[0048] Next, the specific steps of the present invention are described with a specific embodiment:

[0049] Step 1: Establish a multimodal style transfer network;

[0050] A traditional style transfer network is established, including a style image branch and a content image branch; then the style image branch is replaced by the teacher-student model designed by the present invention to obtain a multimodal style transfer network.

[0051] In this embodiment, the AdaIN network is used as the traditional style transfer network. Figure 1As shown in FIG. 1 , the style image branch (red part) and the content image branch (blue part) are composed of a VGG Encoder module, an AdaIN module, and a Decoder module. Then, the teacher-student model designed by the present invention is used to replace the style image branch (red part). Figure 2 shown.

[0052] like Figure 2 and Figure 3 As shown in Figure 1, the teacher-student model consists of a teacher branch (blue VGG branch) and a student branch (green CLIP branch). The teacher branch is composed of VGG19, which is used to extract the style features of the style image as a guide; the student branch is composed of a CLIP encoder (CLIP text encoder is used when the input is text, and image encoder is used when the input is an image), a spatial position expansion module, and a spatial mapping module. The following is a detailed description of the teacher-student model

[0053] (1) Teacher branch design

[0054] In this embodiment, VGG19 is used as a "teacher". VGG19 is a well-trained and well-performing model. It has been proven in many traditional style transfer tasks that it can effectively extract image style features and has been confirmed in ITStyler that the style features extracted by VGG can also be used for multimodal style transfer. In this embodiment, the input during training is an image in the style image dataset WikiArt, the pre-trained VGG19 network model parameters are fixed, and the features extracted by the ReLU4-1 layer are used as standard style features to guide the student model to train.

[0055] (2) Student branch design

[0056] We want to separate the style space from the CLIP space, so the output of the student model needs to have the same architecture as the teacher model. Therefore, the present invention designs a space expansion mapping module to make the features extracted by the CLIP branch have the same dimension as VGG19, so as to make its output as close to the output of the teacher model as possible. The CLIP model realizes the unification of text space and image space, so after being trained under the guidance of the teacher model, the student model has the ability to extract style features from text or images.

[0057] (3) Spatial expansion mapping module design

[0058] As mentioned above, in the student branch design, the features extracted by the CLIP branch have the same dimension as those of VGG19 through the spatial extension mapping module. The spatial extension mapping module consists of two parts: the spatial position extension module and the spatial mapping module, which are introduced separately below:

[0059] Spatial location expansion module:

[0060] like Figure 4 As shown in the figure, in order to expand the dimension to the same dimension as the style feature vector extracted by the traditional style transfer model, a height relative position code Rh and a width relative position code Rw are designed in the spatial position expansion module. The height relative position code Rh and the width relative position code Rw are two learnable random vectors and are randomly initialized. The height relative position code Rh is repeated k times along the height dimension, and the width relative position code Rw is repeated k times along the width dimension. The two codes after being repeated k times are summed to obtain the relative position code.

[0061] The input vector EI(Is) of the spatial position expansion module is the image Is encoded by the ImageEncoder of the CLIP encoder or the text feature Ts encoded by the TextEncoder. The input vector EI(Is) is processed by three 1×1 linear projection layers to form a self-attention (SA) with Q, K, and V. The relative position encoding obtained by the above sum is multiplied by Q and then participates in the SA process to finally obtain the expanded feature vector Fs.

[0062] Spatial Mapping Module:

[0063] Since the designed spatial position expansion module is only expanded in the feature dimension, but the model still needs to learn the mapping relationship from CLIP space to style space, in order to increase the nonlinearity of the model and allow the model to focus on the more worthy learning parts of the image, the present invention also designs a spatial mapping module, such as Figure 5 As shown in the figure, the input is the expanded feature vector Fs output from the spatial position expansion module, which is finally output as the style feature Ft after passing through two layers of cross attention structure. The three 1*1conv layers in each layer of attention structure extract Q, K, and V as nonlinear mappings, keeping the size of the feature map unchanged and maintaining the number of channels of the previous layer, while increasing the depth of the network and enabling the network to learn more complex feature information. By performing multi-layer attention on these image embeddings, the key information of the image can be quickly obtained.

[0064] In this embodiment, the input during training is an image in the style image dataset WikiArt. The student branch first uses the Image Encoder in the CLIP model to encode the image into a 1*512 feature vector and then sends the encoded vector to the spatial extension mapping module to expand and map it into a 32*32*512 feature vector to keep it consistent with the feature dimension output by the teacher model VGG-19. This not only facilitates the calculation of the loss function, but also allows the student model to be seamlessly inserted into any traditional style transfer network in the subsequent reasoning process to replace the input of the style branch.

[0065] Step 2: Get sample data and train the model:

[0066] The sample data includes a content image dataset and a style image dataset. In this embodiment, the content image dataset adopts the coco dataset, and the style image dataset adopts the WikiArt dataset.

[0067] After obtaining the sample data, the sample data is input into the multimodal style transfer network for joint training, such as Figure 2 As shown, the content image is input into the VGG19 of the content image branch for encoding, and the style image is input into the teacher branch and the student branch of the teacher-student model for encoding respectively, and the vector encoded by the student branch is input into the content image branch for feature fusion with the content image, and then sent to the decoder to decode the image, and finally the decoded image is sent to VGG19 for re-encoding for subsequent loss calculation.

[0068] During training, the overall loss L is designed as follows

[0069] L=αL ts +βL s +δL c

[0070] L ts is the teacher-student training loss, L s is the style loss of the traditional style transfer network, L c is the content loss of the traditional style transfer network, and α, β and σ are hyperparameters.

[0071] Teacher-student training loss L ts The channel KL divergence loss is calculated by the feature vector output by the teacher branch and the vector output by the student branch. The specific analysis is as follows:

[0072] During the training process, the teacher model transfers the knowledge it has learned about how to understand the image style to the student model in some form. According to previous literature, the probability distribution of image features of the same style should be as similar as possible, and KL divergence can be used to compare the closeness of two probability distributions. Therefore, in order to narrow the distance between the output of the student model and the output of the teacher model, the present invention designs a channel KL divergence loss to make the probability distribution of each channel close.

[0073] The present invention uses the style features output by the teacher network model VGG19 as the standard style features, and uses the feature vector output by the student branch to perform loss calculation with it. The specific calculation method is shown in the formula:

[0074]

[0075] Among them, H and W represent the height and width of the output feature, and C represents the number of channels; P and Q refer to two different probability distributions, P is the actual probability distribution, and Q is the approximate distribution. According to the KL divergence, we can accurately calculate how much information is lost between the approximate distribution and the other distribution. The KL divergence loss is calculated for each channel of the feature vector output by the student branch and each channel of the standard style feature to ensure that the probability distribution of the style feature of each channel is close to the standard, and then the sum is taken according to the channel to get our channel KL divergence loss.

[0076] Step 3: Model Inference:

[0077] After the model training is completed, we load the trained weights into the model for inference. First, we obtain the content image that needs to be style transferred and the text prompt or style image of the style we want. During inference, we discard the teacher branch and loss calculation part in the multimodal style transfer network and only use our trained student branch and content image branch for model inference, such as Figure 3 As shown in the figure, the text or style image is input into the student branch, and the content image is input into the content image branch. If the input into the student branch is natural language text, the TextEncoder in CLIP is used, and the image is used by ImageEncoder. By utilizing the spatial alignment property of CLIP text and image, multimodal input guidance can be achieved. Then, the style extracted by the student branch is input into the trained content image branch to achieve style transfer of the image.

[0078] In addition, this embodiment also provides a computer device and a computer-readable storage medium:

[0079] The computer device includes a processor and a memory, and the processor implements the above-mentioned multimodal style transfer method based on knowledge distillation when executing the computer program stored in the memory.

[0080] The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the multimodal style transfer method based on knowledge distillation is implemented.

[0081] It can be seen from the above description that the present invention introduces the technology of knowledge distillation into the multimodal style transfer framework, solves the problem that the existing model requires a large number of text-image pairing data sets, and can realize style transfer with two modes of text and image as input. Specifically, since the style space and content space are coupled together in the CLIP space, they cannot be directly used to guide style transfer. Therefore, a teacher-student model is designed, using the trained VGG19 as the "teacher" model to guide the designed "student" model---the space expansion mapping branch, and a new loss---channel KL divergence loss is designed to separate the style space from the CLIP space, thereby guiding the traditional style transfer network to stylize the content image. Therefore, in the present invention, only image-guided training is required, and the property of CLIP text and image space alignment can be used to achieve multimodal style transfer.

[0082] In order to allow the framework proposed by the present invention to be inserted into any traditional style transfer network, the present invention designs a space expansion mapping module, which consists of two parts: a space position expansion module and a space mapping module. The space position expansion module expands the 1*512 feature vector after CLIP encoding to a 32*32*512 feature vector, thereby aligning with the style input vector dimension of the traditional style transfer network, so that the designed teacher-student framework can be inserted into any traditional style transfer network. The space mapping module maps the CLIP space to the style space, strips the style features from the CLIP encoding space, and inputs them into the traditional style transfer network for stylized guidance.

[0083] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention without departing from the principles and intent of the present invention.

Claims

1. A multimodal style transfer method based on knowledge distillation, characterized by: The following steps are involved: Step 1: Build a multimodal style transfer network: Establish a traditional style transfer network, including style image branch and content image branch; Then the style image branch is replaced with the teacher-student model to obtain a multimodal style transfer network; Step 2: Get sample data and train the model: The sample data includes content image dataset and style image dataset; After obtaining the sample data, the sample data is input into the multimodal style transfer network for joint training; The content image is input into the content image branch for encoding, and the style image is input into the teacher branch and the student branch of the teacher-student model for encoding. The vector encoded by the student branch is input into the content image branch for feature fusion with the content image, and then sent to the decoder to decode the image, and the decoded image is re-encoded, and the overall loss L is calculated; During training, the overall loss L is designed as follows L=αL ta +βL s +δL c L ts is the teacher-student training loss, L s is the style loss of the traditional style transfer network, L c is the content loss of the traditional style transfer network, α, β and σ are hyperparameters; Step 3: Model Inference: After the model training is completed, the teacher branch and loss calculation part in the multimodal style transfer network are discarded, and the trained student branch and content image branch are used for model reasoning; the text prompt or style image of the desired style is input into the student branch, and the style input extracted by the student branch and the content image that needs to be style transferred are input into the content image branch to achieve style transfer of the image.

2. According to claim 1, a multimodal style transfer method based on knowledge distillation is characterized in that: The traditional style transfer network adopts the AdaIN network, including a VGG Encoder module, an AdaIN module, and a Decoder module.

3. According to the multimodal style transfer method based on knowledge distillation of claim 1, it is characterized by: The teacher-student model includes a teacher branch and a student branch; the teacher branch is composed of VGG19, which is used to extract the style features of the style image as a guide; the student branch is composed of a CLIP encoder, a spatial position expansion module, and a spatial mapping module, and the output of the student model has the same architecture as the teacher model.

4. The multimodal style transfer method based on knowledge distillation according to claim 3, characterized in that: In the student branch, when the input is text, the CLIP encoder uses the CLIP text encoder, and when the input is an image, the CLIP encoder uses the image encoder.

5. The multimodal style transfer method based on knowledge distillation according to claim 3, characterized in that: The space expansion mapping module is composed of a space position expansion module and a space mapping module; In the spatial position expansion module: a height relative position code Rh and a width relative position code Rw are designed, and the height relative position code Rh is repeated k times along the height dimension, and the width relative position code Rw is repeated k times along the width dimension, and then the two codes after being repeated k times are summed to obtain a relative position code; the input vector EI (Is) of the spatial position expansion module is an image encoded by ImageEncoder or a text feature encoded by TextEncoder from the CLIP encoder, and the input vector EI (Is) is processed through three 1×1 linear projection layers to form a self-attention SA with Q, K, and V, and the relative position code obtained by the sum is multiplied by Q, and then participates in the SA process to obtain an expanded feature vector Fs; The input of the spatial mapping module is the expanded feature vector Fs output from the spatial position expansion module, which passes through a two-layer cross-attention structure and finally outputs the style feature Ft.

6. The multimodal style transfer method based on knowledge distillation according to claim 1, characterized in that: Teacher-student training loss L ts The channel KL divergence loss is calculated by calculating the feature vector output by the teacher branch and the vector output by the student branch: Where H and W represent the height and width of the output features, and C represents the number of channels.

7. A computer device, characterized in that: The method comprises a processor and a memory, wherein when the processor executes a computer program stored in the memory, the method implements the multimodal style transfer method based on knowledge distillation according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the multimodal style transfer method based on knowledge distillation according to any one of claims 1 to 6 is implemented.

Images