Text-driven style migration method based on cross-space adjustment

Abstract

The invention discloses a text-driven style migration method based on cross-space adjustment. The method comprises the following steps: extracting content features from any given photo by using a pre-trained deep convolutional neural network VGG-19; extracting style features from any given text prompt by using a pre-trained CLIP text encoder; constructing a cross-space adjustment module for bridging the difference between the VGG feature space and the CLIP feature space; and constructing a content loss function, a directional CLIP loss function, a block-to-block CLIP loss function and a CLIP-based style comparison loss function for constraining the generation process of the style migration network, and finally obtaining an artistic image conforming to the target content and the target style. According to the method, the art style described in the text can be migrated to a daily photo, a new art image is created, the flexibility and innovativeness of art image creative generation are improved, and the method has important scientific significance and application value in the fields of cultural creativity, film and television production, industrial design and the like.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and deep learning, and in particular relates to a text-driven style transfer method based on cross-space regulation. Background Art

[0002] Style transfer mainly studies how to transfer the target style to the content image (such as daily photos) while keeping its semantic structure unchanged to generate new artistic images. Early research mainly used related technologies of computer graphics to analyze and mathematically model artistic styles. However, such methods often rely on low-level statistical data and cannot capture high-level semantic information, thus failing to achieve ideal style transfer effects. In recent years, the rapid rise of deep learning has pointed out a new development direction for style transfer research. Researchers use pre-trained deep convolutional neural networks to extract content features and style features in images respectively, achieving the decoupling of content and style. After that, the style information in artistic paintings and the content information in daily photos are reorganized and fused to generate new artistic images. Since then, style transfer has entered a rapid development stage, and a series of excellent style transfer methods based on deep learning have emerged, which have improved and improved the effect of style transfer in terms of efficiency, quality, generalization, and other aspects. Although existing style transfer methods have made great progress, since current research mainly uses one or more artistic paintings to represent the target style, there are the following disadvantages: (1) The style image needs to be prepared in advance, and sometimes even the expected style image needs to be manually drawn before style transfer, which adds extra overhead and is very inflexible; (2) Style transfer can only be performed based on existing paintings, which has great limitations and is obviously lacking in innovation. Summary of the invention

[0003] The purpose of the present invention is to provide a text-driven style transfer method based on cross-space regulation, which uses natural language (text) to describe and characterize the target style and transfers it to the content image.

[0004] The technical solution to achieve the purpose of the present invention is: a text-driven style transfer method based on cross-space adjustment, comprising:

[0005] Step 1: Use the pre-trained deep convolutional neural network VGG-19 to extract content features from any given photo to determine the content structure information in the generated image;

[0006] Step 2: Use the pre-trained CLIP text encoder to extract style features from any given text prompt to determine the artistic style information in the generated image;

[0007] Step 3: construct a cross-space adjustment module to bridge the gap between the VGG feature space and the CLIP feature space, and then merge the content features extracted from the photo and the style features extracted from the text and input them into the decoder;

[0008] Step 4: construct a content loss function, a directional CLIP loss function, a block-to-block CLIP loss function, and a CLIP-based style contrast loss function to constrain the generation process of the style transfer network, and finally obtain an artistic image that matches the target content and target style.

[0009] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the above-mentioned text-driven style transfer method based on cross-space adjustment is implemented.

[0010] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned text-driven style transfer method based on cross-space adjustment.

[0011] A computer program product includes a computer program, which implements the above-mentioned text-driven style transfer method based on cross-space adjustment when executed by a processor.

[0012] The text-driven style transfer method based on cross-space adjustment provided by the present invention can transfer the artistic style described in the text to everyday photos to create new artistic images. Compared with the prior art, its beneficial effects are as follows:

[0013] (1) Compared with other style transfer methods, the present invention proposes to use text rather than images to represent the target style, which can further enrich and deepen the connotation of style transfer theory, enhance the flexibility and innovation of creative generation of artistic images, and has important scientific significance and application value in many fields such as cultural creativity, film and television production, and industrial design.

[0014] (2) Compared with other style transfer methods, the present invention introduces a cross-space adjustment module to connect the two different feature spaces of VGG and CLIP, thereby combining the powerful ability of VGG-19 in maintaining spatial structure with the powerful ability of CLIP in text-image representation to generate higher quality artistic images.

[0015] (3) Compared with other style transfer methods, this invention uses a new CLIP-based style contrast loss. Under its guidance, the generated artistic image will be closer to the corresponding input text description (positive sample) and away from other irrelevant text descriptions (negative samples). In this process, this method can learn better style representation. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the background technology, the drawings required for use in the embodiments of the present invention or the background technology will be described below.

[0017] Figure 1 Flow chart of a text-driven style transfer algorithm based on cross-space adjustment in one embodiment.

[0018] Figure 2 The figure is a flowchart of a cross-space adjustment strategy in one embodiment.

[0019] Figure 3 A schematic diagram of a flow chart of a style transfer network optimization solution in one embodiment. DETAILED DESCRIPTION

[0020] Compared with images, natural language is a more direct, universal and flexible way for humans to convey information. Based on this, the present invention provides a text-driven style transfer method, which uses natural language (text) to describe and characterize the target style and transfer it to the content image. This goal is mainly achieved based on a cross-space adjustment strategy, which achieves the fusion of content information and style information by bridging the difference between the image feature space and the text feature space, and finally generates a new artistic image.

[0021] The technical solutions in the embodiments of the present invention will be described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] like Figure 1 As shown, a text-driven style transfer method based on cross-space regulation is used to transfer the artistic style described in the text to everyday photos to create new artistic images. The method includes:

[0023] S1, using the pre-trained deep convolutional neural network VGG-19 to extract content features from any given photo (also called content image) to determine the content structure information in the generated image;

[0024] S2, using the pre-trained CLIP text encoder to extract style features from any given text prompt to determine the artistic style information in the generated image;

[0025] S3, construct a cross-space adjustment module to bridge the gap between the VGG feature space and the CLIP feature space, and then fuse the content features extracted from the photo and the style features extracted from the text and input them into the decoder;

[0026] S4, construct content loss function L c , directional CLIP loss function L dir , patch-to-patch CLIP loss function L p2p And the CLIP-based style contrast loss function L sc , which is used to constrain the generation process of the style transfer network, and finally obtain an artistic image that matches the target content and target style.

[0027] The text-driven style transfer method based on cross-space adjustment only needs to provide text description without preparing style images in advance, which is more flexible and convenient. In addition, the method is not limited to existing paintings, allowing individuals to use their imagination to depict and create more innovative artistic images.

[0028] Several optional methods are also provided below, but they are not intended to be additional limitations on the above-mentioned overall solution, but are merely further supplements or preferences. Under the premise that there are no technical or logical contradictions, each optional method can be combined with the above-mentioned overall solution separately, and multiple optional methods can also be combined.

[0029] Optionally, the text-driven style transfer method based on cross-space adjustment uses images in the MS-COCO dataset as target content and text in the ArtEmis dataset as target style to train the network. MS-COCO is a large-scale image dataset containing 83,000 pictures of everyday objects and people; ArtEmis is a large-scale text dataset containing 455,000 manually generated style attributes and explanatory texts about 80,000 WikiArt art paintings.

[0030] Specifically, S1 uses the deep neural network VGG-19 pre-trained on the classification task of the ImageNet dataset as an image encoder to extract content structure information in the content image (photo):

[0031] f c =E VGG (I c )

[0032] Where: E VGG represents the VGG-19 network, I c Represents the content image. VGG-19 maps the content image to the 4-dimensional VGG feature space (fc ∈R B×H×W×C , B represents the batch size, H represents the height, W represents the width, and C represents the number of channels), which helps to preserve its spatial structure information.

[0033] Specifically, S2 uses CLIP, a large-scale image-text pre-training model based on contrastive learning, as a text encoder to extract artistic style information from text descriptions:

[0034]

[0035] Where: represents the CLIP text encoder, T s Represents the input text. The CLIP text encoder maps the input text to a 2-dimensional CLIP feature space (f s ∈R B×N , B represents the batch size, and N represents the dimension of the output), and does not have the spatial structure information like image features.

[0036] Specifically, the cross-space adjustment strategy in S3 is as follows: Figure 2 As shown, including:

[0037] S301, construct a learnable cross-space adjustment module CSM to bridge the difference between the VGG feature space and the CLIP feature space. The cross-space adjustment module CSM consists of three fully connected layers. The first two layers have 256 units each, and the third layer has 512 units.

[0038] S302, extracting the text feature f from S2 s As the input of the cross-space adjustment module CSM, it outputs the adjusted text features

[0039]

[0040] S303, adjusting the text features Expand the dimension and compare it with the image feature f c Multiply them together to get the feature f that combines content information and style information cs :

[0041]

[0042] Where: Γ represents the dimension expansion operation, From the dimension R B×C Expanded to dimension R B×1×1×C , and image features f c ∈R B×H×W×C Multiply.

[0043] S304, the feature f that combines the content information and the style information cs Input decoder D to generate new artistic image I cs :

[0044] I cs :=D(f cs )

[0045] Combining the powerful ability of VGG-19 in maintaining spatial structure and the powerful ability of CLIP in text-image representation, the two are used as the image encoder and text encoder of the proposed model to extract content features and style features respectively.

[0046] Considering the significant differences between VGG-19 and CLIP in network structure, training data, and training objectives, the content features and style features extracted by them will be in two different feature spaces. To solve this problem, a learnable cross-space adjustment module is introduced to map content features and style features to the same feature space, thereby achieving the fusion of content information and style information.

[0047] Construct the content loss function L in the training phase c , directional CLIP loss function L dir , patch-to-patch CLIP loss function L p2p And the CLIP-based style contrast loss function L sc , constrained to generate image I cs Study the target content and target style.

[0048] The style transfer network optimization scheme in S4 is as follows Figure 3 As shown, including:

[0049] S401, construct content loss function L c To constrain the similarity between the generated image and the input photo in terms of content structure, the calculation formula is as follows:

[0050]

[0051] Where: I c I represents the input photo; cs Represents the generated image; Represents the i-th layer of the VGG-19 network; L represents the number of VGG-19 layers used, specifically including the Relu4_1 and Relu5_1 layers.

[0052] S402, construct the directional CLIP loss function L dir The generated image is constrained from a global perspective to have an artistic style that matches the input text. The calculation formula is as follows:

[0053]

[0054]

[0055]

[0056] Where: Represents CLIP text encoder; stands for CLIP image encoder; T s Represents the input text (also called target text), describing the target style; T src represents the source text, which is simply set to "Photo". ΔT represents the direction of change of the text in the CLIP feature space (the change from the source text to the target text); ΔI represents the direction of change of the image in the CLIP feature space (the change from the content image to the generated image). This loss function constrains the content image I c And generate image I cs The vector formed in CLIP space is consistent with the source text T src and the target text T s The vectors formed in the CLIP space are parallel, and the learning of the target style is completed in this process.

[0057] S403, construct a patch-to-patch CLIP loss function L p2p To further constrain the generated image from a local perspective to have an artistic style that matches the input text, the calculation formula is as follows:

[0058]

[0059]

[0060]

[0061] Where: and ΔI represents the i-th image patch in the generated image and the content image respectively; N represents the number of image patches; aug(·) is a random perspective augmentation operation, which can help the style transfer network to show more vivid and diverse texture effects. i represents the direction of change of the i-th image patch in the CLIP feature space (the change from the i-th content image patch to the i-th generated image patch). This loss function further constrains each local patch in the generated image to have the target style.

[0062] S404, construct a CLIP-based style contrast loss function Lsc To learn a better style representation, the calculation formula is as follows:

[0063]

[0064]

[0065] Where: and Respectively represent CLIP image encoder and CLIP text encoder; T k Represents the target text T s Some random texts other than T are used as negative samples in contrastive learning; s is a positive sample in contrastive learning; <·, ·> represents the inner product between two normalized feature vectors, which is used to measure their similarity; exp represents an exponential function with the natural constant e as the base; τ is a temperature hyper-parameter used to control the pull and push strength in contrastive learning; K represents the number of negative samples; p represents the features extracted from the anchor sample; p + represents the features extracted from the positive sample; p - Represents the features extracted from negative samples. There are usually multiple negative samples. Represents the features of the kth negative sample.

[0066] S405, under the joint constraint of the above loss function, optimizing the style transfer network so that it can generate an artistic image that has both the artistic style in the text description and the content structure in the input photo.

[0067] The text-driven style transfer method based on cross-space adjustment provided by the present invention can transfer the artistic style described in the text to everyday photos, create new artistic images, and enhance the flexibility and innovation of creative generation of artistic images. It has important scientific significance and application value in many fields such as cultural creativity, film and television production, and industrial design.

[0068] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0069] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A text-driven style transfer method based on cross-space adjustment, characterized in that: include: Step 1: Use the pre-trained deep convolutional neural network VGG-19 to extract content features from any given photo to determine the content structure information in the generated image; Step 2: Use the pre-trained CLIP text encoder to extract style features from any given text prompt to determine the artistic style information in the generated image; Step 3: construct a cross-space adjustment module to bridge the gap between the VGG feature space and the CLIP feature space, and then merge the content features extracted from the photo and the style features extracted from the text and input them into the decoder; Step 4: construct a content loss function, a directional CLIP loss function, a block-to-block CLIP loss function, and a CLIP-based style contrast loss function to constrain the generation process of the style transfer network, and finally obtain an artistic image that matches the target content and target style.

2. The text-driven style transfer method based on cross-space adjustment as claimed in claim 1, characterized in that A learnable cross-space adjustment module is introduced to map content features and style features to the same feature space to achieve the fusion of content information and style information; the cross-space adjustment module is a three-layer fully connected neural network, the first two layers have 256 units each, and the third layer has 512 units.

3. The text-driven style transfer method based on cross-space adjustment as claimed in claim 1, characterized in that The content loss function L c It is used to constrain the similarity between the generated image and the input photo in terms of content structure. The calculation formula is as follows: Where: I c Represents input photo; I cs Represents the generated image; Represents the i-th layer of the VGG-19 network; L represents the number of VGG-19 layers used, specifically including the Relu4_1 and Relu5_1 layers.

4. The text-driven style transfer method based on cross-space adjustment as claimed in claim 1, characterized in that The directional CLIP loss function L dir It is used to constrain the generated image from a global perspective to have an artistic style that matches the input text. The calculation formula is as follows: Where: stands for CLIP text encoder, represents the CLIP image encoder, T s Represents input text, T src represents the source text, ΔT represents the direction of change of the text in the CLIP feature space, and ΔI represents the direction of change of the image in the CLIP feature space; this loss function constrains the content image I c And generate image I cs The vector formed in CLIP space is consistent with the source text T src and the target text T s The vectors formed in the CLIP space are parallel, and the learning of the target style is completed in this process.

5. The text-driven style transfer method based on cross-space adjustment as claimed in claim 1, characterized in that: The block-to-block CLIP loss function L p2p It is used to generate an image from a local perspective with an artistic style that matches the input text. The calculation formula is as follows: Where: and represents the i-th image block in the generated image and the content image respectively; N represents the number of image blocks; aug(·) represents the random perspective enhancement operation; ΔI i Indicates the change direction of the i-th image block in the CLIP feature space.

6. The text-driven style transfer method based on cross-space adjustment as claimed in claim 1, characterized in that: The CLIP-based style contrast loss function L sc Used to learn better style representation, the calculation formula is as follows: Where: and Respectively represent CLIP image encoder and CLIP text encoder; T k Represents the target text T s Random text other than T is used as negative sample in contrastive learning; s is a positive sample in contrastive learning; <·, ·> represents the inner product between two normalized feature vectors; exp represents an exponential function with the natural constant e as the base; τ is a temperature hyperparameter used to control the pull and push strength in contrastive learning; K represents the number of negative samples; p represents the feature extracted from the anchor sample; p + represents the features extracted from the positive sample; p - represents the features extracted from negative samples, Represents the features of the kth negative sample.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the text-driven style transfer method based on cross-space adjustment as described in any one of claims 1-6 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the text-driven style transfer method based on cross-space adjustment as described in any one of claims 1 to 6 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the text-driven style transfer method based on cross-space adjustment described in any one of claims 1 to 6 is implemented.

Images