Content semantic separation image style representation learning method and device based on text inversion

Abstract

The invention discloses a content semantic separation image style representation learning method and device based on text inversion, belongs to the field of image style migration, and achieves the purpose of separating content semantic information of a reference style image from learned style text representation. According to the method provided by the invention, style-invariant content destruction data enhancement and content-invariant style transformation data enhancement are respectively carried out on a reference style image, text representation learning based on text inversion is carried out on the two groups of images after data enhancement and the reference style image, and corresponding text representation regularization loss is constructed. And finally obtaining the content semantic-independent style text representation of the reference style image, thereby alleviating the problem of content leakage.

Description

Technical Field

[0001] The present invention relates to the field of image style transfer, and in particular to a content semantic separation image style representation learning method and device based on text inversion. Background Art

[0002] In recent years, large-scale pre-trained text-to-image diffusion models have made progress in the field of text-to-image generation, and can generate realistic and diverse image content driven by text. At present, there is a type of style transfer method based on text-to-image diffusion models, which usually first learns style text representations based on textual inversion technology to characterize the style of reference style images. After learning fine-tuning, the style text representation is used as a condition to control the image generation process to achieve style transfer. However, the style text representation learned by existing methods often contains the content semantic information of the reference style image, which makes the stylized image generated by the style text representation easily mixed with the content semantic interference of the reference style image. This problem is called the content leakage problem in style representation learning. Therefore, how to separate the content semantic information from the style text representation in the process of learning style text representation based on text inversion is the key to solving this problem. Summary of the invention

[0003] The purpose of the present invention is to address the deficiencies of the prior art and to propose a method and device for learning image style representation based on text inversion for content semantic separation, so as to solve the problem of content semantic leakage in the process of image style representation learning.

[0004] The objective of the present invention is achieved through the following technical solutions: In a first aspect, the present invention provides a content semantic separation image style representation learning method based on text inversion, the method comprising the following steps: (1) Obtain the original reference style image and extract style features and content features based on the image style encoder network and image content encoder network respectively; (2) Data augmentation is performed on the original reference style image to preserve style features while destroying content semantics, and style features are extracted based on the image style encoder network; (3) Perform style transformation data enhancement on the original reference style image while keeping the content semantics unchanged, and extract content features based on the image content encoder network; (4) Based on the style adapter network and the content adapter network, the style features and content features obtained in steps (1) to (3) are mapped to the text space to obtain the corresponding style text representation and content semantic text representation; these representations are injected into the text graph diffusion model through the text encoder and the cross attention mechanism, and trained based on the text inversion method; (5) Based on the image style encoder network, the style features of the reference style image are extracted. Based on the trained style adapter network, the final style text representation of the reference style image is obtained, and the style text representation is used to perform image style transfer.

[0005] Furthermore, in step (2), data enhancement of the original reference style image includes the following steps: (2-1) Setting the size of the image block according to the pixel size of the original reference style image and the input requirements of the adopted image style encoder; (2-2) Based on the set image block size, the original reference style image is segmented in a raster scan order from upper left to lower right, and the image block order is marked; (2-3) The image blocks are rearranged in a random manner, destroying the content semantic information of the original reference style image while retaining the style information of the original reference style image, and re-forming a set of data enhanced images.

[0006] Furthermore, in step (3), data enhancement is performed on the original reference style image to obtain a set of images that retain the content semantic information of the original reference style image but change the style information of the original reference style image, specifically including the following method: a. Perform grayscale conversion on the original reference style image to obtain the corresponding grayscale image, and then perform Gaussian blur to the required degree.

[0007] b. Apply a random color shift to the original reference style image, followed by a Gaussian blur of the desired degree.

[0008] Furthermore, style information includes global color distribution and local texture details.

[0009] Furthermore, the style features of the original reference style image are extracted based on the image style encoder network, and the style features are passed through a style adapter network to obtain the final style text representation, which eliminates the content semantic interference in the original reference style image.

[0010] Furthermore, in step (4), the training process and loss function are designed as follows: Using the style text representation and content semantic text representation obtained from the original reference style image as conditions, the original reference style image is trained based on the text inversion method, and its loss function is the reference style image loss; Using the style text representation obtained based on the style features in step (2) as a condition, the data-augmented image in step (2) is trained based on the text inversion method, and its loss function is the overall style loss; Using the content semantic text representation obtained based on the content features in step (3) as a condition, the image enhanced by the data in step (3) is trained based on the text inversion method, and its loss function is the overall content loss; The style text representation obtained by using the style consistency loss to constrain the original reference style image is close to the style text representation of the image obtained by data augmentation in step (2), and far from the style text representation of the image obtained by data augmentation in step (3); the content semantic text representation obtained by using the content consistency loss to constrain the original reference style image is close to the content semantic text representation based on the image obtained by data augmentation in step (3).

[0011] Furthermore, in step (5), the style text representation obtained using the reference style image is Conditional text-to-image style transfer; specifically, construct text prompts Describe the content you want to generate and Append to the text prompt to construct the text prompt =“ , "or" in the style of ",Will Feed it into a pre-trained text-to-image diffusion model for text-to-image generation; or feed it into a reference content image The corresponding inversion noise is obtained after DDIM inversion , use this inverted noise as the initial noise and provide a style text representation As a condition for image style transfer.

[0012] In a second aspect, the present invention also provides a content semantic separation image style representation learning device based on text inversion, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, the content semantic separation image style representation learning method based on text inversion is implemented.

[0013] In a third aspect, the present invention further provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the content semantic separation image style representation learning method based on text inversion.

[0014] In a fourth aspect, the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the content semantic separation image style representation learning method based on text inversion.

[0015] Beneficial effects of the present invention: The present invention can alleviate the content semantic interference in the reference style image in the process of style representation learning based on text inversion, thereby suppressing the content leakage problem that the content semantics of the reference style image are often mixed in the stylized images generated by the current method. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 A schematic diagram of a content semantic separation image style representation learning method based on text inversion provided by the present invention.

[0018] Figure 2 A schematic diagram of the stylized representation learning and training method based on text inversion provided by the present invention.

[0019] Figure 3 Schematic diagram of the style-invariant content-destruction data enhancement method and the content-invariant style-transformation data enhancement method provided by the present invention.

[0020] Figure 4 A schematic diagram of the text-to-image stylization generation result provided by the present invention.

[0021] Figure 5 A structural diagram of a content semantic separation image style representation learning device based on text inversion provided by the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with the accompanying drawings and implementation examples. It should be understood that the specific implementation examples described herein are only used to explain the present invention and are not used to limit the present invention.

[0023] In order to suppress the content leakage problem in the learning of stylized representation based on text inversion, the present invention proposes a method for learning stylized representation of images based on content semantic separation based on text inversion, which achieves the goal of separating the content semantic information of the reference style image from the learned style text representation. The proposed method performs style-invariant content destruction data enhancement and content-invariant style transformation data enhancement on the reference style image respectively, and then performs text representation learning based on text inversion on the two sets of data-enhanced images and the reference style image, and constructs the corresponding text representation regularization loss, and finally obtains the content semantics-independent style text representation of the reference style image.

[0024] like Figure 1 As shown, the present invention proposes a content semantic separation image stylized representation learning method based on text inversion, and the specific steps are as follows: (1) Perform the following operations on the reference style image: Figure 3 The style-invariant content-destroying data enhancement shown in the figure, hereinafter referred to as data enhancement A, obtains a set of images whose style representations are the same as the original reference style images, but the content semantics are destroyed; the specific implementation of data enhancement A is as follows: the original reference style image The input image is cut into patches and the cut image patches are reassembled.

[0025] (1-1) In general, the size of the patch should match the level of detail of the image. In this task, the purpose of splitting the patch is to destroy the content structure of the image while retaining and capturing the local style features of the image. The specific choice depends on the size of the reference image and the input requirements of the image style encoder. 16×16, 32×32 or 64×64 pixels can be considered.

[0026] (1-2) According to the preset image block size, the original reference style image is segmented in a raster scan order from the upper left to the lower right. Specifically, this process involves dividing the entire image into a number of image blocks of fixed size. During the segmentation process, in order to maintain the order information of the image blocks, each image block is marked. These marks are usually numbered based on the position of the image block in the entire image, for example, starting from the first block in the upper left corner, the serial number of each block is marked row by row.

[0027] (1-3) Assume that the original image is divided into N patches, where the index of each patch can be expressed as , where i ranges from i=1,2,…,N. The random number generation algorithm in the computer is used to randomly rearrange the patch sequence, and then the patch blocks are reassembled into an image according to the disrupted sequence. The overall structure of the image is disrupted, but the style information of the original image, such as global color distribution and local texture details, is still retained.

[0028] (2) Build an image style encoder network , used for style feature extraction, here we can use the pre-trained CLIP-ViT image encoder or DINO-ViT model as the image style encoder network Specifically, the [cls-token] feature vector output by the last layer after the image is fed into the network can be extracted as the style feature.

[0029] The input image is the m images in step (1) whose style representation remains unchanged but whose content semantics are destroyed. The process can be described as:

[0030] Since the output is a set of feature vectors of images, in order to preserve the style features of the entire image, it is necessary to integrate the style features of each image to obtain a unified style feature of this set of images. The specific method is to transform the style features of each image Add them together and take the average, the operation is as follows:

[0031] (3) Building a learnable style adapter network , the style features Map it to the text space and get the style text representation.

[0032] (3-1) The style adapter network transforms style features through multiple learnable layers, usually including fully connected layers, normalization layers, or modules based on self-attention mechanisms. Convert to a unified style text representation . It can be expressed as:

[0033] (3-2) Through the text encoder and cross-attention mechanism of a multimodal model (such as CLIP) (query vector Q, key vector K, value vector V) Injected into a pre-trained graph diffusion model (such as Stable Diffusion) as a condition.

[0034] (4) For the original reference style image Proceed as Figure 3 The content-invariant style transformation data enhancement shown, hereinafter referred to as data enhancement B, obtains a set of images whose style representations are different from the original reference style images, but the content semantics remain consistent; the methods are as follows, which can be flexibly adopted according to actual conditions: (4.1) The reference style image is converted to grayscale to obtain a grayscale image. The image can then be Gaussian blurred to a certain extent. Through this process, the texture details in the image are blurred, but the overall structure is retained.

[0035] (4.2) Perform a random color change on the reference style image, and then perform a certain degree of Gaussian blur. The color change method can be achieved using ColorJitter or color histogram matching. Then, a certain degree of Gaussian blur can be performed on the image.

[0036] A set of n images obtained by data augmentation using the above method , which actually retains the content semantic information of the original reference style image and changes the style information of the original reference style image, such as global color distribution and local texture details.

[0037] (5) Build an image content encoder network , used to extract content features, here we can use the pre-trained CLIP-ViT image encoder or DINO-ViT model as the image content encoder network Specifically, the key value feature vector of the last self-attention layer of the network is extracted as the content feature, which can be expressed as:

[0038] Since the output is a set of image feature vectors, in order to preserve the content features of the entire image, it is necessary to integrate the content features of each image to obtain the unified content features of this set of images. The specific method is to transform the content features of each image Add them together and take the average, the operation is as follows:

[0039] (6) Building a learnable content adapter network , the content features Map to text space to get the content text representation , after passing through the text encoder and the cross attention mechanism (query vector Q, key vector K, value vector V) Inject it into the pre-trained Wensheng graph diffusion model as a condition. The specific process is as in step (3).

[0040] (7) For the reference style image , using the image style encoder network constructed in steps (2) and (5) and image content encoder network Extract the corresponding style features respectively and content features , the operation is as follows:

[0041] (8) Use the style adapter network from step (3) and step (6) and content adapter networks , mapping the style and content features to the text space, and obtaining the reference style images Style text representation and content semantic text representation , and then through the text encoder and cross attention mechanism (query vector Q, key vector K, value vector V) and Inject it into the pre-trained Wensheng graph diffusion model as a condition. For the specific process, refer to steps (3) and (6).

[0042] (9) Based on These conditions construct loss functions to train the style adapter network and content adapter network based on the text inversion method, such as Figure 2 As shown: The reference style image loss is and As a condition, for the reference style image The training based on the text inversion method is constructed using the diffusion model training loss, which is expressed as follows:

[0043] in To pre-train the graph diffusion model, such as Stable Diffusion, For pre-trained text encoders, such as CLIP's text encoder, For reference style images The noisy image obtained after t steps of noise addition through the diffusion model.

[0044] The overall style loss is For the condition, This set of images is trained based on the text inversion method, constructed using the diffusion model training loss, which is expressed as follows:

[0045] in For images The noisy image obtained after t steps of noise addition through the diffusion model.

[0046] The overall content loss is For the condition, This set of images is trained based on the text inversion method, constructed using the diffusion model training loss, which is expressed as follows:

[0047] in For images The noisy image obtained after t steps of noise addition through the diffusion model.

[0048] Style consistency loss constraint and Similar to:

[0049] in is the image obtained by data enhancement B in step (4) The text representation is fed into the image style encoder and the style adapter network.

[0050] Content consistency loss constraint and Similar to:

[0051] (10) After the training is completed, users can use Conditional text-to-image style transfer. Specifically, users can construct text prompts Describe the content you want to generate and Append to the text prompt to construct the text prompt =“ , "or" in the style of ",Will Feed it into a pre-trained text-to-image diffusion model for text-to-image style transfer; or feed it into a reference content image The corresponding inversion noise is obtained after DDIM inversion , use this inverted noise as the initial noise, and give the style text representation Image style transfer is performed for conditions. For example, Figure 4 As shown, the user inputs the text: "A house covered with ice and snow.", and obtains a text-to-image generation result that includes the style of the reference style image and the text content input by the user. Compared with the traditional method, the text-guided generation result obtained by the present invention can effectively suppress the semantic content of the reference style image content mixed in the stylized image.

[0052] Corresponding to the aforementioned embodiment of a method for learning image style representation through content semantic separation based on text inversion, the present invention also provides an embodiment of a device for learning image style representation through content semantic separation based on text inversion.

[0053] See also Figure 5 An embodiment of the present invention provides a content semantic separation image style representation learning device based on text inversion, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a content semantic separation image style representation learning method based on text inversion in the above embodiment.

[0054] An embodiment of a content semantic separation image style representation learning device based on text inversion provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From a hardware perspective, if Figure 5 As shown, it is a hardware structure diagram of a device with data processing capability in which a content semantic separation image style representation learning device based on text inversion provided by the present invention is located, except Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0055] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0056] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0057] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a content semantic separation image style representation learning method based on text inversion in the above embodiment is implemented.

[0058] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0059] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the content semantic separation image style representation learning method based on text inversion.

[0060] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modification and change made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A content semantic separation image style representation learning method based on text inversion, characterized in that: The method comprises the following steps: (1) Obtain the original reference style image and extract style features and content features based on the image style encoder network and image content encoder network respectively; (2) Data enhancement is performed on the original reference style image, retaining the style features while destroying the content semantics, and extracting style features based on the image style encoder network; (3) Perform style transformation data enhancement on the original reference style image while keeping the content semantics unchanged, and extract content features based on the image content encoder network; (4) Based on the style adapter network and the content adapter network, the style features and content features obtained in steps (1) to (3) are mapped to the text space to obtain the corresponding style text representation and content semantic text representation; these representations are injected into the text graph diffusion model through the text encoder and the cross attention mechanism, and trained based on the text inversion method; (5) Based on the image style encoder network, the style features of the reference style image are extracted. Based on the trained style adapter network, the final style text representation of the reference style image is obtained, and the style text representation is used to perform image style transfer.

2. The method for learning image style representation based on content semantic separation and text inversion according to claim 1 is characterized in that: In step (2), data augmentation of the original reference style image includes the following steps: (2-1) Setting the size of the image block according to the pixel size of the original reference style image and the input requirements of the adopted image style encoder; (2-2) Based on the set image block size, the original reference style image is segmented in a raster scan order from upper left to lower right, and the image block order is marked; (2-3) The image blocks are rearranged in a random manner, destroying the content semantic information of the original reference style image while retaining the style information of the original reference style image, and re-forming a set of data enhanced images.

3. The method for learning image style representation based on content semantic separation and text inversion according to claim 1 is characterized in that: In step (3), data enhancement is performed on the original reference style image to obtain a set of images that retain the content semantic information of the original reference style image but change the style information of the original reference style image, specifically including the following methods: a. Perform grayscale conversion on the original reference style image to obtain the corresponding grayscale image, and then perform Gaussian blur to the required degree; b. Apply a random color shift to the original reference style image, followed by a Gaussian blur of the desired degree.

4. The method for learning image style representation based on content semantic separation and text inversion according to claim 2 or 3, characterized in that: Style information includes global color distribution and local texture details.

5. The method for learning image style representation based on content semantic separation and text inversion according to claim 1, characterized in that: Based on the image style encoder network, the style features of the original reference style image are extracted, and the style features are passed through a style adapter network to obtain the final style text representation, which eliminates the content semantic interference in the original reference style image.

6. The method for learning image style representation based on content semantic separation and text inversion according to claim 1, characterized in that: In step (4), the training process and loss function are designed as follows: Using the style text representation and content semantic text representation obtained from the original reference style image as conditions, the original reference style image is trained based on the text inversion method, and its loss function is the reference style image loss; Using the style text representation obtained based on the style features in step (2) as a condition, the data-augmented image in step (2) is trained based on the text inversion method, and its loss function is the overall style loss; Using the content semantic text representation obtained based on the content features in step (3) as a condition, the image enhanced by the data in step (3) is trained based on the text inversion method, and its loss function is the overall content loss; The style text representation obtained by using the style consistency loss to constrain the original reference style image is close to the style text representation of the image obtained by data augmentation in step (2), and far from the style text representation of the image obtained by data augmentation in step (3); the content semantic text representation obtained by using the content consistency loss to constrain the original reference style image is close to the content semantic text representation based on the image obtained by data augmentation in step (3).

7. The method for learning image style representation based on content semantic separation and text inversion according to claim 1, characterized in that: In step (5), the style text representation obtained using the reference style image Conditional text-to-image style transfer; specifically, construct text prompts Describe the content you want to generate and Append to the text prompt to construct the text prompt =" , "or" in the style of ",Will Feed it into a pre-trained text-to-image diffusion model for text-to-image generation; or feed it into a reference content image The corresponding inversion noise is obtained after DDIM inversion , use this inverted noise as the initial noise and provide a style text representation As a condition for image style transfer.

8. A content semantic separation image style representation learning device based on text inversion, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a content semantic separation image style representation learning method based on text inversion is implemented as described in any one of claims 1-7.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a content semantic separation image style representation learning method based on text inversion is implemented as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for learning image style representation based on content semantic separation and text inversion according to any one of claims 1 to 7 is implemented.

Images