Art image style migration method, device, equipment, medium and product

Abstract

The invention discloses an art image style migration method and device, equipment, a medium and a product, and relates to the field of image processing, and the method comprises the steps: obtaining a to-be-processed style picture and a to-be-processed content picture; performing style text filtering on the style picture to be processed to obtain a pure content description text; the style text filtering comprises style shift-out and picture-text distance comparison; creating a single pseudo word and expanding the single pseudo word based on a time step stage and a sub-attention layer to obtain a style pseudo word set; optimizing the style pseudo-word set by utilizing inversion of a diffusion model according to the pure content description text to obtain style representation of the style picture to be processed; and taking the style representation of the style picture to be processed as style guidance, determining content guidance according to the content picture, and determining a stylization result by utilizing denoising of the diffusion model. The stylization effect can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and in particular to an artistic image style transfer method and device, equipment, medium and product. BACKGROUND

[0002] In today's society, with the growing demand for spiritual and cultural needs, artistic creation is gradually moving from the professional field to the public. More and more people hope to express their emotions, aesthetics and creativity through artistic creation, especially in image creation. The diversity of artistic style makes it an important means of expression. At the same time, traditional artistic style creation often relies on long-term professional training and rich artistic accumulation, which has a high threshold for the general public, and personalized artistic creation is difficult to popularize. With the development of artificial intelligence technologies such as deep learning, artistic image style transfer technology has emerged, which can combine any image with a specific artistic style, allowing ordinary users to easily create images with unique styles. This technology not only lowers the threshold for artistic creation, but also brings new possibilities to the fields of artistic education, visual design, and cultural dissemination, and has broad application prospects.

[0003] Artistic image style transfer can be divided into style transfer based on traditional methods and style transfer based on diffusion models. Style transfer based on traditional methods generally uses an encoder-decoder architecture, uses statistical quantities of style picture features to explicitly define style, and uses deep features of content pictures as content features. Due to the explicit style definition, this method cannot effectively decouple the style and content of the picture, resulting in results that do not well reflect the style of the style picture. In addition, this method requires a large number of style-content picture pairs as training data, and the consumption of computing resources is also very large. Style transfer based on diffusion models solves the data consumption problem faced by traditional methods. Based on the pre-training of diffusion models, a small amount of samples can be fine-tuned, prompted, and additional control conditions injected to make the diffusion model output controllable results, making it exhibit strong capabilities in few-shot, one-shot, and even zero-shot image generation, greatly improving the quality of generated images. At present, style transfer algorithms based on diffusion models have made great progress, and they have been deployed as core technologies in various downstream tasks. For example, in personalized artistic creation, users only need to input an image and a target style description to quickly generate personalized artistic works, greatly reducing the creation threshold; in cross-modal (text-to-image) creation, text-driven diffusion style transfer methods can generate images with corresponding artistic styles based on user input text descriptions, realizing text-to-artistic image creation.

[0004] In summary, the diffusion-based style transfer method is a research direction with wide influence, which not only promotes the further development of computer vision and image generation technology in artistic expression, but also greatly expands the application boundary of artificial intelligence in creative design and cultural communication and other fields. Although the style transfer technology has made significant progress, the diffusion model-based method is becoming an important research hotspot in this field due to its excellent image synthesis quality and style expression ability. Its commercial application is increasingly mature and is gradually integrated into people's daily life and creative practice. However, the existing diffusion model-based style transfer research cannot effectively decouple the style and content of the style picture, resulting in the appearance of the content of the style picture or the style of the content picture in the stylized result, which reduces the quality of the stylized result. In addition, the results generated by the current diffusion model-based style transfer method are often too biased towards the style of the style picture or the content of the content picture, which reduces the visual harmony of the results. Therefore, the content and style trade-off of the stylized result is also a problem to be solved in style transfer. SUMMARY

[0005] The purpose of the present application is to provide an artistic image style transfer method, device, equipment, medium and product, which can improve the stylization effect.

[0006] To achieve the above-mentioned purpose, the present application provides the following solutions:

[0007] In a first aspect, the present application provides an artistic image style transfer method, comprising:

[0008] obtaining a style picture and a content picture to be processed;

[0009] performing style text filtering on the style picture to be processed to obtain a pure content description text; the style text filtering includes style removal and picture-text distance comparison;

[0010] creating a single pseudo word and expanding the single pseudo word based on a time step stage and a split attention layer to obtain a style pseudo word set;

[0011] optimizing the style pseudo word set according to the pure content description text using the inversion of the diffusion model to obtain a style representation of the style picture to be processed;

[0012] using the style representation of the style picture to be processed as a style guide, determining a content guide according to the content picture, and determining a stylized result using the denoising of the diffusion model.

[0013] In an embodiment, the style text filtering is performed on the style picture to be processed to obtain a pure content description text, specifically comprising:

[0014] The style picture to be processed is used to generate picture description text by using a multi-modal large language model;

[0015] The picture description text is split into a plurality of text tokens by using a tokenizer;

[0016] A real photo is obtained from the style picture to be processed by using a diffusion model;

[0017] A style-free style picture is determined by performing grayscale transformation on the real photo;

[0018] An image-text distance is calculated according to each text token, the style picture to be processed, and the style-free style picture, and the style attribute of each text is determined;

[0019] The style attribute is filtered according to the text token and the style attribute, and a pure content description text is obtained.

[0020] In an embodiment, a single pseudo word is created and expanded based on a time step stage and a split attention layer to obtain a style pseudo word set, specifically including:

[0021] A single pseudo word is created;

[0022] The single pseudo word is copied multiple times and expanded according to a time step stage to obtain a time step expanded pseudo word set;

[0023] Each element in the time step expanded pseudo word set is copied and expanded according to a split attention layer to obtain a style pseudo word set.

[0024] In an embodiment, the style pseudo word set is optimized according to the pure content description text by using an inverse diffusion model to obtain a style representation of the style picture to be processed, specifically including:

[0025] The pure content description text and the style pseudo word set are concatenated to obtain a spliced text;

[0026] The spliced text is encoded to obtain a first text word embedding;

[0027] A gradient is generated according to the first text word embedding based on an inverse diffusion model;

[0028] The style pseudo word set is optimized according to the gradient to obtain a style representation of the style picture to be processed.

[0029] In an embodiment, the style representation of the style picture to be processed is used as a style guide, a content guide is determined according to a content picture, and a denoising diffusion model is used to determine a stylized result, specifically including:

[0030] encoding the text description of the content image to obtain a second text word embedding;

[0031] performing image encoding on the style image and the content image to be processed to obtain a neck feature;

[0032] performing adaptive instance normalization on the neck feature;

[0033] performing depth estimation on the content image to obtain a depth map;

[0034] encoding the depth map;

[0035] using the style representation of the style image to be processed as a style guide, using the second text word embedding, the neck feature after adaptive instance normalization, and the encoded depth map as a content guide, and using a diffusion model to perform denoising to obtain a stylized result.

[0036] In an embodiment, the loss function of the inversion of the diffusion model in the training process is a reconstruction loss.

[0037] In a second aspect, the present application provides an artistic image style transfer device, comprising:

[0038] an acquisition module configured to acquire a style image to be processed and a content image;

[0039] a style text filtering module configured to perform style text filtering on the style image to be processed to obtain a pure content description text; the style text filtering comprises style removal and image-text distance comparison;

[0040] a creation and expansion module configured to create a single pseudo word and expand the single pseudo word based on a time step stage and a split attention layer to obtain a style pseudo word set;

[0041] an optimization module configured to optimize the style pseudo word set according to the pure content description text using an inversion of a diffusion model to obtain a style representation of the style image to be processed;

[0042] a style transfer module configured to use the style representation of the style image to be processed as a style guide, determine a content guide according to the content image, and determine a stylized result using denoising of the diffusion model.

[0043] In a third aspect, the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the artistic image style transfer method.

[0044] In a fourth aspect, the present application provides a computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, implements the artistic image style transfer method.

[0045] In a fifth aspect, the present application provides a computer program product comprising a computer program which, when executed by a processor, implements the artistic image style transfer method.

[0046] According to the specific embodiments provided by the present application, the present application discloses the following technical effects:

[0047] The present application provides an artistic image style transfer method, device, equipment, medium and product, which filters the style text of the to-be-processed style picture to obtain a pure content description text; the style text filtering includes style removal and picture-text distance comparison; a single pseudo word is created and expanded based on a time step stage and a split attention layer to obtain a style pseudo word set; the style pseudo word set is optimized according to the pure content description text using the inversion of a diffusion model to obtain a style representation of the to-be-processed style picture; the style representation of the to-be-processed style picture is taken as a style guide, a content guide is determined according to a content picture, and a stylized result is determined using the denoising of the diffusion model. Through the style text filtering of style removal and picture-text distance comparison, the style of the style picture can be removed, ensuring that the model can effectively decouple the style and content of the style picture; by expanding the single pseudo word based on the time step and the split attention layer, the information capacity and representation ability of the pseudo word can be enhanced, so that it can learn the fine-grained style features in the style picture; through the inversion of the diffusion model, effective style decoupling and content decoupling can be achieved; thereby improving the stylization effect. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0049] Figure 1 The figure is for the training process of the artistic image style transfer method.

[0050] Figure 2 The figure is for the inference process of the artistic image style transfer method.

[0051] Figure 3 The figure is for the style text filtering method flowchart based on style picture style removal and picture-text distance comparison.

[0052] Figure 4A visual diagram of a style removal result; wherein Figure 4 (a) in FIG. 1 is a style picture schematic diagram, Figure 4 (b) in FIG. 1 is a style picture schematic diagram without style.

[0053] Figure 5 A multi-dimensional inversion method flowchart based on a diffusion model and divided by time steps and attention layers.

[0054] Figure 6 A style and content fusion method framework based on multiple condition controls.

[0055] Figure 7 A flowchart of an artistic image style transfer method provided by an embodiment of the present application.

[0056] Figure 8 A functional module schematic diagram of an artistic image style transfer device provided by another embodiment of the present application.

[0057] Figure 9 A structural schematic diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0059] The above purposes, features and advantages of the present application can be more obvious and easy to understand. The present application will be further described in detail below with reference to the drawings and specific embodiments.

[0060] In an exemplary embodiment, as shown in Figure 7 An artistic image style transfer method is provided, which is executed by a computer device, specifically, can be executed by a terminal or a server, or can be executed by a terminal and a server together. In the embodiments of the present application, the method is applied to a server as an example, and includes the following steps.

[0061] Step 701: obtaining a style picture and a content picture to be processed.

[0062] Step 702: performing style text filtering on the style picture to be processed to obtain pure content description text; the style text filtering includes style removal and picture-text distance comparison.

[0063] Step 703: create a single pseudo-word and expand the single pseudo-word based on time step stage and sub-attention layer to obtain a style pseudo-word set.

[0064] Step 704: optimize the style pseudo-word set according to the pure content description text by using the inversion of the diffusion model to obtain the style representation of the style picture to be processed.

[0065] Step 705: taking the style representation of the style picture to be processed as a style guide, determining a content guide according to a content picture, and determining a stylized result by using the denoising of the diffusion model.

[0066] Through style text filtering by style removal and picture-text distance comparison, the style of the style picture can be removed, ensuring that the model can effectively decouple the style and content of the style picture; by expanding the single pseudo-word based on time step and sub-attention layer, the information capacity and representation ability of the pseudo-word can be enhanced, so that it can learn the fine-grained style features in the style picture; through the inversion of the diffusion model, effective style decoupling and content decoupling can be achieved; thereby improving the effect of stylization.

[0067] In an exemplary embodiment, the style picture to be processed is subjected to style text filtering to obtain a pure content description text, specifically comprising:

[0068] The style picture to be processed is used to generate a picture description text by using a multi-modal large language model.

[0069] The picture description text is split into multiple text tokens by using a tokenizer.

[0070] The style picture to be processed is used to obtain a real photo by using a diffusion model.

[0071] The style picture without style is determined by performing gray scale transformation on the real photo.

[0072] The style attribute of each text is determined by performing image-text distance calculation according to each text token, the style picture to be processed and the style picture without style.

[0073] The pure content description text is obtained by performing style attribute filtering according to the text token and the style attribute.

[0074] In an exemplary embodiment, a single pseudo-word is created and expanded based on time step stage and sub-attention layer to obtain a style pseudo-word set, specifically comprising:

[0075] A single pseudo-word is created.

[0076] The single pseudo-word is copied multiple times and expanded according to time step stage to obtain a time step expanded pseudo-word set.

[0077] Each element in the time step extended pseudo word set is respectively copied and extended according to the attention layer, to obtain a style pseudo word set.

[0078] In an exemplary embodiment, the style pseudo word set is optimized according to the inversion of the diffusion model based on the pure content description text, to obtain a style representation of the style picture to be processed, specifically including:

[0079] The pure content description text and the style pseudo word set are concatenated to obtain a spliced text.

[0080] The spliced text is encoded to obtain a first text word embedding.

[0081] According to the first text word embedding, a gradient is generated based on the inversion of the diffusion model.

[0082] According to the gradient, the style pseudo word set is optimized to obtain a style representation of the style picture to be processed.

[0083] In an exemplary embodiment, the style representation of the style picture to be processed is taken as a style guide, a content guide is determined according to a content picture, and a diffusion model is used for denoising to determine a stylized result, specifically including:

[0084] The text description of the content picture is encoded to obtain a second text word embedding; the style picture to be processed and the content picture are image encoded to obtain a neck feature; the neck feature is adaptively instance normalized; the content picture is depth estimated to obtain a depth map; the depth map is encoded; the style representation of the style picture to be processed is taken as a style guide, the second text word embedding, the adaptively instance normalized neck feature and the encoded depth map are taken as a content guide, and the diffusion model is used for denoising to obtain a stylized result.

[0085] In an exemplary embodiment, the loss function of the inversion of the diffusion model in the training process is a reconstruction loss.

[0086] For the decoupling problem of style and content of style pictures, the application proposes a style text filtering method based on style picture style removal and picture-text distance comparison, aiming to automatically align the style and content of style pictures to the filtered pure content text and style pseudo-words using the inversion of the diffusion model to achieve effective style and content decoupling. In addition, the application also introduces a multi-dimensional inversion based on diffusion model time step and attention layer, which enables the pseudo-words to adapt to the spatiotemporal dynamic characteristics of the diffusion model in terms of time step and attention layer, to obtain fine-grained style representation. For the style and content trade-off problem of stylized results, the application proposes a style and content fusion method based on multiple condition control, which uses multiple content control conditions to control the content of the stylized results, aiming to balance the style and content of the stylized results. The method proposed in the application can effectively realize the decoupling of the style and content of the style picture, obtain fine-grained style representation, and output visually harmonious high-quality stylized pictures.

[0087] The application includes two parts: training phase and inference phase. In the training phase, the reconstruction loss is calculated using the inversion of the diffusion model to optimize the style pseudo-word set to learn the fine-grained style representation of the style picture; in the inference phase, the diffusion model accepts the optimized style pseudo-word set from the training phase as the style guide, and accepts multiple content controls as the content guide, to generate high-quality stylized pictures with style and content trade-off.

[0088] As shown in Figure 1 , the training phase steps are:

[0089] Step 1: Use a multi-modal large language model to generate an image description text for a style picture.

[0090] Step 2: Remove the style of the style picture to get a style-free style picture.

[0091] Step 3: Use image-text similarity calculation to distinguish the style or content attribute of each word in the image description text one by one, filter the style attribute words, and get the pure content description text of the style picture.

[0092] Step 4: Create a single pseudo-word and expand it into a style pseudo-word set corresponding to a specific diffusion model time step stage and attention layer.

[0093] Step 5: The style pseudo-word set and the pure content description text of the style picture obtained in step 3 are jointly input into the inversion of the diffusion model to get the prediction of the style picture.

[0094] Step 6: Calculate the reconstruction loss according to the style picture and the prediction of the style picture, and optimize the style pseudo-word set. The style pseudo-word set can decouple the fine-grained style representation from the style picture, which will be used as the style guide condition in the inference phase.

[0095] The purpose of the training stage is to decouple the fine-grained style of the style picture and learn it to the target pseudo-words (the inverse of the diffusion model) that is the style pseudo-words set as the style representation of the style picture.

[0096] As shown in Figure 2 , the inference stage steps are:

[0097] Step 1: Initialize random noise as the starting point of the inference stage diffusion process.

[0098] Step 2: Send the artificially generated semantic text condition, the content picture and the depth map of the content picture into the condition coding respectively to obtain the content guidance condition.

[0099] Step 3: The optimized style pseudo-words set from the training stage is used as the style guidance condition.

[0100] Step 4: Send the style guidance and the content guidance into the diffusion model to generate a high-quality stylized picture with style and content trade-off from the random noise.

[0101] In the inference stage, the optimized style pseudo-words set from the training stage is used as the style representation, and the coding information integrated with multiple conditions (i.e. semantic text condition, content picture) is used as the content representation; and the style representation and the content representation are fused in this stage to output a high-quality stylized result.

[0102] The method provided in the present application has three key modules, which are: a style text filtering method based on style picture style removal and picture-text distance comparison, a multi-dimensional inversion module based on diffusion model time step and attention layer, and a style and content fusion method based on multiple condition control. The following introduces the use method and steps of the three modules in the present application:

[0103] (1) Style text filtering method based on style picture style removal and picture-text distance comparison: In the research of artistic image style transfer based on diffusion model inversion, the most important problem is that the single pseudo-word in the traditional inversion method cannot effectively decouple the style and content of the target picture, resulting in style redundancy of the content picture and content residue of the style picture in the stylized result. Therefore, the present application automatically aligns the style and content of the style picture to the extended style pseudo-words set and the pure content description text by using the diffusion model inversion to solve this problem. The core lies in obtaining the pure content description text of the style picture, and the present application uses the style picture style removal and text-picture distance comparison to filter out the style text in the style picture description to obtain the pure content description text. The flow of the method is shown in Figure 3 .

[0104] The inversion of the diffusion model aims to create a single pseudo-word to learn a specific concept or style in the target image. The principle is to input the single pseudo-word as a text condition into the diffusion model during training, and optimize the single pseudo-word through the reconstruction loss of the image predicted by the diffusion model and the target image, so that the pseudo-word can represent the concept or style of the target image. The reconstruction loss in training is represented as:

[0105]

[0106] where, is the mathematical expectation (Expectation), which is a core concept in probability theory and statistics. Its meaning is to average all possible values of a random variable according to its probability; z represents the latent vector; E(x) represents the prior distribution of the latent vector z, which is a standard normal distribution here; ε ~ N(0, 1) represents that the random noise ε follows the standard normal distribution N(0, 1); y represents additional conditions; c(y) represents the encoder used to encode the additional conditions y, and t represents the time step.

[0107] At each denoising time step t, the noise prediction network ε θ reconstructs the noisy latent vector z t under the guidance of the additional conditions c(y), where c(·) represents an encoder for encoding additional conditions, such as a CLIP text encoder.

[0108] However, the above training process lacks an explicit image style and content decoupling mechanism, and the optimized single pseudo-word will learn both the style and content of the target image. This application uses the inversion of the diffusion model to automatically align the style of the style image and the content description text of the style image to the extended style pseudo-word set and the pure content description text of the style image, respectively, to achieve the decoupling of the style and content of the style image. In addition, there are often problems of incomplete semantic coverage and style redundancy when generating the pure content description text of the style image. Therefore, this application designs a style text filtering method based on style image style removal and image-text distance comparison to obtain the pure content description text of the style image, thereby solving the problems of incomplete semantic coverage and style redundancy.

[0109] Step 1: To solve the problem of incomplete semantic coverage, this application uses a multi-modal large language model (MLLM) to generate a detailed image description text T that contains information such as the location of objects, the occurrence of actions, and the atmosphere of the scene. And use the tokenizer to split the image description text into multiple text tokens, represented as:

[0110] token = Tokenizer(T) = [token1, token2, … token n] (2)

[0111] where Tokenizer(·) is a tokenizer, token i denotes the i-th text token, and n denotes the number of text tokens, which depends on the length of the image description text T. i = 1, …, n.

[0112] Inevitably, the image description text T will also contain style information of the style image, which will cause the style information of the style image to be incorrectly aligned to the style information of T during training, resulting in a deviation of style and content decoupling. The next steps 2 and 3 of the present application will filter the text containing style information in T to solve the style redundancy problem.

[0113] Step 2: In order to solve the style redundancy problem, the style information in the image description text of the style image in step 1 needs to be removed. The present application compares the text-image distance of each text token obtained in step 1 with the style image and the style-free style image respectively to determine whether each text token has style attributes, thereby filtering out text tokens with style attributes for subsequent input into the CLIP encoder.

[0114] In order to obtain the style-free style image, the style image is input into the diffusion model as a guide condition to generate a real photo, removing artistic style elements such as brush strokes and textures in the style image. Then, by grayscale conversion, the color information is removed to obtain the style-free style image I nS . The visualization of the style removal result is shown in Figure 4 , where Figure 4 (a) in (b) is the style image, Figure 4 the style-free style image. In practical applications, the style image needs to be sent to the diffusion model, which can generate a real photo. The real photo is an intermediate quantity converted from the style image to the style-free style image. In the subsequent formula, I nS is the style-free style image. Since the real photo is converted to the style-free style image after grayscale conversion.

[0115] Compared with the style image, the style-free style image retains the semantic elements and spatial position relationships in the style image, but does not have the unique artistic features such as texture and brush strokes of the style image. In addition, the color information is further removed by grayscale processing, so it can be regarded as a picture without style information.

[0116] Step 3: Next, we need to compare the distance of each text token with the two images (style image and unstyled style image) one by one and calculate the distance difference, that is, to determine whether the text token has style attribute. Since text and image belong to two different modalities, it is difficult to directly calculate their similarity. However, CLIP encoder unifies text and image modalities, so we will calculate the image-text similarity in CLIP space. We send the style image, the unstyled style image obtained in step 2, and the text token into the corresponding CLIP text / image encoder. The process is represented as:

[0117] F s =E CLIP-Image (I S ),F′ S =E CLIP-Image (I nS ) (3

[0118] F ti =E CLIP-Text (token i ) (4)

[0119] Where I S and I nS represent the style image and the unstyled style image respectively, E CLIP-Image represents the CLIP image encoder, E CLIP-Text represents the CLIP text encoder, F s , F' S and F ti represent the feature vectors of the encoded style image, unstyled style image and text token respectively.

[0120] Then we calculate the image-text distance of F ti with F s and F' S respectively, which is used to determine whether the current text token token i has style attribute. The process is represented as:

[0121]

[0122] Where L i is the image-text distance of the i-th text token and the style image, and L' i is the image-text distance of the i-th text token and the unstyled style image. The image-text distance difference ΔL i is defined as:

[0123] ΔL i =| L i-L′ i | (7

[0124] When ΔL i If the similarity to a certain set hyperparameter threshold is greater than the image-text distance difference between the current text tag and the unstyled style image, it means that the current text tag contains style information and needs to be filtered.

[0125] Step 4: Finally, repeat step 3 to calculate the distance between each text tag and the style image and the unstyled style image, filter out the text tags containing style information, and you will get the pure content text description of the style image without style information.

[0126] (2) Multidimensional Inversion Based on Diffusion Model with Separate Time Steps and Separate Attention Layers: Existing research has shown that, in terms of time steps, the diffusion model exhibits a temporal characteristic of initially constructing global features of the image and later constructing local details; in terms of attention layers, it exhibits a spatial characteristic of shallow layers focusing on color appearance and deep layers focusing on content structure. Therefore, in order to improve the fineness of style representation, this application introduces a multidimensional inversion method with separate time steps and separate attention layers, the structure of which is shown in the figure below. Figure 5 As shown.

[0127] This module, based on the inversion of the diffusion model, expands a single pseudoword into a set of style pseudowords corresponding to a specific time step and attention layer. The aim is to increase the information capacity of a single pseudoword, enabling it to better cope with the spatiotemporal dynamics of the diffusion model, thereby improving the fineness of style representation.

[0128] Step 1: A typical diffusion model has 1000 diffusion time steps and 7 attention layers. These 1000 time steps can be divided into N stages. Therefore, the total number of pseudowords after expansion should be N×7 across the time step stages and attention layers.

[0129] Create a single pseudoword S * The word embedding p is defined as:

[0130] p = E CLIP-Text ('S * ') (8)

[0131] To accelerate the training process, this application initializes p with word embeddings of a specific style of words. Then, p is copied N times, and time-step expansion is performed, represented as a time-step expanded pseudo-word set.

[0132]

[0133] Where, p i This represents the word embedding of the pseudo-word corresponding to the i-th time step during reverse training. Step 2 will... The extension of the attention layers is performed on the basis of the above.

[0134] The text conditions of the diffusion model are processed and interacted by cross-attention in the sampling layers with attention mechanism, and there are 7 such layers. Even if the pseudo word extension with attention is performed on all the layers, the effect of overfitting is not obvious, so the present application uses a separate pseudo word to correspond to all the sampling layers with cross-attention.

[0135] The set of pseudo word embeddings with hierarchical extension Each element of the set is duplicated 7 times respectively to perform the extension of the attention layers, denoted as

[0136]

[0137] Then, they are grouped into sets corresponding to a certain time step stage and attention layer, that is, the word embeddings p of the final style pseudo word set S are obtained S , denoted as:

[0138]

[0139] wherein, represents the pseudo word corresponding to the jth attention layer of the ith time step stage. is the set of word embeddings after two extensions, a total of N x 7 elements, and this formula groups the N x 7 elements, 7 elements for a group, a total of N groups, corresponding to different time step stages and different attention layers.

[0140] Step 3: Perform the inversion of the diffusion model: the pure content description text of the style picture obtained in (1) and the style pseudo word set obtained in step 2 of (2) are jointly used as the input of the inversion of the diffusion model. The optimized style pseudo word set obtained in step 3 of (2) is used as the style representation of the style picture, which will be fused with the content information of the content picture in (3) to generate the final stylized result. The inversion of the diffusion model is based on the above input, predicts the noise when the style picture is forwardly added with noise, compares with the noise of the real forwardly added noise and calculates the reconstruction loss to optimize the word embedding p of the style pseudo word set S so as to learn the style of the style picture.

[0141] Specifically, the inversion of the diffusion model first performs forward noise addition on the style picture, and the noise-added picture at a certain time step t is z st , denoted as:

[0142]

[0143] wherein, zs0 is the style picture I S In the diffusion model, the latent representation is, ∈ is the random noise following the normal distribution, is the true label for computing the loss function, and α t is the noise scheduling coefficient. Next, the word embedding p C of the pure content description text of the input style picture and the word embedding p S of the style pseudo-word set are obtained, the noise prediction ∈ θ applied at time step t is output, and p S is optimized using the following loss function:

[0144]

[0145] L S aims to align the style of the style picture to p S and align the content to p C , so as to achieve the decoupling of the style and content of the style picture. When L S converges, the optimized style pseudo-word set obtained can be used as the style representation of the style picture.

[0146] The pure content description text of the style picture obtained in (1) and the style pseudo-word set obtained in step 2 of (2) are used as the input of the diffusion model inversion in step 3 of (2). The style pseudo-word set optimized by the diffusion model inversion is used as the style representation of the style picture, which will be fused with the content information of the content picture in (3) to generate the final stylization result.

[0147] (3) Style and content fusion based on multiple condition control: In the stylization result, the content of the content picture is maintained and harmoniously fused with the style of the style picture, which is also one of the core problems of the style transfer task. This application designs a style and content fusion method based on multiple condition control, which aims to incorporate the content structure information of the content picture into the model and ensure the harmony and unity of style and content in the stylization result. The flowchart of this method is shown in Figure 6 .

[0148] In the inference stage, this module takes the optimized style pseudo-word set S obtained from the multi-dimensional inversion module based on the diffusion model divided by time step and attention layer as the style guide of the output stylization result, takes the semantic text, the neck feature of the content picture, and the encoding feature of the content picture as the content guide, and the style guide and the content guide jointly control the diffusion model to generate a visually harmonious stylization result. Figure Three

[0149] ​Step 1: Semantic text conditions play a crucial role in the diffusion model as they have strong semantic signals that often dominate the generation process, determining the semantic content features of the final image. Therefore, semantic text conditions are necessary for content control.

[0150] Furthermore, during the inference phase, the module takes the optimized style noise set S from the multi-dimensional inversion module based on the diffusion model at each time step and attention layer as the style guidance. Since both the style noise set and the semantic text condition belong to the text modality, they can be processed together and then input into the diffusion model. Specifically, the style noise set S and the artificially generated semantic text condition T C are concatenated, denoted as:

[0151] T CS =concat(S,T C ) (14)

[0152] where concat denotes directly concatenating two texts, and T CS denotes the concatenated text. Then, T CS is encoded into text word embeddings, denoted as:

[0153] e CS =E CLIP-Text (T CS ) (15)

[0154] where E CLIP-Text (·) denotes the CLIP text encoder. In the diffusion model, the text condition is mainly processed by the cross-attention in the sampling layer with attention mechanism. The word embedding vectors are first mapped into key vectors (Key) and value vectors (Value), and then perform attention operations with the query vectors (Query) mapped by the image feature vectors to incorporate the text condition into the diffusion model. The above process is represented by the following formula:

[0155] K=W K e CS +b K ,V=W V e CS +b V (16)

[0156]

[0157] where W K and W V denote the mapping matrices for keys and values, and b K and b V denote the bias vectors. Q t is the query vector mapped by the image feature vector from the output of the previous layer, KT is the transpose of the key vector.

[0158] The diffusion model accepts text conditions by such cross-attention computation to encode the text conditions e CS The input cross-attention mechanism computation is e CS The data flow of the input diffusion model.

[0159] Step 2: The content picture is the most direct content guidance way, but directly sending the content picture into the diffusion model without processing will cause the style information of the content picture to interfere with the stylization result. Therefore, the project first converts the content picture I C The gray scale conversion is performed to remove the color information to obtain I' C , and the down-sampling and intermediate layers of the diffusion model are copied as the picture encoder E εD , the output feature of which is called the neck feature, and the first-order statistics of the style picture feature are aligned to the style picture feature using adaptive instance normalization (AdaIN) to avoid the style interference of the content picture. The formula is expressed as:

[0160]

[0161] where I' C and I S represent the style picture and the content picture respectively, E εD (·) represents the picture encoder, μ(·) represents the mean value operation, σ(·) represents the variance operation, is the neck feature of the style picture after removing the color, is the wave feature of the style picture. The finally obtained content picture feature v C is one of the content guidance conditions, which is injected into the diffusion model by directly adding it to the intermediate layer output feature of the diffusion model, expressed as:

[0162] H' = H + v C .

[0163] where H is the intermediate layer feature output of the diffusion model, and H' is the intermediate layer output fused with v C .

[0164] Step 3: The depth map introduces spatial constraints in the generation process by providing the relative depth information of each pixel, effectively guiding the spatial structure in the stylization result. The pre-trained Dense Prediction Transformer (DPT) is used to extract the depth map I C of the content picture, expressed as:

[0165]

[0166] wherein, denotes a pre-trained DPT model, i.e. the depth map of the content image. Then the pre-trained ControlNet is used as an encoder for the depth map, which is introduced into the up-sampling layer of the diffusion model in the form of a residual link through a zero convolution layer, thereby guiding the generation of the model.

[0167] The data flow mode of the depth map of the content image as a control condition: through the pre-trained ControlNet input the up-sampling layer of the diffusion model.

[0168] Step 4: The above steps inject the semantic text, the neck feature of the content image and the depth map of the content image as content guidance into the diffusion model, and also inject the optimized style pseudo word set obtained from the multi-dimensional inversion module based on the diffusion model into the diffusion model as style guidance. On this basis, the denoising of the diffusion model is performed, and the diffusion model can generate the final stylized image from random noise.

[0169] The association and data flow relationship between (1), (2) and (3) are as follows:

[0170] The "style and content fusion based on multiple conditions control" explained in part (3) plays a role in the inference stage. It receives the optimized style pseudo word set output by the multi-dimensional inversion module based on the diffusion model as described in part (2) as style guidance, and then injects the semantic text, the neck feature of the content image and the depth map of the content image as content guidance. The encoding of the encoder of the content image. The data flow relationship between (2) and (1) is emphasized when modifying (2). The pure content description text of the style image obtained in (1) and the style pseudo word set obtained in step 2 of (2) are input into the diffusion model inversion in step 3 of (2). Figure Three The data flow relationship of the content in part (3) is as follows:

[0171] In step 1, the optimized style pseudo word set output by the multi-dimensional inversion module based on the diffusion model and the semantic text condition are input into the diffusion model through cross-attention (because the style pseudo word set and the semantic text are both text conditions, they are input into the diffusion model in the same way).

[0172] In step 2, the extracted neck feature of the content image is directly added to the intermediate layer output feature of the diffusion model.

[0173]

[0174] ​In step 3, the depth map of the content picture is encoded by the pre-trained ControlNet and added to the output features of the intermediate layers and up-sampling layers of the diffusion model.

[0175] As shown in Figure 6 The above three steps are parallel, but their styles and content data all flow to the diffusion model.

[0176] Finally in step 4, the diffusion model performs a denoising process to generate a stylized result picture from random noise according to the style and content guidance provided by the above 3 steps.

[0177] This method mainly aims at the key title in the research of artistic image style transfer based on diffusion model inversion. Compared with traditional style transfer methods, the diffusion model based inversion method only needs a single sample data to generate high quality result image, which does not need to collect a large amount of training data, and relieves the problem of training resources. However, the current research on diffusion model based inversion method has the problems of being unable to effectively decouple the style of the style picture, and being unable to balance the style and content in the stylized result. A large number of existing researches have carried out in-depth research from the aspects of fine-tuning of diffusion model, adding additional attention layer, etc. The present application carries out related research from the aspects of style filtering, multi-dimensional inversion and multi-condition guidance.

[0178] Comparison of the present application with traditional methods:

[0179] (1) The ordinary diffusion model inversion method lacks an explicit picture style and content decoupling mechanism. After optimization, the pseudo word will learn the style and content of the target picture at the same time, and cannot effectively decouple the style and content of the target picture. Therefore, a style text filtering method based on style picture style removal and picture-text distance comparison is used to remove the style of the style picture, and to compare the distance between each text label in the description text of the style picture and the style picture, and the distance between the style picture without style to judge and filter the text containing style information, to obtain the pure content description text of the style picture. So that the inversion of the diffusion model can automatically align the style and content of the style picture to the pseudo word and the pure content description text respectively. It ensures that the model can effectively decouple the style and content of the style picture.

[0180] (2) The inversion of the diffusion model generally uses a single pseudo word for the entire inversion process. Due to the information capacity limitation of the single pseudo word, the pseudo word has limitations in showing the time and space dynamic characteristics of the diffusion model in the time step and attention layer. Therefore, a multi-dimensional inversion based on diffusion model is proposed, which expands a single pseudo word into a pseudo word set corresponding to a specific time step stage and attention layer, aiming to enhance the information capacity and representation ability of the pseudo word, so that it can learn the fine-grained style features in the style picture.

[0181] (3) Existing style transfer methods often face the problem of style and content trade-off in the stylized results. To solve this problem, a style and content fusion method based on multiple condition control is proposed. The semantic text, content image and its depth map are used to control the content structure of the stylized results. In order to avoid the influence of the style of the content image on the results, the first-order statistics of the style image features are aligned to the content image features using adaptive instance normalization. The model can balance the style and content of the results, and generate visually harmonious stylized results.

[0182] In the research of artistic image style transfer based on diffusion model inversion, the most important problem is that the single pseudo word of traditional inversion method cannot effectively decouple the style and content of the target image, resulting in style redundancy of the content image and content residue of the style image in the stylized results. Based on the style image style removal and image-text distance comparison, the style text filtering method identifies and filters out the style-related text in the description text of the style image through style image style removal and image-text distance calculation, and obtains the pure content description text of the style image. Further, the inversion framework can align the content of the style image to the pure content description text, and align the style to the expanded style pseudo word set, effectively decoupling the style and content of the style image.

[0183] For the artistic image style transfer task, the colors, textures, brush strokes in the style image and the unique elements in the artistic work need to be transferred to the stylized results. Therefore, how to make the network learn the fine-grained style representation in artistic images is a problem worth thinking about. In existing inversion-based research, a single pseudo word is usually directly mapped to all attention layers and all diffusion time steps of the diffusion model. However, this approach ignores the spatiotemporal dynamic characteristics of the diffusion model in terms of time steps and attention layers. Therefore, a multi-dimensional inversion method based on diffusion model time step and attention layer is designed. This module expands a single pseudo word into a style pseudo word set, divides the diffusion model into several attention layers, and divides the diffusion model time steps into several stages. In the inversion process, each pseudo word in the style pseudo word set is mapped to a specific time step stage and attention layer. The style pseudo word set has more dimensional representation ability than the single pseudo word, thereby improving the fine granularity of style representation.

[0184] In stylization results, content preservation of the content image and its harmonious integration with the style of the style image are core issues in style transfer tasks. Existing diffusion-based studies tend to favor either style or content too much, failing to achieve a proper balance and resulting in poor visual quality. This multi-condition controlled style and content fusion approach considers the impact of different content control conditions on the balance between content and style. It uses three conditions—semantic conditions, depth map, and neck features of the content image—to control the content of the stylization results from multiple dimensions, thereby achieving a balance between style and content.

[0185] Based on the same inventive concept, this application also provides an art image style transfer apparatus for implementing the art image style transfer method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more art image style transfer apparatus embodiments provided below can be found in the limitations of the art image style transfer method described above, and will not be repeated here.

[0186] In one exemplary embodiment, such as Figure 8 As shown, an art image style transfer device is provided, comprising:

[0187] The acquisition module is used to acquire the style image and content image to be processed.

[0188] The style text filtering module is used to perform style text filtering on the style image to be processed, so as to obtain pure content description text; the style text filtering includes style removal and image-text distance comparison.

[0189] A creation and expansion module is used to create a single pseudoword and expand the single pseudoword based on the time step stage and the attention layer to obtain a set of style pseudowords.

[0190] The optimization module is used to optimize the style pseudoword set based on the pure content description text using the inversion of the diffusion model, so as to obtain the style representation of the style image to be processed.

[0191] The style transfer module uses the style representation of the style image to be processed as a style guide, determines the content guide based on the content image, and uses the diffusion model for denoising to determine the stylization result.

[0192] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 9As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores art style transfer data. The I / O interfaces allow the processor to exchange information with external devices. The communication interface allows communication with external terminals via a network connection. When executed by the processor, the computer program implements an art style transfer method.

[0193] Those skilled in the art will understand that Figure 9 The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method embodiments.

[0194] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the above-described method embodiments.

[0195] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described method embodiments.

[0196] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0197] In this application, all actions to acquire signals, information, or data are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with the authorization granted by the owner of the relevant device.

[0198] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0199] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0200] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0201] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for style transfer in artistic images, characterized in that, The art image style transfer method includes: Obtain the style image and content image to be processed; The style image to be processed is subjected to style text filtering to obtain pure content description text; the style text filtering includes style removal and image-text distance comparison. Create a single pseudoword and expand the single pseudoword based on the time step stage and the attention layer to obtain a set of style pseudowords; The style pseudoword set is optimized using the inversion of the diffusion model based on the pure content description text to obtain the style representation of the style image to be processed; The style representation of the style image to be processed is used as the style guide, the content guide is determined based on the content image, and the stylization result is determined by using the diffusion model for denoising.

2. The art image style transfer method according to claim 1, characterized in that, The style image to be processed is subjected to style text filtering to obtain pure content description text, specifically including: The style images to be processed are used to generate image description text using a multimodal large language model; The image description text is split into multiple text tags using a word segmenter; The style image to be processed is used to obtain a real photo using a diffusion model; A styleless image is determined by performing grayscale transformation on the real photo. Image-text distance is calculated based on each text tag, the style image to be processed, and the unstyled style image to determine the style attribute of each text. Style attribute filtering is performed based on the text tags and style attributes to obtain pure content description text.

3. The art image style transfer method according to claim 1, characterized in that, A single pseudoword is created and then expanded based on the time step stage and the attention layer to obtain a set of style pseudowords, specifically including: Create a single pseudoword; The single pseudoword is copied multiple times and expanded according to the time step stage to obtain the time step expanded pseudoword set. Each element in the time-step expanded pseudoword set is copied and expanded according to the attention layer to obtain the style pseudoword set.

4. The art image style transfer method according to claim 1, characterized in that, The style pseudoword set is optimized based on the pure content description text using the inversion of the diffusion model to obtain the style representation of the style image to be processed, specifically including: The pure content description text and the set of style pseudowords are concatenated to obtain the concatenated text; The concatenated text is encoded to obtain the first text word embedding; Gradients are generated based on the inversion of the first text word embedding based on the diffusion model; The style pseudoword set is optimized based on the gradient to obtain the style representation of the style image to be processed.

5. The art image style transfer method according to claim 4, characterized in that, The style representation of the style image to be processed is used as a style guide, the content guide is determined based on the content image, and the stylization result is determined using a diffusion model for denoising. Specifically, this includes: The text description of the content image is encoded to obtain the second text word embedding; The style image and content image to be processed are image encoded to obtain neck features; Adaptive instance normalization is performed on the neck features; Depth estimation is performed on the content image to obtain a depth map; Encode the depth map; The style representation of the style image to be processed is used as style guidance, and the second text word embedding, adaptive instance normalized neck features and encoded depth map are used as content guidance. A diffusion model is used for denoising to obtain the stylized result.

6. The art image style transfer method according to claim 1, characterized in that, The loss function for the inversion of the diffusion model during training is the reconstruction loss.

7. An art image style transfer device, characterized in that, The art image style transfer device includes: The acquisition module is used to acquire the style image and content image to be processed. The style text filtering module is used to perform style text filtering on the style image to be processed, so as to obtain pure content description text; the style text filtering includes style removal and image-text distance comparison. A creation and expansion module is used to create a single pseudoword and expand the single pseudoword based on the time step stage and the attention layer to obtain a set of style pseudowords; The optimization module is used to optimize the style pseudoword set based on the pure content description text using the inversion of the diffusion model to obtain the style representation of the style image to be processed. The style transfer module uses the style representation of the style image to be processed as a style guide, determines the content guide based on the content image, and uses the diffusion model for denoising to determine the stylization result.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the artistic image style transfer method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the artistic image style transfer method according to any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the artistic image style transfer method according to any one of claims 1-6.

Images