A finely controllable image stylization method based on diffusion model

Abstract

The present invention discloses a finely controlled image stylization method based on a diffusion model. The method comprises: obtaining training data, wherein the training data is a set of sample data with aligned style attributes, and the sample data includes a content image, a brushstroke image, a color image, text information, and a target image; implicitly performing feature decoupling processing on the content image, brushstroke image, and color image to obtain content attribute features, brushstroke attribute features, and color attribute features; fusing the content attribute features, brushstroke attribute features, color attribute features, and text information through an adaptive gating mechanism and injecting them into a controllable diffusion model, and combining them with the target image to obtain a predicted image. The present invention solves the technical problem in the prior art that the multiple fine-grained style attributes in the reference image are not decoupled, making it difficult to finely and flexibly control style transfer.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a finely controllable image stylization method based on a diffusion model. Background Art

[0002] Image processing is a key technology in artificial intelligence, and image generation is a core foundation of this field. In recent years, image generation technology has been widely used in a variety of fields, including art creation, entertainment, design, education, and film and television. However, existing image stylization methods still exhibit certain limitations when it comes to transferring detailed style attributes from a given image. Inversion-based methods often utilize learnable universal textual markers to represent specific styles. In related literature, an Implicit Style Prompt Bank (ISPB) was designed to store and transfer implicit representations of various styles. However, this coarse textual marker cannot easily transfer specific style information from a reference image. Fine-tuning-based methods aim to fine-tune a lightweight network to transfer detailed style from a reference image, achieving excellent stylized image generation performance. In another related literature, semantic-style decoupling and non-reconstructive learning strategies were employed to effectively transfer detailed style from a reference image. However, because these methods treat the style of the reference image as a single, indivisible entity, they struggle to accurately decouple the various stylistic attributes in the reference image. This limits the application of stylized image generation techniques for refined style decoupling and controllable transfer. For example, in a poster-making task, the user may only want to transfer the brushstroke style of the reference image without including other information such as color. In this case, if the style of the reference image is treated as a unified whole, it will not be possible to accurately generate a stylized image that meets the expectations. Summary of the Invention

[0003] The embodiment of the present invention provides a finely controllable image stylization method based on a diffusion model, which at least solves the technical problem in the prior art that the multiple fine-grained style attributes in the reference image are not decoupled, making it difficult to finely and flexibly control style transfer.

[0004] According to one aspect of an embodiment of the present invention, a method for fine-grained, controllable image stylization based on a diffusion model is provided. The method may include: obtaining training data, wherein the training data is a set of sample data with aligned style attributes, and the sample data includes a content image, a brushstroke image, a color image, text information, and a target image; implicitly performing feature decoupling processing on the content image, brushstroke image, and color image to obtain content attribute features, brushstroke attribute features, and color attribute features; fusing the content attribute features, brushstroke attribute features, color attribute features, and text information through an adaptive gating mechanism and injecting them into a controllable diffusion model, and combining them with the target image to obtain a predicted image.

[0005] Optionally, before obtaining training data, the method further includes: collecting a number of content, stroke and color words to construct a vocabulary library; generating a text description of the image based on the content and stroke words in the vocabulary library using a prompt template of a large language model, and synthesizing an image with a variety of stroke styles based on the text description of the image using a text-to-image generation model; generating a canny edge map based on the images with a variety of stroke styles; constructing text information of a controllable diffusion model in combination with color words and the text description of the image, inputting the text information and the canny edge map into a ControlNet model to obtain a data set with aligned stroke images, color images, content images and text information; and extracting a group of data sets with aligned stroke images, color images, content images and text information as training data.

[0006] Optionally, after acquiring the training data, the method further comprises: adding noise to the features of the target image through a forward process to obtain target image features containing noise.

[0007] Optionally, the implicit feature decoupling processing of the content image, brushstroke image and color image to obtain content attribute features, brushstroke attribute features and color attribute features includes: performing global feature extraction on the content image, brushstroke image and color image respectively through a pre-trained image encoder to obtain global features of the content image, global features of the brushstroke image and global features of the color image; performing feature extraction on the global features of the content image, global features of the brushstroke image and global features of the color image respectively through a fine-grained decoupling module to obtain content attribute features, brushstroke attribute features and color attribute features.

[0008] Optionally, the content attribute features, stroke attribute features, color attribute features and text information are fused through an adaptive gating mechanism and injected into a controllable diffusion model, and combined with a target image to obtain a predicted image, including: cross-attention of the text information and the potential representation of the noise of each cross-attention layer in the controllable diffusion model to obtain text-related embedding of each layer, wherein the cross-attention layer includes a coarse layer and a fine layer; cross-attention of the content attribute features and the potential representation of the noise of each layer of the coarse layer to obtain content-related embedding of each layer of the coarse layer; cross-attention of the stroke attribute features and the potential representation of the noise of each layer of the fine layer to obtain stroke-related embedding of each layer of the fine layer; cross-attention of the color attribute features and the potential representation of the noise of each layer of the fine layer to obtain color-related embedding of each layer of the fine layer; integrating the text-related embedding of each layer into all cross-attention layers of the controllable diffusion model, integrating the content-related embedding of each layer of the coarse layer into each layer of the coarse layer, integrating the stroke-related embedding of each layer of the fine layer into each layer of the fine layer, and integrating the color-related embedding of the fine layer into each layer of the fine layer for guidance, and combining with the target image to obtain a predicted image.

[0009] Optionally, the text-related embedding of each layer is integrated into all cross-attention layers of the controllable diffusion model, the content-related embedding of each layer of the coarse layer is integrated into each layer of the coarse layer, the stroke-related embedding of each layer of the fine layer is integrated into each layer of the fine layer, and the color-related embedding of the fine layer is integrated into each layer of the fine layer for guidance, and combined with the target image to obtain a predicted image, including: processing the text-related embedding, content-related embedding, stroke-related embedding and color-related embedding through a learnable gate to obtain target injection style information and target injection content information; guiding the controllable diffusion model based on the target injection style information and the target injection content information to obtain noise of the predicted image; and obtaining the predicted image based on the target image features containing noise and the noise of the predicted image.

[0010] Beneficial effects of the present invention:

[0011] The present invention proposes a finely controlled image stylization method based on a diffusion model. By introducing a fine decoupling mechanism and an adaptive injection mechanism, the fine attributes in multiple reference images will be effectively decoupled and then adaptively guided to generate stylized images using the diffusion model. This strategy realizes the migration of fine style attributes of the image and the effective fusion and balance between multiple conditions. In addition, the present invention opens up multiple control channels for users, through which users can achieve more flexible and accurate image stylization. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0013] Figure 1 This is a flowchart of a finely controllable image stylization method based on a diffusion model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0014] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or inherent to these processes, methods, products or devices.

[0016] Example 1

[0017] According to an embodiment of the present invention, a method for finely controllable image stylization based on a diffusion model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system comprising at least one set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0018] Figure 1 is a flow chart of a fine controllable image stylization method based on a diffusion model according to an embodiment of the present invention. Figure 1 As shown, the method may include the following steps:

[0019] Step S101 : acquiring training data, wherein the training data is a set of sample data with alignment style attributes, and the sample data includes content image, stroke image, color image, text information and target image.

[0020] In the technical solution provided in step S101 of the present invention, the expression of the sample data is:

[0021]

[0022] In this embodiment, and Represent the target image and the corresponding text information respectively, and the text information is also a text description; 、 and Represents a reference image that is consistent with the target image in content, stroke, and color, respectively.

[0023] Step S102 : implicitly performing feature decoupling processing on the content image, the brushstroke image, and the color image to obtain content attribute features, brushstroke attribute features, and color attribute features.

[0024] In the technical solution provided in step S102 of the present invention, fine-grained controllable image stylization aims to effectively decouple different attributes of the reference image and use these decoupled representations as guiding conditions for generating stylized images. A diffusion model with multiple conditions is constructed based on the controllable diffusion model. Therefore, assuming that the input of the diffusion model is ,in is a text description, is the reference image. Using the forward process Gradually reduce the noise Image features added to the latent space middle:

[0025]

[0026] in, is the feature of the target image extracted at each t from 1 to T, is the feature of the target image extracted at time t, is the identity matrix, is the feature of the target image extracted at the previous moment of t, is the prediction time step, is the total time step, is a predefined hyperparameter, usually monotonically decreasing with time steps; the goal is to use the reverse process Generate target image step by step , which contains multiple conditions , Describe the conditions for the text, are the conditions corresponding to the content image, stroke image and color image respectively, where, , :

[0027]

[0028] in, , is the denoising model, For text encoder, For the image encoder, based on the control information , For the denoising model The noise of the step prediction.

[0029] In step S103 , the content attribute features, the stroke attribute features, the color attribute features and the text information are fused through an adaptive gating mechanism and injected into a controllable diffusion model, and combined with the target image to obtain a predicted image.

[0030] In the technical solution provided in the above step S103 of the present invention, the controllable diffusion model processes the content attribute features, the stroke attribute features, the color attribute features, the text information and the target image to obtain a predicted image.

[0031] The above method of this embodiment is further introduced below.

[0032] As an optional embodiment, step S101, before obtaining training data, the method further includes: collecting a number of content, stroke and color words to construct a vocabulary library; generating a text description of the image based on the content and stroke words in the vocabulary library using a prompt template of a large language model, and synthesizing an image with multiple stroke styles based on the text description of the image using a text-to-image generation model; generating a canny edge map based on the images with multiple stroke styles; combining color words and the text description of the image to construct text information of a controllable diffusion model, inputting the text information and the canny edge map into a ControlNet model to obtain a data set with aligned stroke images, color images, content images and text information; and extracting a group of data sets with aligned stroke images, color images, content images and text information as training data.

[0033] In this embodiment, the implicit decoupling of the style of the detailed style attributes in the image requires paired images as reference and target samples for constraint; in this invention, a lightweight adapter is used to achieve the migration of multiple image attributes (content, strokes, and color); in addition, in order to maintain the controllability of the text, it is also necessary to construct a corresponding target text description. Assume that the target image and the corresponding text description are and , then the corresponding visual condition should be a reference image that is consistent with it in content, brushstrokes and colors. ; First, a vocabulary set describing content, brushstrokes, and colors was collected: , , Based on these vocabulary sets, the corresponding text description is generated using a large language model Secondly, a controllable image generation model can generate paired image-text data with a large number of consistent strokes; finally, a high-quality image dataset with a large number of consistent fine attributes and related descriptions is generated by using color cue vocabulary and Canny edge detection technology, and any high-quality image is used as the target image; in this dataset, a set of sample data can be randomly selected For training.

[0034] As an optional embodiment, in step S101, after obtaining the training data, the method further includes: adding noise to the features of the target image through a forward process to obtain the target image features containing noise.

[0035] In this embodiment, the forward process is adopted Gradually reduce the noise Image features added to the latent space middle:

[0036]

[0037] in, is the feature of the target image extracted at each t from 1 to T, is the feature of the target image extracted at time t, is the identity matrix, is the feature of the target image extracted at the previous moment of t, is the prediction time step, is the total time step, are predefined hyperparameters.

[0038] As an optional implementation method, step S102, the content image, brushstroke image and color image are implicitly subjected to feature decoupling processing to obtain content attribute features, brushstroke attribute features and color attribute features, including: performing global feature extraction on the content image, brushstroke image and color image respectively through a pre-trained image encoder to obtain global features of the content image, global features of the brushstroke image and global features of the color image; performing feature extraction on the global features of the content image, global features of the brushstroke image and global features of the color image respectively through a fine-grained decoupling module to obtain content attribute features, brushstroke attribute features and color attribute features.

[0039] In this embodiment, global features of the content image, brushstroke image, and color image are extracted using a pre-trained image encoder to obtain global features of the content image, global features of the brushstroke image, and global features of the color image. The fine-grained decoupling module extracts features from the global features of the content image, global features of the brushstroke image, and global features of the color image to obtain expressions for content attribute features, brushstroke attribute features, and color attribute features:

[0040]

[0041] in, , , For a general CLIP image encoder, For fine-grained decoupling modules, for content image, stroke image and color image, Represents the global features of the content image, the global features of the stroke image, and the global features of the color image, is a shared image projection, is a trainable adapter for decoupling different attributes, They are content attribute features, stroke attribute features and color attribute features.

[0042] As an optional embodiment, step S103, the content attribute features, stroke attribute features, color attribute features and text information are fused and injected into the controllable diffusion model through an adaptive gating mechanism, and combined with the target image to obtain a predicted image, including: cross-attention of the text information and the potential representation of the noise of each cross-attention layer in the controllable diffusion model to obtain the text-related embedding of each layer, wherein the cross-attention layer includes a coarse layer and a fine layer; cross-attention of the content attribute features and the potential representation of the noise of each layer of the coarse layer to obtain the content-related embedding of each layer of the coarse layer; cross-attention of the stroke attribute features and the potential representation of the noise of each layer of the fine layer to obtain the stroke-related embedding of each layer of the fine layer; cross-attention of the color attribute features and the potential representation of the noise of each layer of the fine layer to obtain the color-related embedding of each layer of the fine layer; integrating the text-related embedding of each layer into all cross-attention layers of the controllable diffusion model, integrating the content-related embedding of each layer of the coarse layer into each layer of the coarse layer, integrating the stroke-related embedding of each layer of the fine layer into each layer of the fine layer, and integrating the color-related embedding of the fine layer into each layer of the fine layer for guidance, and combining with the target image to obtain a predicted image.

[0043] In this embodiment, the text information is cross-attended with the potential representation of the noise of each cross-attention layer in the controllable diffusion model, and the expression of the text-related embedding of each layer is obtained as follows:

[0044]

[0045] in, , and are the linear layer weights of the diffusion potential features and text features, For each layer of text-related embedding, is the cross attention function of text features, is the text information, z is the potential representation of the noise of each cross attention layer, They are the text information corresponding to the cross attention function value, For the text-related embedding of each layer, it should be noted that the cross-attention layer contains 16 layers, the coarse layer includes layers 4 to 8, and the fine layer is the remaining cross-attention layers except layers 4 to 8; the deep layer in the middle of U-Net is the coarse layer, and the shallow layers at both ends are the fine layers.

[0046] The content attribute features are cross-attended with the potential representation of the noise in the coarse layer of the controllable diffusion model to obtain the content-related embedding of each layer in the coarse layer; the stroke attribute features are cross-attended with the potential representation of the noise in the fine layer of the controllable diffusion model to obtain the stroke-related embedding of each layer in the fine layer; the color attribute features are cross-attended with the potential representation of the noise in the fine layer of the controllable diffusion model to obtain the color-related embedding of each layer in the fine layer.

[0047]

[0048] in, is the relevant embedding of image attribute information at each layer, , is the cross attention function corresponding to content, stroke and color image features, For each layer of the coarse layer, the content-dependent embedding For each layer of fine-grained stroke embedding, For each layer of the fine layer, the color-dependent embedding For different attribute characteristics, , is the content attribute feature, is the brush stroke attribute feature, is the color attribute feature, 、 and shared linear layers representing content, stroke, and color image attributes, respectively, The corresponding values ​​of content image, stroke image and color image in the cross attention function are When the content attribute features are cross-attended with the potential representation of the noise in the coarse layer of the controllable diffusion model, z is the potential representation of the noise in each coarse layer corresponding to the cross-attention layer. When the stroke attribute features are cross-attended with the potential representation of the noise in the fine layer of the controllable diffusion model, z is the potential representation of the noise in each fine layer corresponding to the cross-attention layer. When the color attribute features are cross-attended with the potential representation of the noise in the fine layer of the controllable diffusion model, z is the potential representation of the noise in each fine layer corresponding to the cross-attention layer. The cross-attention function of text features and the cross-attention function of image features can both be calculated using the following expressions:

[0049]

[0050] in, is the dimension of the cross attention function space, T is the transpose, are the Q, K, and V values ​​of the cross attention function.

[0051] The text-related embedding of each layer is integrated into all cross-attention layers of the controllable diffusion model, the content-related embedding of each layer of the coarse layer is integrated into each layer of the coarse layer, the stroke-related embedding of each layer of the fine layer is integrated into each layer of the fine layer, and the color-related embedding of the fine layer is integrated into each layer of the fine layer for guidance. Combined with the target image, the predicted image is obtained.

[0052] As an optional implementation method, the text-related embedding of each layer is integrated into all cross-attention layers of the controllable diffusion model, the content-related embedding of each layer of the coarse layer is integrated into each layer of the coarse layer, the stroke-related embedding of each layer of the fine layer is integrated into each layer of the fine layer, and the color-related embedding of the fine layer is integrated into each layer of the fine layer for guidance, and combined with the target image to obtain a predicted image, including: processing the text-related embedding, content-related embedding, stroke-related embedding and color-related embedding through a learnable gate to obtain target injection style information and target injection content information; guiding the controllable diffusion model based on the target injection style information and the target injection content information to obtain the noise of the predicted image; and obtaining the predicted image based on the target image features containing noise and the noise of the predicted image.

[0053] In this embodiment, the text-related embedding, content-related embedding, stroke-related embedding, and color-related embedding are processed through a learnable gate to obtain the expressions of target injection style information and target injection content information:

[0054]

[0055] Among them, the gate function ,here is a learnable temperature-controlled activation function. For a feedforward network, selectively embed the brush strokes and color embedding Fuse into layers with more local information to inject style information In addition, embedding content Integrate into layers with more global information and inject content information ,On this basis, various conditions can be adaptively fused and injected into the ,diffusion model, and the learnable gating parameters can flexibly adapt to different ,injection requirements, preventing excessive information from leaking into the model, ,and achieving better fine-grained image stylization.

[0056] Training and testing process:

[0057] Given a training data set , the text control information and image control information will be encoded by the text encoder and image encoder respectively, and the dimension is The text embedding and dimension are Image embedding of , where B represents the batch size, and denote the number of channels for text and image embedding respectively, Refers to the feature dimension of the cross-attention space. Among them, the visual embedding will be implicitly decoupled from the style attributes through the fine decoupling module to obtain multiple The style attribute embedding of the image is achieved by embedding the conditional information. This conditional information will be further used by the adaptive injection module to guide the features of the denoising model, thereby controlling the generation of the image. The entire network is trained using the MSE loss:

[0058]

[0059] in, is the loss function of the entire network, is a compression encoder based on a controllable diffusion model, is the input target image, is a Gaussian distribution, is the identity matrix, For expectations, To control information, is the prediction time step, The real noise introduced into the target image, For the denoising model The noise of the step prediction, It is a denoising network model.

[0060] Once the training is completed, the present invention can be directly applied to fine-grained and controllable image stylization tasks. During the testing phase, a 30-step DDIM sampler is used to generate stylized images under specified control information.

[0061] The consistency between the prompt information (text, content reference image, brushstroke reference image, and color reference image) and the generated image was established based on CLIP cosine similarity, and the fidelity of the generated image was measured using a quality assessment model. The model trained on the constructed simulation dataset achieved excellent performance in text alignment (27.8), style similarity (24.5), image fidelity (6.14), and average consistency of image sub-attributes (22.2). These performances all achieved varying degrees of improvement compared to state-of-the-art algorithms. Furthermore, a subjective visualization survey of users indicated that the image stylization results of our invention achieved the highest user satisfaction (87.9%).

[0062] In an embodiment of the present invention, training data is obtained, wherein the training data is a set of sample data with aligned style attributes, and the sample data includes a content image, a stroke image, a color image, text information, and a target image; feature decoupling processing is implicitly performed on the content image, the stroke image, and the color image to obtain content attribute features, stroke attribute features, and color attribute features; the content attribute features, stroke attribute features, color attribute features, and text information are fused and injected into a controllable diffusion model through an adaptive gating mechanism, and combined with the target image to obtain a predicted image. This solves the technical problem in the prior art that multiple fine-grained style attributes in the reference image are not decoupled, making it difficult to finely and flexibly control style transfer. This achieves the goal of constructing a dataset with a large number of fine style attributes by utilizing a large language model and a text-to-image generation model, expanding the text-to-image generation model, and constructing a multi-conditional image stylization framework. Based on the framework, a fine-grained decoupling mechanism is established to separate multiple image attributes, and a gated feature injection mechanism is designed to adaptively inject multiple conditions to guide the diffusion model to synthesize stylized images, thereby generating the expected technical effect of accurately stylized images.

[0063] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0064] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0065] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0066] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected to achieve the purpose of the present embodiment according to actual needs.

[0067] In addition, the functional units in various embodiments of the present invention may be integrated into a first processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0068] The above are only preferred embodiments of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A finely controllable image stylization method based on a diffusion model, characterized in that: include: Collect a number of content, brushstroke and color words to build a vocabulary library; Based on the content and brush stroke vocabulary in the vocabulary library, the prompt template of the large language model is used to generate a text description of the image. Based on the text description of the image, the text-to-image generation model is used to synthesize images with various brush stroke styles. Generate canny edge maps based on images with multiple brush stroke styles; The text information of the controllable diffusion model is constructed by combining color vocabulary and text description of the image. The text information and canny edge map are input into the ControlNet model to obtain a dataset with aligned stroke images, color images, content images and text information. Extracting a set of data sets with aligned stroke images, color images, content images, and text information as training data, wherein the training data is a set of sample data with aligned style attributes, and the sample data includes content images, stroke images, color images, text information, and target images; Implicitly perform feature decoupling processing on the content image, brush stroke image and color image to obtain content attribute features, brush stroke attribute features and color attribute features; The content attribute features, stroke attribute features, color attribute features, and text information are fused through an adaptive gating mechanism and injected into a controllable diffusion model. Combined with the target image, a predicted image is obtained, including: cross-attention is performed on the text information and the potential representation of the noise of each cross-attention layer in the controllable diffusion model to obtain the text-related embedding of each layer, where the cross-attention layer includes a coarse layer and a fine layer; Cross-attention is performed on the content attribute features and the potential representation of the noise of each layer in the coarse layer to obtain the content-related embedding of each layer in the coarse layer; Cross-attention is performed on the stroke attribute features and the potential representation of the noise of each layer in the fine layer to obtain the stroke-related embedding of each layer in the fine layer; Cross-attention is performed on the color attribute features and the potential representation of the noise of each layer of the fine layer to obtain the color-related embedding of each layer of the fine layer; The text-related embedding of each layer is integrated into all cross-attention layers of the controllable diffusion model, the content-related embedding of each layer of the coarse layer is integrated into each layer of the coarse layer, the stroke-related embedding of each layer of the fine layer is integrated into each layer of the fine layer, and the color-related embedding of the fine layer is integrated into each layer of the fine layer for guidance. Combined with the target image, the predicted image is obtained.

2. The method according to claim 1, characterized in that After obtaining the training data, the method further includes: The noise is added to the features of the target image through the forward process to obtain the target image features containing noise.

3. The method according to claim 1, characterized in that The implicit feature decoupling process of the content image, the brush stroke image and the color image is performed to obtain content attribute features, brush stroke attribute features and color attribute features, including: The global features of the content image, the stroke image and the color image are extracted respectively by using a pre-trained image encoder to obtain the global features of the content image, the global features of the stroke image and the global features of the color image; The fine-grained decoupling module is used to extract the global features of the content image, the global features of the stroke image and the global features of the color image respectively to obtain content attribute features, stroke attribute features and color attribute features.

4. The method according to claim 1, wherein The content attribute features, stroke attribute features, color attribute features and text information are fused through an adaptive gating mechanism and injected into a controllable diffusion model, and combined with the target image to obtain a predicted image, including: Cross-attention is performed on the text information and the potential representation of the noise of each cross-attention layer in the controllable diffusion model to obtain the text-related embedding of each layer, where the cross-attention layer includes a coarse layer and a fine layer; Cross-attention is performed on the content attribute features and the potential representation of the noise of each layer in the coarse layer to obtain the content-related embedding of each layer in the coarse layer; Cross-attention is performed on the stroke attribute features and the potential representation of the noise of each layer in the fine layer to obtain the stroke-related embedding of each layer in the fine layer; Cross-attention is performed on the color attribute features and the potential representation of the noise of each layer of the fine layer to obtain the color-related embedding of each layer of the fine layer; The text-related embedding of each layer is integrated into all cross-attention layers of the controllable diffusion model, the content-related embedding of each layer of the coarse layer is integrated into each layer of the coarse layer, the stroke-related embedding of each layer of the fine layer is integrated into each layer of the fine layer, and the color-related embedding of the fine layer is integrated into each layer of the fine layer for guidance. Combined with the target image, the predicted image is obtained.

5. The method according to claim 1, characterized in that The method integrates the text-related embedding of each layer into all cross-attention layers of the controllable diffusion model, integrates the content-related embedding of each layer of the coarse layer into each layer of the coarse layer, integrates the stroke-related embedding of each layer of the fine layer into each layer of the fine layer, and integrates the color-related embedding of the fine layer into each layer of the fine layer for guidance, and combines the target image to obtain a predicted image, including: The text-related embedding, content-related embedding, stroke-related embedding, and color-related embedding are processed through a learnable gate to obtain target injection style information and target injection content information; The controllable diffusion model is guided based on the target injection style information and target injection content information to obtain the noise of the predicted image; A predicted image is obtained based on the noisy target image features and the noise of the predicted image.

6. A computer system, characterized in that include: One or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method of claim 1.

7. A computer-readable storage medium, characterized in that Computer-executable instructions are stored, and when the instructions are executed, they are used to implement the method of claim 1.

8. A computer program product, characterized in that The invention comprises computer executable instructions, which are used to implement the method of claim 1 when the instructions are executed.

Images