Fine controllable image stylization method based on diffusion model

Abstract

The invention discloses a fine controllable image stylization method based on a diffusion model. The method comprises the steps that training data is acquired, the training data is a group of sample data with aligned style attributes, and the sample data comprises a content image, a stroke image, a color image, text information and a target image; implicitly performing feature decoupling processing on the content image, the stroke image and the color image to obtain a content attribute feature, a stroke attribute feature and a color attribute feature; and fusing the content attribute feature, the stroke attribute feature, the color attribute feature and the text information through an adaptive gating mechanism, injecting the fused features into a controllable diffusion model, and combining the target image to obtain a prediction image. The technical problem that style transfer is difficult to finely and flexibly control due to the fact that various fine-grained style attributes in the reference image are not decoupled in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a finely controllable image stylization method based on a diffusion model. Background Art

[0002] Image processing is a key technology in artificial intelligence, and image generation is the core foundation of this field. In recent years, image generation technology has been widely used in many fields such as art creation, entertainment, design, education, film and television. However, existing image stylization methods still show certain limitations in handling the task of transferring detailed style attributes of specified images. Inversion-based methods often use learnable general text tags to represent specific styles. In related literature, an Implicit Style Prompt Bank (ISPB) is designed to store implicit expressions and transfers of various styles. However, this rough text tag is difficult to handle the transfer of specific style information in the reference image. The fine-tuning-based method aims to fine-tune a lightweight network to transfer the detailed style in the reference image, achieving excellent stylized image generation performance. In another related literature, semantic-style decoupling and non-reconstructive learning strategies are used to achieve effective transfer of the detailed style of the reference image. However, since these methods regard the style in the reference image as a single, indivisible entity, it is difficult for them to accurately decouple multiple style attributes in the reference image. This limits the application of stylized image generation technology in refined style decoupling and controllable transfer. For example, in a poster-making task, the user may only want to transfer the brushstroke style in the reference image without other information such as color. In this case, if the style of the reference image is considered as a unified whole, it is impossible to accurately generate a stylized image that meets the expectations. Summary of the invention

[0003] The embodiment of the present invention provides a finely controllable image stylization method based on a diffusion model, so as to at least solve the technical problem in the prior art that multiple fine-grained style attributes in a reference image are not decoupled, making it difficult to finely and flexibly control style transfer.

[0004] According to one aspect of an embodiment of the present invention, a fine controllable image stylization method based on a diffusion model is provided. The method may include: obtaining training data, wherein the training data is a set of sample data with aligned style attributes, and the sample data includes content image, brush stroke image, color image, text information and target image; implicitly performing feature decoupling processing on the content image, brush stroke image and color image to obtain content attribute features, brush stroke attribute features and color attribute features; fusing the content attribute features, brush stroke attribute features, color attribute features and text information through an adaptive gating mechanism and injecting them into a controllable diffusion model, and combining them with the target image to obtain a predicted image.

[0005] Optionally, before obtaining training data, the method further includes: collecting a number of content, stroke and color words to construct a vocabulary library; generating a text description of the image based on the content and stroke words in the vocabulary library using a prompt template of a large language model, and synthesizing an image with a variety of stroke styles based on the text description of the image using a text-to-image generation model; generating a canny edge map based on the image with a variety of stroke styles; constructing text information of a controllable diffusion model in combination with color words and the text description of the image, inputting the text information and the canny edge map into a ControlNet model to obtain a data set with aligned stroke images, color images, content images and text information; and extracting a group of data sets with aligned stroke images, color images, content images and text information as training data.

[0006] Optionally, after acquiring the training data, the method further comprises: adding noise to the features of the target image through a forward process to obtain the target image features containing noise.

[0007] Optionally, the implicit feature decoupling processing of the content image, the brush stroke image and the color image to obtain content attribute features, brush stroke attribute features and color attribute features includes: performing global feature extraction on the content image, the brush stroke image and the color image respectively through a pre-trained image encoder to obtain global features of the content image, global features of the brush stroke image and global features of the color image; performing feature extraction on the global features of the content image, the global features of the brush stroke image and the global features of the color image respectively through a fine-grained decoupling module to obtain content attribute features, brush stroke attribute features and color attribute features.

[0008] Optionally, the method of fusing content attribute features, brush stroke attribute features, color attribute features and text information through an adaptive gating mechanism and injecting them into a controllable diffusion model, combining with a target image, and obtaining a predicted image includes: cross-attention of text information and a potential representation of noise in each cross-attention layer in the controllable diffusion model to obtain text-related embedding of each layer, wherein the cross-attention layer includes a coarse layer and a fine layer; cross-attention of content attribute features and a potential representation of noise in each layer of the coarse layer to obtain content-related embedding of each layer of the coarse layer; cross-attention of brush stroke attribute features and a potential representation of noise in each layer of the fine layer to obtain brush stroke-related embedding of each layer of the fine layer; cross-attention of color attribute features and a potential representation of noise in each layer of the fine layer to obtain color-related embedding of each layer of the fine layer; integrating text-related embedding of each layer into all cross-attention layers of the controllable diffusion model, integrating content-related embedding of each layer of the coarse layer into each layer of the coarse layer, integrating brush stroke-related embedding of each layer of the fine layer into each layer of the fine layer, and integrating color-related embedding of the fine layer into each layer of the fine layer for guidance, and combining with the target image to obtain a predicted image.

[0009] Optionally, the method integrates the text-related embedding of each layer into all cross-attention layers of the controllable diffusion model, integrates the content-related embedding of each layer of the coarse layer into each layer of the coarse layer, integrates the stroke-related embedding of each layer of the fine layer into each layer of the fine layer, and integrates the color-related embedding of the fine layer into each layer of the fine layer for guidance, and combines with the target image to obtain a predicted image, including: processing the text-related embedding, content-related embedding, stroke-related embedding and color-related embedding through a learnable gate to obtain target injection style information and target injection content information; guiding the controllable diffusion model based on the target injection style information and the target injection content information to obtain noise of the predicted image; and obtaining the predicted image based on the target image features containing noise and the noise of the predicted image.

[0010] Beneficial effects of the present invention: The present invention proposes a fine controllable image stylization method based on a diffusion model. By introducing a fine decoupling mechanism and an adaptive injection mechanism, the fine attributes in multiple reference images will be effectively decoupled and then adaptively guide the diffusion model to generate stylized images. This strategy realizes the migration of fine style attributes of the image and the effective fusion and balance between multiple conditions. The present invention opens multiple control channels for users, through which users can achieve more flexible and accurate image stylization. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 The present invention is a flowchart of a method for finely controlling image stylization based on a diffusion model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0012] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only embodiments of a part of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0013] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0014] Example 1 According to an embodiment of the present invention, a fine controllable image stylization method based on a diffusion model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system comprising at least one set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0015] Figure 1 is a flow chart of a fine controllable image stylization method based on a diffusion model according to an embodiment of the present invention. Figure 1 As shown, the method may include the following steps: Step S101, obtaining training data, wherein the training data is a set of sample data with alignment style attributes, and the sample data includes content image, stroke image, color image, text information and target image.

[0016] In the technical solution provided in the above step S101 of the present invention, the expression of the sample data is:

[0017] In this embodiment, and Respectively represent the target image and the corresponding text information, and the text information is also a text description; , and Respectively represent reference images that are consistent with the target image in content, stroke, and color.

[0018] Step S102 , implicitly performing feature decoupling processing on the content image, the brush stroke image and the color image to obtain content attribute features, brush stroke attribute features and color attribute features.

[0019] In the technical solution provided in step S102 of the present invention, fine-grained controllable image stylization aims to effectively decouple different attributes of the reference image, and use these decoupled representations as guiding conditions for generating stylized images, and construct a diffusion model with multiple conditions based on the controllable diffusion model. Therefore, assuming that the input of the diffusion model is ,in is a text description, is the reference image. Using the forward process Gradually reduce the noise Image features added to the latent space middle:

[0020] in, is the feature of the target image extracted at each t from 1 to T, is the feature of the target image extracted at time t, is the identity matrix, is the feature of the target image extracted at the previous moment of t, is the prediction time step, is the total time step, is a predefined hyperparameter, usually monotonically decreasing with time steps; the goal is to use the reverse process Generate target image step by step , which contains multiple conditions , Describe the conditions for the text, are the conditions corresponding to the content image, stroke image and color image respectively, where , :

[0021] in, , is the denoising model, For text encoder, For the image encoder, based on the control information , For the denoising model The noise of the step prediction.

[0022] Step S103 , the content attribute features, the stroke attribute features, the color attribute features and the text information are fused through an adaptive gating mechanism and injected into a controllable diffusion model, and combined with the target image to obtain a predicted image.

[0023] In the technical solution provided in the above step S103 of the present invention, the controllable diffusion model processes the content attribute features, the stroke attribute features, the color attribute features, the text information and the target image to obtain a predicted image.

[0024] The above method of this embodiment is further introduced below.

[0025] As an optional implementation mode, step S101, before obtaining training data, the method also includes: collecting a number of content, stroke and color words to build a vocabulary library; generating a text description of the image based on the content and stroke words in the vocabulary library using a prompt template of a large language model, and synthesizing an image with a variety of stroke styles based on the text description of the image using a text-to-image generation model; generating a canny edge map based on the image with a variety of stroke styles; combining color words and the text description of the image to build text information of a controllable diffusion model, and inputting the text information and the canny edge map into a ControlNet model to obtain a data set with aligned stroke images, color images, content images, and text information; extracting a group of data sets with aligned stroke images, color images, content images, and text information as training data.

[0026] In this embodiment, the implicit decoupling of the style of the detailed style attributes in the image requires pairs of images to be constrained as reference and target samples respectively; in the present invention, a lightweight adapter is used to achieve the migration of multiple image attributes (content, strokes, and colors); in addition, in order to maintain the controllability of the text, it is also necessary to construct the corresponding target text description. Assume that the target image and the corresponding text description are and , then the corresponding visual condition should be a reference image that is consistent with it in content, strokes, and colors. ; First, a vocabulary set describing content, brushstrokes, and colors was collected: , , Based on these vocabulary sets, the corresponding text description is generated using a large language model ; Secondly, a controllable image generation model can generate paired image-text data with a large number of consistent strokes; Finally, a high-quality image dataset with a large number of consistent fine attributes and related descriptions is generated using color cue vocabulary and Canny edge detection technology, and any high-quality image is used as the target image; in this dataset, a set of sample data can be randomly selected For training.

[0027] As an optional implementation manner, in step S101, after acquiring the training data, the method further includes: adding noise to the features of the target image through a forward process to obtain the target image features containing noise.

[0028] In this embodiment, the forward process is adopted Gradually reduce the noise Image features added to the latent space middle:

[0029] in, is the feature of the target image extracted at each t from 1 to T, is the feature of the target image extracted at time t, is the identity matrix, is the feature of the target image extracted at the previous moment of t, is the prediction time step, is the total time step, are pre-defined hyperparameters.

[0030] As an optional implementation method, step S102, the implicit feature decoupling processing of the content image, the brush stroke image and the color image to obtain content attribute features, brush stroke attribute features and color attribute features includes: performing global feature extraction on the content image, the brush stroke image and the color image respectively through a pre-trained image encoder to obtain global features of the content image, global features of the brush stroke image and global features of the color image; performing feature extraction on the global features of the content image, the global features of the brush stroke image and the global features of the color image respectively through a fine-grained decoupling module to obtain content attribute features, brush stroke attribute features and color attribute features.

[0031] In this embodiment, the global features of the content image, the stroke image and the color image are extracted respectively by the pre-trained image encoder to obtain the global features of the content image, the global features of the stroke image and the global features of the color image; the global features of the content image, the global features of the stroke image and the global features of the color image are extracted respectively by the fine-grained decoupling module to obtain the expressions of the content attribute features, the stroke attribute features and the color attribute features:

[0032] in, , , is a general CLIP image encoder, To decouple modules at a fine-grained level, for content image, stroke image and color image, Represents the global features of the content image, the global features of the stroke image, and the global features of the color image, is a shared image projection, is a trainable adapter for decoupling different attributes. They are content attribute features, stroke attribute features and color attribute features.

[0033] As an optional implementation method, step S103, the content attribute features, stroke attribute features, color attribute features and text information are fused and injected into the controllable diffusion model through an adaptive gating mechanism, combined with the target image, to obtain a predicted image, including: cross-attention of the text information and the potential representation of the noise of each cross-attention layer in the controllable diffusion model to obtain the text-related embedding of each layer, wherein the cross-attention layer includes a coarse layer and a fine layer; cross-attention of the content attribute features and the potential representation of the noise of each layer of the coarse layer to obtain the content-related embedding of each layer of the coarse layer; cross-attention of the stroke attribute features and the potential representation of the noise of each layer of the fine layer to obtain the stroke-related embedding of each layer of the fine layer; cross-attention of the color attribute features and the potential representation of the noise of each layer of the fine layer to obtain the color-related embedding of each layer of the fine layer; integrating the text-related embedding of each layer into all cross-attention layers of the controllable diffusion model, integrating the content-related embedding of each layer of the coarse layer into each layer of the coarse layer, integrating the stroke-related embedding of each layer of the fine layer into each layer of the fine layer, and integrating the color-related embedding of the fine layer into each layer of the fine layer for guidance, and combining with the target image to obtain a predicted image.

[0034] In this embodiment, the text information is cross-attended with the potential representation of the noise of each cross-attention layer in the controllable diffusion model, and the expression of the text-related embedding of each layer is obtained as follows:

[0035] in, , and are the linear layer weights of the diffusion latent features and text features, respectively. For each layer of text-dependent embedding, is the cross attention function of text features, is the text information, z is the potential representation of the noise of each cross attention layer, They are the text information corresponding to the cross attention function. value, For the text-related embedding of each layer, it should be noted that the cross-attention layer contains 16 layers, the coarse layer includes the 4th to 8th layers, and the fine layer is the remaining cross-attention layers except the 4th to 8th layers; the deep layer in the middle of U-Net is the coarse layer, and the shallow layers at both ends are the fine layers.

[0036] The content attribute features are cross-attended with the potential representation of the noise in the coarse layer of the controllable diffusion model to obtain the content-related embedding of each layer of the coarse layer; the brush stroke attribute features are cross-attended with the potential representation of the noise in the fine layer of the controllable diffusion model to obtain the brush stroke-related embedding of each layer of the fine layer; the color attribute features are cross-attended with the potential representation of the noise in the fine layer of the controllable diffusion model to obtain the color-related embedding of each layer of the fine layer:

[0037] in, is the relevant embedding of image attribute information at each layer, , is the cross attention function corresponding to content, stroke and color image features, is the content-dependent embedding for each layer of the coarse layer, For each layer of fine-grained strokes, is the color-dependent embedding for each layer of the fine layer, For different attribute characteristics, , is the content attribute feature, is the brush stroke attribute feature, is the color attribute feature, , and shared linear layers representing content, stroke, and color image attributes, respectively, They are the content image, stroke image and color image corresponding to the cross attention function. Value, when the content attribute feature is cross-attended with the potential representation of the noise of the coarse layer of the controllable diffusion model, z is the potential representation of the noise of each coarse layer corresponding to the cross-attention layer, when the stroke attribute feature is cross-attended with the potential representation of the noise of the fine layer of the controllable diffusion model, z is the potential representation of the noise of each fine layer corresponding to the cross-attention layer, when the color attribute feature is cross-attended with the potential representation of the noise of the fine layer of the controllable diffusion model, z is the potential representation of the noise of each fine layer corresponding to the cross-attention layer, where the cross-attention function of the text feature and the cross-attention function of the image feature can be calculated using the following expressions:

[0038] in, is the dimension of the cross attention function space, T is the transpose, are the Q, K, and V values ​​of the cross attention function.

[0039] The text-related embedding of each layer is integrated into all cross-attention layers of the controllable diffusion model, the content-related embedding of each layer of the coarse layer is integrated into each layer of the coarse layer, the stroke-related embedding of each layer of the fine layer is integrated into each layer of the fine layer, and the color-related embedding of the fine layer is integrated into each layer of the fine layer for guidance, and combined with the target image to obtain the predicted image.

[0040] As an optional implementation method, the text-related embedding of each layer is integrated into all cross-attention layers of the controllable diffusion model, the content-related embedding of each layer of the coarse layer is integrated into each layer of the coarse layer, the stroke-related embedding of each layer of the fine layer is integrated into each layer of the fine layer, and the color-related embedding of the fine layer is integrated into each layer of the fine layer for guidance, and combined with the target image to obtain a predicted image, including: processing the text-related embedding, content-related embedding, stroke-related embedding and color-related embedding through a learnable gate to obtain target injection style information and target injection content information; guiding the controllable diffusion model based on the target injection style information and the target injection content information to obtain the noise of the predicted image; obtaining the predicted image based on the target image features containing noise and the noise of the predicted image.

[0041] In this embodiment, the text-related embedding, content-related embedding, stroke-related embedding, and color-related embedding are processed through a learnable gate to obtain expressions of target injection style information and target injection content information:

[0042] Among them, the gate function ,here is a learnable temperature-controlled activation function. For a feed-forward network, selectively embed the brush strokes and color embedding Fuse into layers with more local information to inject style information In addition, embedding content Integrate into layers with more global information, inject content information ,On this basis, various conditions can be adaptively fused and injected into the ,diffusion model, and the learnable gating parameters can flexibly adapt to different ,injection requirements, preventing excessive information from leaking into the ,model, and achieving better fine-grained image stylization.

[0043] Training and testing process: Given a training data set , the text control information and image control information will be encoded by the text encoder and image encoder respectively, and the dimension is The text embedding and dimension are image embedding, where B represents the batch size, and Represent the number of channels for text and image embedding, refers to the feature dimension of the cross-attention space. Among them, the visual embedding will be implicitly decoupled from the style attributes through the fine decoupling module to obtain multiple The style attribute embedding of the image is shown in Figure 2. These conditional information will be further guided by the adaptive injection module to guide the features of the denoising model, thereby controlling the generation of the image. The entire network uses the MSE loss to train the model:

[0044] in, is the loss function of the entire network, is a compression encoder for the controllable diffusion model, is the input target image, is a Gaussian distribution, is the identity matrix, For expectations, To control information, is the prediction time step, The real noise introduced into the target image, For the denoising model The noise of the step prediction, It is a denoising network model.

[0045] Once the training is completed, the present invention can be directly applied to fine-grained and controllable image stylization tasks. In the testing phase, a 30-step DDIM sampler is used to generate stylized images under specified control information.

[0046] The consistency between the prompt information (text, content reference image, stroke reference image, and color reference image) and the generated image was constructed based on the CLIP cosine similarity, and the fidelity of the generated image was measured based on the quality assessment model. The model trained on the constructed simulation dataset achieved excellent performance in text alignment (27.8), style similarity (24.5), image fidelity (6.14), and average consistency of image sub-attributes (22.2). Compared with the most advanced algorithms, they have achieved different degrees of improvement. In addition, the subjective visualization results survey conducted on users showed that the image stylization effect of the present invention achieved the best user satisfaction (87.9%).

[0047] In an embodiment of the present invention, training data is obtained, wherein the training data is a set of sample data with aligned style attributes, and the sample data includes a content image, a stroke image, a color image, text information and a target image; feature decoupling processing is implicitly performed on the content image, the stroke image and the color image to obtain content attribute features, stroke attribute features and color attribute features; the content attribute features, the stroke attribute features, the color attribute features and the text information are fused and injected into a controllable diffusion model through an adaptive gating mechanism, and combined with the target image to obtain a predicted image, thereby solving the technical problem that multiple fine-grained style attributes in the reference image are not decoupled in the prior art, making it difficult to finely and flexibly control style transfer, and achieving a data set with a large number of fine style attributes by using a large language model and a text-to-image generation model, and expanding the text-to-image generation model, and constructing a multi-conditional image stylization framework, on the basis of which a fine-grained decoupling mechanism is established to separate multiple image attributes, and a gated feature injection mechanism is designed to adaptively inject multiple conditions to guide the diffusion model to synthesize stylized images, and generate the expected technical effect of accurately stylized images.

[0048] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0049] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0050] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0051] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0052] In addition, each functional unit in each embodiment of the present invention may be integrated into a first processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0053] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A finely controllable image stylization method based on a diffusion model, characterized in that: include: Acquire training data, wherein the training data is a set of sample data with aligned style attributes, and the sample data includes a content image, a stroke image, a color image, text information, and a target image; Implicitly perform feature decoupling processing on the content image, the brush stroke image and the color image to obtain content attribute features, brush stroke attribute features and color attribute features; The content attribute features, stroke attribute features, color attribute features and text information are fused through an adaptive gating mechanism and injected into the controllable diffusion model, and combined with the target image to obtain a predicted image.

2. The method according to claim 1, characterized in that Before acquiring the training data, the method further includes: Collect several content, stroke and color words to build a vocabulary library; Generate text descriptions of images based on the content and stroke vocabulary in the vocabulary library using the prompt template of the large language model. Based on the text descriptions of images, use the text-to-image generation model to synthesize images with multiple stroke styles. Generate canny edge maps based on images with multiple brush stroke styles; Combine color vocabulary and text description of the image to build text information of the controllable diffusion model, input the text information and canny edge map into the ControlNet model, and obtain a dataset with aligned stroke images, color images, content images and text information; A set of datasets with aligned stroke images, color images, content images, and text information is extracted as training data.

3. The method according to claim 1, characterized in that After acquiring the training data, the method further includes: The target image features are added with noise through the forward process to obtain the target image features containing noise.

4. The method according to claim 1, characterized in that: The implicit feature decoupling process is performed on the content image, the stroke image and the color image to obtain content attribute features, stroke attribute features and color attribute features, including: The global features of the content image, the stroke image and the color image are extracted respectively by using a pre-trained image encoder to obtain the global features of the content image, the global features of the stroke image and the global features of the color image; The global features of the content image, the global features of the stroke image and the global features of the color image are extracted respectively through the fine-grained decoupling module to obtain content attribute features, stroke attribute features and color attribute features.

5. The method according to claim 3, characterized in that: The content attribute features, stroke attribute features, color attribute features and text information are fused through an adaptive gating mechanism and injected into a controllable diffusion model, combined with a target image, to obtain a predicted image, including: Cross-attention is performed on the text information and the potential representation of the noise of each cross-attention layer in the controllable diffusion model to obtain the text-related embedding of each layer, where the cross-attention layer includes a coarse layer and a fine layer; Cross-attention is performed between the content attribute features and the potential representation of the noise of each layer of the coarse layer to obtain the content-related embedding of each layer of the coarse layer; Cross-attention is performed on the stroke attribute features and the potential representation of the noise of each layer of the fine layer to obtain the stroke-related embedding of each layer of the fine layer; Cross-attention is performed on the color attribute features and the potential representation of the noise of each layer of the fine layer to obtain the color-related embedding of each layer of the fine layer; The text-related embedding of each layer is integrated into all cross-attention layers of the controllable diffusion model, the content-related embedding of each layer of the coarse layer is integrated into each layer of the coarse layer, the stroke-related embedding of each layer of the fine layer is integrated into each layer of the fine layer, and the color-related embedding of the fine layer is integrated into each layer of the fine layer for guidance, and combined with the target image to obtain the predicted image.

6. The method according to claim 5, characterized in that The method integrates the text-related embedding of each layer into all cross-attention layers of the controllable diffusion model, integrates the content-related embedding of each layer of the coarse layer into each layer of the coarse layer, integrates the stroke-related embedding of each layer of the fine layer into each layer of the fine layer, and integrates the color-related embedding of the fine layer into each layer of the fine layer for guidance, and combines the target image to obtain a predicted image, including: The text-related embedding, content-related embedding, stroke-related embedding, and color-related embedding are processed through a learnable gate to obtain target injection style information and target injection content information; The controllable diffusion model is guided based on the target injection style information and the target injection content information to obtain the noise of the predicted image; A predicted image is obtained based on the noisy target image features and the noise of the predicted image.

7. A computer system, characterized in that include: One or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method of claim 1.

8. A computer-readable storage medium, characterized in that Computer executable instructions are stored, and when the instructions are executed, they are used to implement the method of claim 1.

9. A computer program product, characterized in that The invention comprises computer executable instructions, which are used to implement the method of claim 1 when being executed.

Images