Image generation method based on style feature injection

Abstract

The present application relates to a style feature injection-based image generation method, which can efficiently migrate the spatial layout and color texture of a style reference image on the basis of a text-to-image. The proposed method learns the spatial layout and preliminary style of an image using the Dreambooth binding feature capability on the basis of the Stable Diffusion text-to-image, and then performs attention injection to strengthen the color features and detail texture. Therefore, the proposed method has high generalization on various styles, and for a single specific style, no tedious parameter tuning is required, and it is highly sensitive to context features such as composition, texture, color, etc. In addition, the model has a small data requirement, only 3-5 training images are required, and only one attention reference image is required in the inference process, and no additional training is required. Therefore, the proposed style image generation method can adapt to a variety of different styles, so as to have higher freedom in personalized tasks.

Description

Technical Field

[0001] This invention is an image generation method based on a specific style. Background Technology

[0002] Image generation technology has broad application prospects in fields such as artistic creation, film special effects, and virtual reality. The application of Convolutional Neural Networks (CNNs) and Generative Adversarial Networks (GANs) has led to significant progress in image generation technology. Deep learning-based image generation methods, by learning from large amounts of image data, can generate more realistic and complex images. The emergence of Generative Adversarial Networks (GANs) has brought new breakthroughs to image generation.

[0003] Compared to GANs, Diffusion Models further enhance image generation capabilities, and the control constraints for image generation are no longer through specific loss functions, but rather decoupled into conditions (textual information, auxiliary image features, etc.). This results in stronger control over the latent space and greater practicality. Stable Diffusion models generate images by learning the noise distribution in an image and progressively reducing noise. They utilize deep learning techniques, including convolutional neural networks (CNNs) and self-attention mechanisms, to learn the features and structure of images. The model first compresses the image using an autoencoder (AE), then generates the image in the latent space of the compressed image using a diffusion model (DDPM). By training on a large amount of image data, the model can learn the details and textures of images, thereby generating realistic images.

[0004] However, StableDiffusion can only generate images from real-world scenes. Relying solely on the diffusion model cannot customize image generation; therefore, a single diffusion model cannot generate styled images. Before the advent of neural networks, the basic idea of ​​image style transfer was to analyze images of a specific style, build a mathematical or statistical model for it, and then adjust the target image to conform to that model. While this method could achieve style transfer, its applicability was narrow, and the transfer effect was not ideal. Deep learning-based style transfer methods, by learning the content and style features of images, can more effectively apply the style of one image to another. For example, the classic neural style transfer method uses the VGG-16 network to extract image features and generates stylized images through content loss and style loss.

[0005] Combining StableDiffusion image generation with traditional style transfer tasks remains a problem to be solved. Therefore, it is necessary to propose an efficient style image generation method. Summary of the Invention

[0006] To address the problem that a single diffusion model cannot generate images with a specific style, this invention proposes an image generation method based on style feature injection. The proposed method can efficiently transfer the spatial layout and color texture of a style reference image based on a textural image. For example... Figure 1 As shown, the technical solution for implementing this invention includes two processes: network training and inference.

[0007] The network training process includes the following steps:

[0008] Step 1: Use a large language model to assign cue words to specific styles, such as "style_00,oil painting" for the oil painting style. Then, bind the cue words to specific styles.

[0009] Step 2, Data Augmentation. For a dataset of a specified style (3-5 images, 512*512 pixels), augment the training set using methods such as random cropping, random blurring, and random flipping;

[0010] Step 3: Input the enhanced training data into the Dreambooth+Lora network for training;

[0011] Step 4: Save the LoRa weights for inference;

[0012] Furthermore, in step three, the pre-trained model for Dreambooth+Lora is StableDiffusion2.1. During forward inference, the SD model repeatedly calls U-Net to remove the predicted noise residuals from the original noise matrix, obtaining progressively denoised Latent Features. These Latent Features are then reconstructed into pixel-level images using the VAE's Decoder structure. The VAE is based on an Encoder-Decoder architecture, where the Encoder architecture converts the image into low-dimensional Latent Features, which are then used as input to U-Net. The Decoder structure reconstructs and restores the low-dimensional Latent Features into pixel-level images.

[0013] The SD VAE model has three basic components: a GSC component, a Downsample component, and an Upsample component. The SD VAE Encoder contains three DownBlock modules, one ResNetBlock module, and one MidBlock module, compressing the input image to the Latent space and converting it to the Gaussian Distribution. The VAE Decoder, on the other hand, takes Latent space features as input and reconstructs them into a pixel-level image as output. It contains three UpBlock modules, one ResNetBlock module, and one MidBlock module.

[0014] Stablediffusion 2.1's text encoder is primarily the CLIP model. CLIP mainly consists of two models: a Text Encoder and an Image Encoder. The Text Encoder extracts text features and can use a commonly used NLP text transformer model. The Image Encoder extracts image features and uses a CNN / vision transformer model (ResNet and ViT). The CLIP model's task is to extract embedding vectors from the labeled text and image using the Text Encoder and Image Encoder respectively. Then, it uses cosine similarity to compare the similarity between the two embedding vectors to determine whether the randomly selected labeled text and image match. Gradient backpropagation is then performed for continuous optimization and training.

[0015] The U-Net in Stable Diffusion adds a ResNetBlock module (including TimeEmbedding), a Spatial Transformer module (SelfAttention + CrossAttention + FeedForward), and CrossAttnDownBlock, CrossAttnUpBlock, and CrossAttnMidBlock modules to the traditional U-Net. In Stable Diffusion, the U-Net model is a key core component, capable of predicting noise residuals and reconstructing the input feature matrix using sampling methods (scheduling algorithms: PNDM, DDIM, K-LMS, etc.), gradually transforming it from random Gaussian noise into latent features of the image.

[0016] Furthermore, the specific training process of Dreambooth+Lora in step three is as follows:

[0017] (1) Using a pre-trained text-to-image diffusion model This model is based on a given initial noisy image ∈ ~ Generate an image given the condition vector c = Γ(P) This model is trained using squared error loss to process images or latent codes z with varying degrees of noise. t :=α t x+σ t For denoising, the loss function is as follows:

[0018]

[0019] Where x is the real image, c is the conditional vector (e.g., obtained from the text prompt), and α t ,σ t ,w t These are terms that control the noise process and sample quality; these terms are the diffusion process time. The function.

[0020] (2) Assemble Lora weights for the Dreambooth model. This significantly reduces the required parameter size by encoding task-specific parameter increments. For the pre-trained model weights Φ0, the fine-tuning parameter is |ΔΦ|. Typically, Φ0 and |ΔΦ| have the same dimension. Lora uses a parameter set Θ, much smaller than |ΔΦ|, instead of |ΔΦ|, as shown in the following formula:

[0021]

[0022] In the style feature injection-based image generation method, the weight initialization method of Lora configuration is set to gaussian, the target_module is specified as ["to_k","to_q","to_v","to_out.0","add_k_proj","add_v_proj"], and lora_dropout=0.01 is added.

[0023] (3) Set the optimizer to AdamW optimizer. Learning rate lr = 5e-4, adaim_epsilon = 1e-8.

[0024] (4) Specify parameters such as training epoch and warm-up steps to train the LOA weights.

[0025] The network inference process is specifically described as follows:

[0026] Step 1: Construct the network input. Bind the main prompt to be generated to specific cue words used in training. Taking "boat" and "oil painting style" as an example, the input for the inference stage is "boat in style_00,oilpainting".

[0027] Step 2: Using the LoRa weights obtained in the training step, generate preliminary results with spatial layout and preliminary style features;

[0028] Step 3: Attention injection is applied to the preliminary results obtained in Step 2 to enhance the stylized color and texture. Attention injection will be applied to the 7th-12th Decoder layers in Stable Diffusion. After iterative iteration, a clear, stylized image will be obtained.

[0029] Furthermore, the specific process of attention injection in step three is as follows:

[0030] (1) Select an image from the style training set as the style reference image, and perform DDIM Inversion on the style reference image and the content image obtained in step one respectively:

[0031] i) Use the DDIM method to gradually add noise to the content map and style map respectively. This will result in a noisy content image. Style Image Noise Both the noise addition and denoising processes are set to be T steps.

[0032] ii) During the DDIM inversion process, self-attention features of the image are collected at each time step. For the content graph, Q is saved. C For style diagrams, save K S V S .

[0033] (2) Initial latent features are obtained by mixing using the AdaIN normalization method. For content image noise... and style image noise The normalization formula is:

[0034]

[0035] in The initial noise obtained from the mixing. μ is the mean, σ is the standard deviation; initial potential noise. Retained The content information, along with the channel mean and standard deviation, are compared with... Alignment. The mean and standard deviation can roughly describe the tone of an image; that is, for the content image, destylating is performed first, and then stylizing to the style of the reference image.

[0036] (3) Stylized images are obtained by reverse sampling;

[0037] i) In step T of the reverse process, based on the self-attention information saved in step one, It is gradually injected into the self-attention layer of the UET network.

[0038] ii) Controlling style confidence. During the inverse sampling process, the stylized... And the content image saved in the first step These are combined and used throughout the iteration process to preserve the original content in the styled graph. The style confidence control formula is as follows:

[0039]

[0040] A higher γ retains more content, while a lower γ enhances the style transfer effect.

[0041] iii) After style injection at each self-attention layer, attention scaling is performed. Experiments show that if the self-attention method in Stable Diffusion is applied directly, after style injection in 1), The standard deviation decreases, resulting in an overly smooth and distorted attention map. Therefore, an attention temperature scaling parameter τ is used to correct for the decreased standard deviation. The attention scaling formula is:

[0042]

[0043] The default value for τ is 1.5.

[0044] After attention is injected, a clear stylized image will be obtained.

[0045] The advantages of this method are as follows: The proposed style feature injection-based image generation method, building upon StableDiffusion text-based images, leverages Dreambooth's feature binding capabilities to learn the spatial layout and initial style of the image, and then performs attention injection to enhance color features and detailed textures. Therefore, the proposed method exhibits high generalization across various styles, requires no tedious parameter tuning for a single specific style, and is highly sensitive to contextual features such as composition, texture, and color. Furthermore, the model requires minimal data, needing only 3-5 training images, and only one attention reference image is needed during inference, without requiring additional training, thus demonstrating strong robustness. Attached Figure Description

[0046] Appendix Figure 1 This is a schematic diagram of the algorithm framework of the style image generation method based on feature injection of the present invention.

[0047] Appendix Figure 2 This is a diagram illustrating the DDIM inversion process for the style enhancement module.

[0048] Appendix Figure 3 This is a diagram illustrating the reversion process of the style enhancement module.

[0049] Appendix Figure 4 Schematic diagram of DDIM noise addition to the content image.

[0050] Appendix Figure 5 Style reference image: DDIM noise addition illustration.

[0051] Appendix Figure 6 A diagram illustrating the initial noise reversion of AdaIN. Detailed Implementation

[0052] The following details an image generation method based on style feature injection according to the present invention, providing a further specific description of the method. It is important to note that the following embodiments are only for further illustrative purposes and should not be construed as limiting the scope of protection of this method. Any non-essential improvements and adjustments made to this method by those skilled in the art based on the above description still fall within the scope of protection of this invention.

[0053] This invention proposes an image generation method based on style injection, such as... Figure 1 As shown, this method includes two processes: network training and inference.

[0054] The network training process includes the following steps:

[0055] Step 1: Use a large language model to assign cue words to specific styles, such as "style_00,oil painting" for the oil painting style. Then, bind the cue words to specific styles.

[0056] Step 2, Data Augmentation. For a dataset of a specified style (3-5 images, 512*512 pixels), augment the training set using methods such as random cropping, random blurring, and random flipping;

[0057] Step 3: Input the enhanced training data into the Dreambooth+Lora network for training;

[0058] Step 4: Save the LoRa weights for inference;

[0059] Furthermore, in step three, the pre-trained model for Dreambooth+Lora is StableDiffusion2.1. During forward inference, the SD model repeatedly calls U-Net to remove the predicted noise residuals from the original noise matrix, obtaining progressively denoised Latent Features. These Latent Features are then reconstructed into pixel-level images using the VAE's Decoder structure. The VAE is based on an Encoder-Decoder architecture, where the Encoder architecture converts the image into low-dimensional Latent Features, which are then used as input to U-Net. The Decoder structure reconstructs and restores the low-dimensional Latent Features into pixel-level images.

[0060] The SD VAE model has three basic components: a GSC component, a Downsample component, and an Upsample component. The SD VAE Encoder contains three DownBlock modules, one ResNetBlock module, and one MidBlock module, compressing the input image to the Latent space and converting it to the Gaussian Distribution. The VAE Decoder, on the other hand, takes Latent space features as input and reconstructs them into a pixel-level image as output. It contains three UpBlock modules, one ResNetBlock module, and one MidBlock module.

[0061] Stablediffusion 2.1's text encoder is primarily the CLIP model. CLIP mainly consists of two models: a Text Encoder and an Image Encoder. The Text Encoder extracts text features and can use a commonly used NLP text transformer model. The Image Encoder extracts image features and uses a CNN / vision transformer model (ResNet and ViT). The CLIP model's task is to extract embedding vectors from the labeled text and image using the Text Encoder and Image Encoder respectively. Then, it uses cosine similarity to compare the similarity between the two embedding vectors to determine whether the randomly selected labeled text and image match. Gradient backpropagation is then performed for continuous optimization and training.

[0062] The U-Net in Stable Diffusion adds a ResNetBlock module (including TimeEmbedding), a Spatial Transformer module (SelfAttention + CrossAttention + FeedForward), and CrossAttnDownBlock, CrossAttnUpBlock, and CrossAttnMidBlock modules to the traditional U-Net. In Stable Diffusion, the U-Net model is a key core component, capable of predicting noise residuals and reconstructing the input feature matrix using sampling methods (scheduling algorithms: PNDM, DDIM, K-LMS, etc.), gradually transforming it from random Gaussian noise into latent features of the image.

[0063] Furthermore, the specific training process of Dreambooth+Lora in step three is as follows:

[0064] (1) Using a pre-trained text-to-image diffusion model The model is based on a given initial noisy image. Generate an image given the condition vector c = Γ(P) This model is trained using squared error loss to process images or latent codes z with varying degrees of noise. t :=α t x+σ t For denoising, the loss function is as follows:

[0065]

[0066] Where x is the real image, c is the conditional vector (e.g., obtained from the text prompt), and α t ,σ t ,w t These are terms that control the noise process and sample quality; these terms are the diffusion process time. The function.

[0067] (2) Assemble Lora weights for the Dreambooth model. This significantly reduces the required parameter size by encoding task-specific parameter increments. For the pre-trained model weights Φ0, the fine-tuning parameter is |ΔΦ|. Typically, Φ0 and |ΔΦ| have the same dimension. Lora uses a parameter set Θ, much smaller than |ΔΦ|, instead of |ΔΦ|, as shown in the following formula:

[0068]

[0069] In the style feature injection-based image generation method, the weight initialization method of Lora configuration is set to gaussian, the target_module is specified as ["to_k","to_q","to_v","to_out.0","add_k_proj","add_v_proj"], and lora_dropout=0.01 is added.

[0070] (3) Set the optimizer to AdamW optimizer. Learning rate lr = 5e-4, adaim_epsilon = 1e-8.

[0071] (4) Specify parameters such as training epoch and warm-up steps to train the LOA weights.

[0072] The network inference process is specifically described as follows:

[0073] Step 1: Construct the network input. Bind the main prompt to be generated to specific cue words used in training. Taking "boat" and "oil painting style" as an example, the input for the inference stage is "boat in style_00,oilpainting".

[0074] Step 2: Using the LoRa weights obtained in the training step, generate preliminary results with spatial layout and preliminary style features;

[0075] Step 3: Attention injection is applied to the preliminary results obtained in Step 2 to enhance the stylized color and texture. Attention injection will be applied to the 7th-12th Decoder layers in Stable Diffusion. After iterative iteration, a clear, stylized image will be obtained.

[0076] Furthermore, the specific process of attention injection in step three is as follows:

[0077] (1) Select an image from the style training set as the style reference image, and perform DDIM Inversion on the style reference image and the content image obtained in step one respectively:

[0078] i) Use the DDIM method to gradually add noise to the content map and style map respectively. This will result in a noisy content image. Style Image Noise Both the noise addition and denoising processes are set to be T steps.

[0079] ii) During the DDIM inversion process, self-attention features of the image are collected at each time step. For the content graph, Q is saved. C For style diagrams, save KS V S .

[0080] (2) Initial latent features are obtained by mixing using the AdaIN normalization method. For content image noise... and style image noise The normalization formula is:

[0081]

[0082] in The initial noise obtained from the mixing. μ is the mean, σ is the standard deviation; initial potential noise. Retained The content information, along with the channel mean and standard deviation, are compared with... Alignment. The mean and standard deviation can roughly describe the tone of an image; that is, for the content image, destylating is performed first, and then stylizing to the style of the reference image.

[0083] (3) Stylized images are obtained by reverse sampling;

[0084] i) In step T of the reverse process, based on the self-attention information saved in step one, It is gradually injected into the self-attention layer of the UET network.

[0085] ii) Controlling style confidence. During the inverse sampling process, the stylized... And the content image saved in the first step These are combined and used throughout the iteration process to preserve the original content in the styled graph. The style confidence control formula is as follows:

[0086]

[0087] A higher γ retains more content, while a lower γ enhances the style transfer effect.

[0088] iii) After style injection at each self-attention layer, attention scaling is performed. Experiments show that if the self-attention method in Stable Diffusion is applied directly, after style injection in 1), The standard deviation decreases, resulting in an overly smooth and distorted attention map. Therefore, an attention temperature scaling parameter τ is used to correct for the decreased standard deviation. The attention scaling formula is:

[0089]

[0090] The default value for τ is 1.5.

[0091] After attention is injected, a clear stylized image will be obtained.

[0092] In this invention, the training set used during network training is the publicly available dataset from the Style Image Generation track of the 4th Jitu AI Challenge, which contains 28 different styles. Each style has three training images. During inference, one image is randomly selected from the three training images as the style reference image. During training, hyperparameters such as epoch, warm-up steps, and data augmentation quantity need to be specified. Specifically, the epoch is generally selected as a constant between 300 and 1500 that is divisible by 100, the warm-up steps are generally set to half the epoch value, and the data augmentation quantity is generally set to 1 / 10 of the epoch value. Generally, for most styles, epoch = 1000, warm-up = 500, and items = 100. In general, if the style features are not obvious, the number of epoch steps can be increased appropriately. If the style features are obvious and the generated content emphasizes text consistency, it is recommended to reduce the number of epochs to avoid overfitting. The Lora configuration initializes weights using Gaussian, specifying `target_module` as `["to_k","to_q","to_v","to_out.0","add_k_proj","add_v_proj"]`, and adding `lora_dropout = 0.01`. The optimizer is set to AdamW. The learning rate is `lr = 5e-4`, and `adaim_epsilon = 1e-8`. During inference, the number of steps for image generation is set to `step = 25`. In the style enhancement module, the number of steps for DDIM inversion and reversion is set to `step = 50`. That is, the number of steps `T` for both noise addition and denoising is 50. Additionally, the hyperparameter `tao` for attention temperature scaling is set to `1.5`. The blending parameter `gamma` for content and style maps is set to `0.7` by default. Experiments show that when `gamma` is 0.7, it can effectively preserve the content of the content map while maximizing the restoration of the style features of the style reference map, and minimizing image quality degradation. Style injection will be applied to layers 7-12 of the Self-attention layer in UET. The training and inference environment of the proposed method is a PyTorch framework with an Nvidia RTX 4090 GPU and an Intel Core™ i7-7820X CPU at 3.60 Hz.

[0093] Table 1. Different Image Generation Methods for Different Styles: CLIP-R, FID, DinoV2

[0094]

[0095] This invention aims to verify the effectiveness and generalization ability of the proposed method by testing it on images of 28 different styles to demonstrate its performance in practical applications. To achieve this goal, we used the publicly available dataset from the Style Image Generation track of the 4th Jitu AI Challenge, which contains images of various styles. During testing, several baseline methods were employed, including Textual inversion and Style align. Textual inversion is a style transfer method based on text reversal, which achieves personalized style transfer by combining textual information with image style. However, this method has certain limitations in practical applications and struggles to meet diverse style transfer needs. Style align is a style feature alignment method that achieves style transfer by aligning style features in an image. However, due to its high requirements for the reference image, this method performs poorly in style transfer tasks and is difficult to meet the needs of practical applications. Compared with traditional style transfer methods, the proposed method has stronger generalization ability and higher image generation quality. As shown in Table 1, the proposed method exhibits stronger generalization across different styles, higher image generation quality, and higher text consistency. This indicates that our method performs better in practical applications.

[0096] Furthermore, we conducted visualization tests on the proposed method. The visualization results show that the proposed style-injection-based image generation method is stable and effective. This further verifies the feasibility and effectiveness of our method in practical applications. Experimental results demonstrate that the proposed method exhibits stronger generalization across different styles, higher image generation quality, and higher text consistency. Simultaneously, the visualization test results further validate the feasibility and effectiveness of our method in practical applications. Therefore, we can conclude that the proposed style-injection-based image generation method is a stable and effective image generation method with broad application prospects.

Claims

1. An image generation method based on style feature injection, characterized in that, Includes the following steps: S1. The network training process is described in detail as follows: S11. Use a large language model to assign prompt words to specific styles and bind the prompt words to specific styles; S12, Data Augmentation: For datasets with a specified style, the training set is augmented using methods such as random cropping, random blurring, and random flipping. S13. Input the enhanced training data into the Dreambooth+Lora network for training; S14. Save the LoRa weights for inference; S2, the network inference process, specifically described as follows: S21. Construct the input to the network by binding the main prompt to be generated with the specific prompt words used in training; S22. Using the LoRa weights obtained in the training step, generate preliminary results with spatial layout and preliminary style features; S23. Attention injection is applied to the preliminary results obtained in step two to enhance the stylized color and texture. This attention injection will be applied to the 7th-12th Decoder layers in Stable Diffusion 2.

1. After iterative iteration, a clear stylized image will be obtained. ; Step S23 is as follows: (1) Select an image from the dataset of the specified style as the style reference image, and perform DDIM Inversion on the style reference image and the preliminary results obtained in S22 respectively: i) Use the DDIM method to add noise to the content image and style image step by step; obtain the noise in the content image. Style image noise The noise addition and denoising processes are both set to T steps; ii) During the DDIM inversion process, self-attention features of the image are collected at each time step; for the content graph, they are saved. For style diagrams, save , ; (2) Initial latent features were obtained by mixing using the AdaIN normalization method; For content image noise and style image noise The normalization formula is: in The initial noise obtained by mixing; The mean, Standard deviation; initial latent noise Retained The content information, along with the channel mean and standard deviation, are compared with... Alignment; the mean and standard deviation can describe the tone of the image, that is, for the content image, destylating is performed first, and then stylizing to the style of the reference image; (3) Stylized images are obtained by reverse sampling; i) In step T of the reverse process, based on the self-attention information stored in S23(1), , , Gradually inject it into the self-attention layer of the UNET network; ii) Control style confidence; During the inverse sampling process, the stylized query ( The query of the content graph saved in S23(1) and S23(1) These are combined and used throughout the iteration process to preserve the original content in the stylized diagram; the style confidence control formula is as follows: Among them, the higher Keep more content, and lower This enhances the effect of style transfer; iii) After style injection at each self-attention layer, attention scaling is performed; if the self-attention method in Stable Diffusion 2.1 is applied directly, after style injection in S23(3)i), The standard deviation will decrease, resulting in an overly smooth and distorted attention map; therefore, an attention temperature scaling parameter is used. The reduced standard deviation is corrected; the attention scaling formula is: in, ; After attention is injected, a clear stylized image will be obtained.

2. The image generation method based on style feature injection according to claim 1, characterized in that, StableDiffusion 2.1 consists of VAE, text-encoder, and unet; (1)VAE VAE is based on an Encoder-Decoder architecture, where the Encoder architecture converts the image into low-dimensional Latent features and uses them as input to UNet; the Decoder structure can reconstruct and restore the low-dimensional Latent features into a pixel-level image. The VAE model has three basic components: i) GSC components: GroupNorm + Swish + Conv ii) Downsample component: Padding + Conv iii) Upsample component: Interpolate + Conv Meanwhile, the VAE Encoder part includes three DownBlock modules, one ResNetBlock module, and one MidBlock module, which compress the input image into the Latent space and convert it into the Gaussian Distribution. The VAE Decoder, on the other hand, takes Latent spatial features as input and reconstructs them into pixel-level images as output; it contains three UpBlock modules, one ResNetBlock module, and one MidBlock module. (2) Text encoder The text encoder in Stable diffusion 2.1 is the CLIP model. CLIP consists of two models: Text Encoder and Image Encoder. The Text Encoder extracts text features using the commonly used text transformer model in NLP. The Image Encoder extracts image features using the CNN / vision transformer model. The CLIP model's task is to extract the embedding vector of the labeled text using the Text Encoder and the image embedding vector using the Image Encoder. Then, it uses cosine similarity to compare the similarity between the two embedding vectors to determine whether the randomly selected labeled text and image match. Gradient backpropagation is then performed to continuously optimize the training. (3)Unet Stable Diffusion 2.1's U-Net adds ResNetBlock, Spatial Transformer, and CrossAttnDownBlock, CrossAttnUpBlock, and CrossAttnMidBlock modules to the traditional U-Net. Among them, ResNetBlock includes Time Embedding, and Spatial Transformer includes SelfAttention, CrossAttention, and FeedForward. In Stable Diffusion 2.1, the U-Net model is a key core component, used to predict noise residuals and reconstruct the input feature matrix using the sampling method, gradually transforming it from random Gaussian noise into latent features of the image. Specifically, during the forward inference process, the StableDiffusion2.1 model repeatedly calls U-Net to remove the predicted noise residuals from the original noise matrix, obtaining the gradually denoised image Latent Feature. Then, the Latent Feature is reconstructed into a pixel-level image through the VAE's Decoder structure.

3. The image generation method based on style feature injection according to claim 1, characterized in that, Step S13 uses Dreambooth to bind style features and introduces Lora, using a low-rank matrix to adjust the original weight matrix, thus achieving efficient customization of the model. The process is described as follows: (1) Using a pre-trained text-to-image diffusion model The model is based on a given initial noisy image. and condition vector Generate image under the condition This model is trained using squared error loss to process images or latent code with varying levels of noise. For denoising, the loss function is as follows: Where x is the real image, c is the conditional vector, and These are terms that control the noise process and sample quality; these terms are the diffusion process time. The function; (2) Assemble Lora weights for the Dreambooth model; significantly reduce the required parameter size by encoding task-specific parameter increments; for pre-trained model weights Fine-tuning parameters are Lora uses a much smaller size parameter set replace The specific formula is as follows: In the style feature injection-based image generation method, the weight initialization method of Lora configuration is set to gaussian, the target_module is specified as ["to_k", "to_q", "to_v", "to_out.0", "add_k_proj", "add_v_proj"], and lora_dropout=0.01 is added; (3) Set the optimizer to AdamW optimizer; Learning rate lr=5e-4, adaim_epsilon=1e-8; (4) Specify parameters such as training epoch and warm-up steps to train the LOA weights.

Images