Image generation method based on style feature injection

Abstract

The invention provides an image generation method based on style feature injection, which can efficiently migrate the spatial layout and color texture of a style reference image on the basis of a text image. According to the method provided by the invention, on the basis of a Stable Diffusion text map, the spatial layout and the preliminary style of an image are learned by utilizing the capability of binding features of Dreambooth, and then attention injection is executed to strengthen color features and detail textures. Therefore, the method provided by the invention has high generalization in various styles, does not need tedious parameter adjustment for a single specific style, and is highly sensitive to context features such as composition, texture, color and the like. In addition, the model needs less data, only 3-5 training images are needed, only one attention reference image is needed in the reasoning process, additional training is not needed, and the robustness is very high. Therefore, the style image generation method provided by the invention can adapt to various different styles so as to have a higher degree of freedom in a personalized task.

Description

Technical Field

[0001] The invention relates to an image generation method based on a specific style. Background Art

[0002] Image generation technology has broad application prospects in the fields of artistic creation, film special effects, virtual reality, etc. With the application of convolutional neural networks (CNN) and generative adversarial networks (GAN), image generation technology has made significant progress. Image generation methods based on deep learning can generate more realistic and complex images by learning a large amount of image data. The emergence of generative adversarial networks (GAN) has brought new breakthroughs in image generation.

[0003] Compared with GAN, the image generation capability of Diffusion Model is further enhanced, and the control constraints of image generation are no longer through specific loss, but decoupled into some conditions (textual information, auxiliary image features, etc.), which has stronger control over Latent Space and higher practicality. The Stable Diffusion model generates images by learning the noise distribution in the image and gradually reducing the noise. It uses deep learning techniques, including convolutional neural networks (CNNs) and self-attention mechanisms, to learn the features and structure of images. The model first compresses the image with an autoencoder (AE), and then generates the image with a diffusion model (DDPM) in the latent space where the compressed image is located. By training a large amount of image data, the model can learn the details and texture of the image, thereby generating realistic images.

[0004] However, StableDiffusion can only generate images in real scenes. Image generation cannot be customized by relying solely on diffusion models. Therefore, a single diffusion model cannot generate style images. Before the emergence of neural networks, the basic idea of ​​image style transfer was to analyze images of a specific style, establish a mathematical or statistical model for it, and then adjust the target image to conform to the model. Although this method can achieve style transfer, its scope of application is narrow and the transfer effect is not ideal. The style transfer method based on deep learning can more effectively apply the style of one image to another by learning the content and style features of the image. For example, the classic neural style transfer method uses the VGG-16 network to extract image features and generates stylized images through content loss and style loss.

[0005] How to combine StableDiffusion image generation with traditional style migration tasks is still a problem that needs to be solved. Therefore, it is necessary to propose an efficient style image generation method. Summary of the invention

[0006] In order to solve the problem that a single diffusion model cannot generate images with a specific style, this paper proposes an image generation method based on style feature injection. The proposed image generation method can efficiently transfer the spatial layout and color texture of the style reference image based on the style map. Figure 1 As shown, the technical solution for implementing the present invention includes two processes: network training and reasoning.

[0007] The network training process includes the following steps:

[0008] Step 1: Use a large language model to specify prompt words for a specific style. For example, the prompt word for oil painting style is "style_00, oil painting". Bind the prompt word to the specific style;

[0009] Step 2: Data enhancement: For a data set of a specified style (3-5 images, 512*512 pixels), use random cropping, random blurring, random flipping and other methods to expand the training set and perform data enhancement;

[0010] Step 3: Input the enhanced training data into the Dreambooth+Lora network for training;

[0011] Step 4: Save the lora weight for reasoning;

[0012] Furthermore, the pre-trained model of Dreambooth+Lora in step three is StableDiffusion2.1. In the forward reasoning process, the SD model removes the predicted noise residual from the original noise matrix by repeatedly calling U-Net, obtains the Latent Features of the image after gradual denoising, and then reconstructs the Latent Features into pixel-level images through the Decoder structure of VAE. VAE is based on the Encoder-Decoder architecture, in which the Encoder architecture converts the image into low-dimensional Latent features and serves as the input of Unet. The Decoder structure can reconstruct the low-dimensional Latent features and restore them to pixel-level images.

[0013] There are three basic components in the SD VAE model: GSC component, Downsample component, and Upsample component. At the same time, the SD VAE Encoder part contains three DownBlock modules, one ResNetBlock module, and one MidBlock module, which compresses the input image into the Latent space and converts it into a Gaussian Distribution. The VAE Decoder part is just the opposite. It inputs the Latent space features and reconstructs them into pixel-level images as output. It contains three UpBlock modules, one ResNetBlock module, and one MidBlock module.

[0014] The text encoder of Stablediffusion2.1 is mainly the CLIP model. CLIP mainly includes two models: TextEncoder and Image Encoder. The Text Encoder is used to extract the features of the text, and the text transformer model commonly used in NLP can be used as the Text Encoder; while the Image Encoder is mainly used to extract the features of the image, and the CNN / vision transformer model (ResNet and ViT) is used as the Image Encoder. The task of the CLIP model is mainly to extract the embedding vectors of the label text and the picture through the Text Encoder and Image Encoder respectively, and then use the cosine similarity to compare the similarity of the two embedding vectors to determine whether the randomly extracted label text and picture match, and perform gradient backpropagation to continuously optimize the training.

[0015] The Unet in Stable Diffusion adds ResNetBlock (including TimeEmbedding) module, Spatial Transformer (SelfAttention+CrossAttention+FeedForward) module and CrossAttnDownBlock, CrossAttnUpBlock and CrossAttnMidBlock modules on the basis of traditional Unet. In Stable Diffusion, the U-Net model is a key core part that can predict noise residuals and reconstruct the input feature matrix in combination with Samplingmethod (scheduling algorithm: PNDM, DDIM, K-LMS, etc.), gradually converting it from random Gaussian noise into the Latent Feature of the image.

[0016] Furthermore, the specific training process of Dreambooth+Lora in step 3 is as follows:

[0017] (1) Using a pre-trained text-to-image diffusion model The model is given an initial noise image image∈~ The image is generated with the conditional vector c = Γ(P) This model is trained with squared error loss to detect images or latent codes z with different levels of noise. t :=α t x+σ t ∈ is used for denoising, and the loss function is as follows:

[0018]

[0019] where x is the real image, c is the conditioning vector (e.g., obtained from a textual prompt), and α t ,σ t ,w t are terms that control the noise process and sample quality. These terms are the diffusion process time function.

[0020] (2) Assemble the Lora weights for the Dreambooth model. By encoding task-specific parameter increments, the size of the required parameters is greatly reduced. For the pre-trained model weights Φ 0 , the fine-tuning parameter is |ΔΦ|. Usually, Φ 0 Equal to the dimension of |ΔΦ|. Lora uses a parameter set Θ that is much smaller than |ΔΦ| to replace |ΔΦ|. The specific formula is as follows:

[0021]

[0022] In the image generation method based on style feature injection, set the weight initialization method of Lora configuration to Gaussian, specify target_module to ["to_k", "to_q", "to_v", "to_out.0", "add_k_proj", "add_v_proj"], and add lora_dropout = 0.01.

[0023] (3) Set the optimizer to AdamW optimizer. Learning rate lr = 5e-4, adaim_epsilon = 1e-8.

[0024] (4) Specify parameters such as training epochs and warm up steps to train lora weights.

[0025] The network reasoning process is specifically described as follows:

[0026] Step 1: Construct the input of the network. Bind the subject prompt to be generated with the specific prompt words used in training. Taking boat and oil painting style as an example, the input of the inference stage is "boat in style_00, oil painting";

[0027] Step 2: Use the lora weights obtained in the training step to generate preliminary results with spatial layout and preliminary style features;

[0028] Step 3: Perform attention injection on the preliminary results obtained in step 2 to enhance the color texture of the style. Attention injection will be applied to the 7th to 12th Decoder layers in Stable Diffusion. After repeated iterations, a clear stylized image will be obtained.

[0029] Furthermore, the specific process of attention injection in step 3 is as follows:

[0030] (1) Select an image from the style training set as the style reference image, and perform DDIM Inversion on the style reference image and the content image obtained in step 1 respectively:

[0031] i) Use the DDIM method to gradually add noise to the content image and style image. Get the content image noise Style Image Noise Set the denoising and denoising processes to T steps;

[0032] ii) During the DDIM inversion process, the self-attention features of the image are collected at each time step. For the content graph, Q C , for the style map, save K S ,V S .

[0033] (2) Use the AdaIN normalization method to mix and obtain the initial latent features. and style image noise The normalization formula is:

[0034]

[0035] in is the initial noise obtained by mixing. μ is the mean, σ is the standard deviation; initial potential noise Reserved The channel mean and standard deviation are Alignment. The mean and standard deviation can roughly describe the tone of the image, that is, for the content image, de-stylization is performed first, and then stylized to the style of the reference image.

[0036] (3) Reverse sampling to obtain the stylized image;

[0037] i) In the T-step of the reverse process, based on the self-attention information saved in step 1, Gradually injected into the self-attention layer of the unet network.

[0038] ii) Control the style confidence. During the inverse sampling process, the styled and the content graph saved in the first step Mixed together and used throughout the iterative process to maintain the original content in the stylized image. The style confidence control formula is as follows:

[0039]

[0040] Among them, a higher γ preserves more content, while a lower γ enhances the effect of style transfer.

[0041] iii) After the style injection of each self-attention layer, attention scaling is performed. Experiments show that if the self-attention method in Stable Diffusion is directly applied, after the style injection in 1), The standard deviation of will be reduced, resulting in the obtained attention map being too smooth and distorted. Therefore, the attention temperature scaling parameter τ is used to correct the reduced standard deviation. The attention scaling formula is:

[0042]

[0043] The default τ=1.5.

[0044] After attention injection, a clear stylized image will be obtained.

[0045] The beneficial effect of this method is that the proposed image generation method based on style feature injection uses the ability of Dreambooth binding features to learn the spatial layout and preliminary style of the image based on the StableDiffusion image, and then performs attention injection to enhance the color features and detailed textures. Therefore, the proposed method has high generalization in various styles. For a single specific style, there is no need for tedious parameter adjustment, and it is highly sensitive to contextual features such as composition, texture, and color. In addition, the model has low data requirements, only 3-5 training images, and only one attention reference map is required during the reasoning process. No additional training is required, and it has strong robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Attached Figure 1 Schematic diagram of the algorithm framework of the style image generation method based on feature injection of the present invention.

[0047] Attached Figure 2 Schematic diagram of the style enhancement module DDIM inversion process.

[0048] Attached Figure 3 Schematic diagram of the reversion process of the style enhancement module.

[0049] Attached Figure 4 Content image: DDIM noise addition diagram.

[0050] Attached Figure 5 Style reference picture DDIM noise diagram.

[0051] Attached Figure 6 Schematic diagram of AdaIN initial noise reversion. DETAILED DESCRIPTION

[0052] The following is a detailed description of an image generation method based on style feature injection of the present invention, and further specific description of the method. It is necessary to point out that the following embodiments are only used to further illustrate the method and cannot be understood as limiting the protection scope of the method. Personnel skilled in the art in the field may make some non-essential improvements and adjustments to the method based on the above content of the method, which still fall within the protection scope of the present invention.

[0053] The present invention proposes an image generation method based on style injection, such as Figure 1 As shown, the method includes two processes: network training and reasoning.

[0054] The network training process includes the following steps:

[0055] Step 1: Use a large language model to specify prompt words for a specific style. For example, the prompt word for oil painting style is "style_00, oil painting". Bind the prompt word to the specific style;

[0056] Step 2: Data enhancement: For a data set of a specified style (3-5 images, 512*512 pixels), use random cropping, random blurring, random flipping and other methods to expand the training set and perform data enhancement;

[0057] Step 3: Input the enhanced training data into the Dreambooth+Lora network for training;

[0058] Step 4: Save the lora weight for reasoning;

[0059] Furthermore, the pre-trained model of Dreambooth+Lora in step three is StableDiffusion2.1. In the forward reasoning process, the SD model removes the predicted noise residual from the original noise matrix by repeatedly calling U-Net, obtains the Latent Features of the image after gradual denoising, and then reconstructs the Latent Features into pixel-level images through the Decoder structure of VAE. VAE is based on the Encoder-Decoder architecture, in which the Encoder architecture converts the image into low-dimensional Latent features and serves as the input of Unet. The Decoder structure can reconstruct the low-dimensional Latent features and restore them to pixel-level images.

[0060] There are three basic components in the SD VAE model: GSC component, Downsample component, and Upsample component. At the same time, the SD VAE Encoder part contains three DownBlock modules, one ResNetBlock module, and one MidBlock module, which compresses the input image into the Latent space and converts it into a Gaussian Distribution. The VAE Decoder part is just the opposite. It inputs the Latent space features and reconstructs them into pixel-level images as output. It contains three UpBlock modules, one ResNetBlock module, and one MidBlock module.

[0061] The text encoder of Stablediffusion2.1 is mainly the CLIP model. CLIP mainly includes two models: TextEncoder and Image Encoder. The Text Encoder is used to extract the features of the text, and the text transformer model commonly used in NLP can be used as the Text Encoder; while the Image Encoder is mainly used to extract the features of the image, and the CNN / vision transformer model (ResNet and ViT) is used as the Image Encoder. The task of the CLIP model is mainly to extract the embedding vectors of the label text and the picture through the Text Encoder and Image Encoder respectively, and then use the cosine similarity to compare the similarity of the two embedding vectors to determine whether the randomly extracted label text and picture match, and perform gradient backpropagation to continuously optimize the training.

[0062] The Unet in Stable Diffusion adds ResNetBlock (including TimeEmbedding) module, Spatial Transformer (SelfAttention+CrossAttention+FeedForward) module and CrossAttnDownBlock, CrossAttnUpBlock and CrossAttnMidBlock modules on the basis of traditional Unet. In Stable Diffusion, the U-Net model is a key core part that can predict noise residuals and reconstruct the input feature matrix in combination with Samplingmethod (scheduling algorithm: PNDM, DDIM, K-LMS, etc.), gradually converting it from random Gaussian noise into the Latent Feature of the image.

[0063] Furthermore, the specific training process of Dreambooth+Lora in step 3 is as follows:

[0064] (1) Using a pre-trained text-to-image diffusion model The model is given an initial noisy image. The image is generated with the conditional vector c = Γ(P) This model is trained with squared error loss to detect images or latent codes z with different levels of noise. t :=α t x+σ t ∈ is used for denoising, and the loss function is as follows:

[0065]

[0066] where x is the real image, c is the conditioning vector (e.g., obtained from a textual prompt), and α t ,σ t ,w t are terms that control the noise process and sample quality. These terms are the diffusion process time function.

[0067] (2) Assemble the Lora weights for the Dreambooth model. By encoding task-specific parameter increments, the size of the required parameters is greatly reduced. For the pre-trained model weights Φ 0 , the fine-tuning parameter is |ΔΦ|. Usually, Φ 0 Equal to the dimension of |ΔΦ|. Lora uses a parameter set Θ that is much smaller than |ΔΦ| to replace |ΔΦ|. The specific formula is as follows:

[0068]

[0069] In the image generation method based on style feature injection, set the weight initialization method of Lora configuration to Gaussian, specify target_module to ["to_k", "to_q", "to_v", "to_out.0", "add_k_proj", "add_v_proj"], and add lora_dropout = 0.01.

[0070] (3) Set the optimizer to AdamW optimizer. Learning rate lr = 5e-4, adaim_epsilon = 1e-8.

[0071] (4) Specify parameters such as training epochs and warm up steps to train lora weights.

[0072] The network reasoning process is specifically described as follows:

[0073] Step 1: Construct the input of the network. Bind the subject prompt to be generated with the specific prompt words used in training. Taking boat and oil painting style as an example, the input of the inference stage is "boat in style_00, oil painting";

[0074] Step 2: Use the lora weights obtained in the training step to generate preliminary results with spatial layout and preliminary style features;

[0075] Step 3: Perform attention injection on the preliminary results obtained in step 2 to enhance the color texture of the style. Attention injection will be applied to the 7th to 12th Decoder layers in Stable Diffusion. After repeated iterations, a clear stylized image will be obtained.

[0076] Furthermore, the specific process of attention injection in step 3 is as follows:

[0077] (1) Select an image from the style training set as the style reference image, and perform DDIM Inversion on the style reference image and the content image obtained in step 1 respectively:

[0078] i) Use the DDIM method to gradually add noise to the content image and style image. Get the content image noise Style Image Noise Set the denoising and denoising processes to T steps;

[0079] ii) During the DDIM inversion process, the self-attention features of the image are collected at each time step. For the content graph, Q C, for the style map, save K S ,V S .

[0080] (2) Use the AdaIN normalization method to mix and obtain the initial latent features. and style image noise The normalization formula is:

[0081]

[0082] in is the initial noise obtained by mixing. μ is the mean, σ is the standard deviation; initial potential noise Reserved The channel mean and standard deviation are Alignment. The mean and standard deviation can roughly describe the tone of the image, that is, for the content image, de-stylization is performed first, and then stylized to the style of the reference image.

[0083] (3) Reverse sampling to obtain the stylized image;

[0084] i) In the T-step of the reverse process, based on the self-attention information saved in step 1, Gradually injected into the self-attention layer of the unet network.

[0085] ii) Control the style confidence. During the inverse sampling process, the styled and the content graph saved in the first step Mixed together and used throughout the iterative process to maintain the original content in the stylized image. The style confidence control formula is as follows:

[0086]

[0087] Among them, a higher γ preserves more content, while a lower γ enhances the effect of style transfer.

[0088] iii) After the style injection of each self-attention layer, attention scaling is performed. Experiments show that if the self-attention method in Stable Diffusion is directly applied, after the style injection in 1), The standard deviation of will be reduced, resulting in the obtained attention map being too smooth and distorted. Therefore, the attention temperature scaling parameter τ is used to correct the reduced standard deviation. The attention scaling formula is:

[0089]

[0090] The default τ=1.5.

[0091] After attention injection, a clear stylized image will be obtained.

[0092] In the example of the present invention, the training set used for network training is the public data set of the style image generation track of the 4th Jitu Artificial Intelligence Challenge, with a total of 28 different styles. Each style has three training images. During inference, one of the three training images is randomly selected as a style reference image. During the training process, it is first necessary to specify hyperparameters such as epoch, warm up steps, and the number of data enhancements. Specifically described as follows: epoch generally selects a constant between 300-1500 that is divisible by 100 as needed, the warm up steps are generally set to half of the epoch value, and the number of data enhancements is generally set to 1 / 10 of the epoch. Generally, for most styles, set epoch=1000, warm up=500, and items=100. Under normal circumstances, if the style features are not obvious, the number of epoch steps can be appropriately increased. If the style features are obvious and the generated content pays more attention to text consistency, it is recommended to reduce the number of epochs to avoid overfitting. The weight initialization method of Lora configuration is gaussian, target_module is specified as ["to_k", "to_q", "to_v", "to_out.0", "add_k_proj", "add_v_proj"], and lora_dropout = 0.01 is added. The optimizer is set to AdamW optimizer. Learning rate lr = 5e-4, adaim_epsilon = 1e-8. In the inference stage, the number of steps for generating images is set to step = 25. In the style enhancement module, the number of steps for DDIM inversion and reversion is set to step = 50. That is, the number of steps T for denoising and denoising is 50. In addition, the hyperparameter tao for attention temperature scaling is set to 1.5. The default mixing parameter gamma of the content map and style map Query is 0.7. Experiments show that when gamma is 0.7, it can better restore the style characteristics of the style reference map to the maximum extent while maintaining the content of the content map, while reducing the degradation of image quality. The style injection will be applied to the 7th to 12th layers of the Self-attention layer in unet. The training and reasoning environment of the proposed method is the Pytorch framework with Nvidia RTX 4090 GPU and Intel Core(TM) i7-7820X CPU@3.60Hz.

[0093] Table 1 Different style image generation methods CLIP-R, FID, DinoV2

[0094]

[0095] The present invention aims to verify the effectiveness and generalization ability of the proposed method, and to demonstrate its performance in practical applications by testing it on 28 different style images. To achieve this goal, we used the public dataset of the style image generation track of the 4th JiTu AI Challenge, which contains a variety of style images. During the test, a variety of baseline methods were used, including Textual inversion and Style align. Textual inversion is a style transfer method based on text inversion, which combines text information with image style to achieve personalized style transfer. However, this method has certain limitations in practical applications and is difficult to meet the diverse style transfer needs. Style align is a style feature alignment method that aligns style features in images to achieve style transfer. However, due to its high requirements for reference images, this method has poor performance in style transfer tasks and is difficult to meet the needs of practical applications. Compared with traditional style transfer methods, the proposed method has stronger generalization ability and higher image generation quality. According to the results in Table 1, the proposed method has stronger generalization in different styles, higher image generation quality, and higher text consistency. This shows that our method has better performance in practical applications.

[0096] In addition, we also conducted a visualization test on the proposed method. From the visualization results, it can be seen that the proposed image generation method based on style injection is stable and effectively operated. This further verifies the feasibility and effectiveness of our method in practical applications. The experimental results show that the proposed method has stronger generalization in different styles, higher image generation quality, and higher text consistency. At the same time, the visualization test results also further verify the feasibility and effectiveness of our method in practical applications. Therefore, we can conclude that the proposed image generation method based on style injection is a stable and effective image generation method with broad application prospects.

Claims

1. The image generation method based on style feature injection is characterized by: It includes two processes: network training and reasoning: The network training process is specifically described as follows: Step 1: Use a large language model to specify prompt words for a specific style. For example, the prompt word for oil painting style is "style_00,oilpainting", and bind the prompt word to the specific style. Step 2: Data enhancement: For a data set of a specified style (3-5 images, 512*512 pixels), random cropping, random blurring, random flipping and other methods are used to expand the training set and perform data enhancement. Step 3: Input the enhanced training data into the Dreambooth+Lora network for training; Step 4: Save lora weights for reasoning; The network reasoning process is specifically described as follows: Step 1: Construct the input of the network and bind the subject prompt to be generated with the specific prompt words used in training. Taking boat and oil painting style as an example, the input of the inference stage is "boat in style_00, oil painting"; Step 2: Use the lora weights obtained in the training step to generate preliminary results with spatial layout and preliminary style features; Step 3: Perform attention injection on the preliminary results obtained in step 2 to enhance the color texture of the style. Attention injection will be applied to the 7th to 12th Decoder layers in Stable Diffusion. After repeated iterations, a clear stylized image will be obtained.

2. The image generation method based on style feature injection according to claim 1, characterized in that: StableDiffusion2.1 consists of VAE, text-encoder, and unet; (1) VAE VAE is based on the Encoder-Decoder architecture, where the Encoder architecture converts images into low-dimensional Latent features and serves as the input of Unet; the Decoder structure can reconstruct low-dimensional Latent features and restore them to pixel-level images; There are three basic components in the SD VAE model: i) GSC components: GroupNorm+Swish+Conv ii) Downsample component: Padding+Conv iii) Upsample component: Interpolate+Conv At the same time, the SD VAE Encoder part contains three DownBlock modules, one ResNetBlock module and one MidBlock module, which compresses the input image into Latent space and converts it into Gaussian Distribution; The VAE Decoder part is just the opposite. It inputs the Latent spatial features and reconstructs them into pixel-level images as output. It contains three UpBlock modules, one ResNetBlock module and one MidBlock module. (2)Text encoder The text encoder of Stablediffusion2.1 is mainly the CLIP model; CLIP mainly includes two models: Text Encoder and Image Encoder. The Text Encoder is used to extract text features, and the text transformer model commonly used in NLP can be used as the Text Encoder; while the Image Encoder is mainly used to extract image features, and the CNN / vision transformer model (ResNet and ViT) is used as the Image Encoder; the task of the CLIP model is mainly to extract embedding vectors from the label text and image through the Text Encoder and Image Encoder respectively, and then use cosine similarity to compare the similarity of the two embedding vectors to determine whether the randomly extracted label text and image match, and perform gradient backpropagation to continuously optimize training; (3)Unet The Unet in Stable Diffusion adds ResNetBlock (including TimeEmbedding) module, Spatial Transformer (SelfAttention+CrossAttention+FeedForward) module and CrossAttnDownBlock, CrossAttnUpBlock and CrossAttnMidBlock modules on the basis of traditional Unet; In Stable Diffusion, the U-Net model is a key core part, which can predict the noise residual and reconstruct the input feature matrix in combination with the Sampling method (scheduling algorithm: PNDM, DDIM, K-LMS, etc.), gradually converting it from random Gaussian noise into the Latent Feature of the image; Specifically, during the forward reasoning process, the SD model repeatedly calls U-Net to remove the predicted noise residual from the original noise matrix, obtains the gradually denoised image Latent Features, and then reconstructs the Latent Features into a pixel-level image through the Decoder structure of VAE.

3. The image generation method based on style feature injection according to claim 1, characterized in that: Based on the StableDiffusion text map, Dreambooth is used to bind style features, and Lora is introduced. The low-rank matrix is ​​used to adjust the original weight matrix to achieve efficient customization of the model. The process can be described as: (1) Using a pre-trained text-to-image diffusion model The model is given an initial noisy image. The image is generated with the conditional vector c = Γ(P) This model is trained with squared error loss to detect images or latent codes z with different levels of noise. t :=α t x+σ t ∈ is used for denoising, and the loss function is as follows: where x is the real image, c is the conditioning vector (e.g., obtained from a textual prompt), and α t ,σ t ,w t are terms that control the noise process and sample quality. These terms are the diffusion process time Function of (2) Assemble the Lora weights for the Dreambooth model; by encoding task-specific parameter increments, the size of the required parameters is greatly reduced; for the pre-trained model weight Φ0, the fine-tuning parameter is |ΔΦ|; usually, the dimensions of Φ0 and |ΔΦ| are equal; Lora uses a parameter set Θ that is much smaller than |ΔΦ| to replace |ΔΦ|. The specific formula is as follows: In the image generation method based on style feature injection, set the weight initialization method of Lora configuration to Gaussian, specify target_module to ["to_k", "to_q", "to_v", "to_out.0", "add_k_proj", "add_v_proj"], and add lora_dropout = 0.01; (3) Set the optimizer to AdamW optimizer; learning rate lr = 5e-4, adaim_epsilon = 1e-8; (4) Specify parameters such as training epochs and warm up steps to train lora weights.

4. The image generation method based on style feature injection according to claim 1, wherein the steps of style injection are as follows: (1) Select an image from the style training set as the style reference image, and perform DDIM Inversion on the style reference image and the content image obtained in step 1 respectively: i) Use the DDIM method to gradually add noise to the content image and style image respectively; obtain the content image noise Style Image Noise Set the denoising and denoising processes to T steps; ii) During the DDIM inversion process, the self-attention features of the image are collected at each time step; for the content graph, Q is saved C , for the style map, save K S ,V S ; (2) Use the AdaIN normalization method to mix and obtain the initial latent features; For content image noise and style image noise The normalization formula is: in is the initial noise obtained by mixing; μ is the mean, σ is the standard deviation; initial potential noise Reserved The channel mean and standard deviation are Alignment; the mean and standard deviation can roughly describe the tone of the image, that is, for the content image, first perform de-stylization and then stylize it to the style of the reference image; (3) Reverse sampling to obtain the stylized image; i) In the T-step of the reverse process, based on the self-attention information saved in step 1, Gradually inject into the self-attention layer of the unet network; ii), control style confidence; During the inverse sampling process, the stylized query and the query of the content graph saved in the first step Mixed together and used in the entire iterative process to maintain the original content in the stylized image; the style confidence control formula is as follows: Among them, a higher γ retains more content, while a lower γ enhances the effect of style transfer; iii) After the style injection of each self-attention layer, attention scaling is performed; experiments show that if the self-attention method in Stable Diffusion is directly applied, after the style injection in 1), The standard deviation of will be reduced, resulting in the obtained attention map being too smooth and distorted; therefore, the attention temperature scaling parameter τ is used to correct the reduced standard deviation; the attention scaling formula is: The default value is τ=1.5; After attention injection, a clear stylized image will be obtained.

Images