Image customization method based on diffusion model

Abstract

The invention discloses a customized image method based on a diffusion model, which introduces an image encoder module and injects image prior information, not only can improve the consistency of an object and a reference object in a generated image, but also needs to optimize model parameters, is low in calculation cost, and reduces time and example cost. A lightweight identity consistency network is adopted to carry out feature extraction on a reference object, and the parameter quantity required to be optimized by a model is reduced. And the interference of background information is reduced while the consistency is ensured by using the loss of mask attenuation. In order to improve the editability of a text to a reference object, text prior and a text protection mechanism are introduced, a text without an identification vector is used as a learning target, the difference between other vectors except the identification vector is reduced, and text protection is carried out in a vector orthogonal decomposition mode. The method can be widely applied to actual scenes, the identity consistency of the reference object is high, and meanwhile, the editability of the attribute of the reference object is high.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and relates to a customized image method based on a diffusion model. Background Art

[0002] The diffusion model is a new model that has emerged in the field of computer vision in recent years. Its progress has promoted significant progress in image generation technology, thereby promoting the development of image editing technology. The diffusion model can be divided into a forward diffusion process and a reverse denoising process. The forward diffusion process starts from a real picture and gradually adds noise until the data is completely converted into noise. This process can be regarded as a Markov chain, that is, each state depends only on the previous state. The reverse denoising process is to train a neural network to minimize the gap between the network's predicted noise and the actual value of the noise added in the forward process, and then gradually denoise until the original data is restored from the noise.

[0003] Customized images have attracted great interest in applying diffusion models to image generation and image editing. This application aims to generate realistic images of reference objects under various circumstances given a reference object. Existing customized image methods can be divided into three types according to the training method: no fine-tuning, fine-tuning, and training. For the method without fine-tuning, Mingdeng Ca et al.'s MasaCtrl (Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing) and Shilin Lu et al.'s TF-ICON (Diffusion-Based Training-Free Cross-Domain Image Composition) gradually add noise to the image containing the reference object to obtain an intermediate state. In the process of gradually denoising and generating a new image, the features of the reference object in the intermediate state are inserted into it, thereby achieving the purpose of customizing the reference object. Although this method can be plug-and-play, greatly reducing time and computing power costs, it also faces the following disadvantages: the consistency between the generated image and the reference object is low, and the consistency between the generated image and the reference object is easily affected by other factors. The fine-tuning method usually uses Lora (Low-Rank Adaptation of Large Language Models) or Multilayer Perceptron MLP (Multilayer Perceptron, MLP). The work DreamBooth (DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation) by Nataniel Ruiz et al. uses Lora, and the work StableIdentity (StableIdentity: Inserting Anybody into Anywhere at First Sight) by Qinghe Wang et al. uses MLP to train a small number of parameters while ensuring the effect, thereby achieving a balance between efficiency and performance.The category of training refers to the optimization of the network parameters in the entire diffusion model. The work of Xi Chen et al., AnyDoor (AnyDoor: Zero-shot Object-level Image Customization) and Binxin Yang et al., Paint by Example (Paint by Example: Exemplar-based Image Editing with Diffusion Models), uses an image encoder to encode features and injects the features into the Transformer layer of the diffusion model. The advantage of using this method is that after training, the generated image is highly consistent with the reference image for any reference object, and the stability is high. However, the disadvantages are also obvious. The training time and computing power resource cost are high. At the same time, it also faces the collection and processing of large data sets, and the labor cost is greatly increased.

[0004] The above-mentioned image customization methods mainly focus on regenerating the entire image containing the reference object, which leads to the inability to control the background of the regenerated image and the specific position of the reference object in the image, limiting its application in photography and photo post-processing. The emergence of the image in-painting method can solve the above problems. It takes the user-specified position as input to achieve precise editing at the specified position of the image while keeping the background unchanged. Therefore, in order to realize the function of image editing at the specified position, the image in-painting method and the image customization method are combined to achieve a balance between performance and time, and expand the application scope of the diffusion model in image editing. Summary of the invention

[0005] The present invention aims to provide a customized image method based on a diffusion model, which solves the problem that the position and background of the reference object cannot be controlled in the existing image editing field and customized image methods, and optimizes some parameters in the network model by fine-tuning methods, saving time and computing power costs while taking into account the performance of the model. The method includes diffusion models, multimodal fusion, image segmentation and other technologies, and generates realistic images with foreground and background fusion under the conditions of given reference objects, background images and specified positions.

[0006] The technical solution of the present invention:

[0007] A customized image method based on a diffusion model consists of three parts, including an image encoder, a text encoder, and a U-shaped network. The specific steps are as follows:

[0008] Step 1: Given several images of reference objects, use an image segmenter to separate their foreground images;

[0009] Step 2: After the size of the foreground image extracted in step 1 is transformed to 224×224, it is sent to the image encoder, that is, the image part of the CLIP model, to extract the image features of the foreground image, and the global vector is selected as the feature vector; the feature vector is then sent to a multi-layer perceptron with 3 convolutional layers, and the feature vector is mapped into two word vectors with a dimension of 768 as the identification vectors s1 and s2 of the reference object, which are used to learn more dimensional information in the reference object;

[0010] Step 3: Send the text to the text encoder to obtain the word vector, concatenate it with the identification vector of the reference object obtained in step 2 as the text vector, and then send it to the text Transformer model to obtain the text feature;

[0011] The specific operation of obtaining word vectors is as follows: First, a full English text is passed through a word segmenter to convert English words into corresponding IDs, that is, one or several numbers, and then the corresponding IDs are converted into word vectors with a dimension of 768 using a dictionary;

[0012] In the text Transformer model, the masked attention mechanism is used to perform global attention on the text vector;

[0013] Step 4: The specific location of the edit in the user-specified image is given in the form of a bounding box, that is, four point coordinates, which are processed into a mask and spliced ​​together with the noise image and the background image; for the spliced ​​image, the variational autoencoder is used to map it to the latent space, and the 8-fold downsampled potential features are extracted. The attention operation is performed in the U-shaped network with the text features obtained in step 2 to achieve the fusion of text modality and image modality; during the training process, the noise image is obtained by data augmentation of any image containing the reference object and then adding Gaussian noise; the background image is obtained by data augmentation of any image containing the reference object and multiplying it by 1 minus the mask; during the inference process, the noise image is a tensor that completely obeys the Gaussian distribution, the background image is specified by the user, and the mask is converted from the position in the form of a bounding box given by the user, and it also supports the form of refined masks input by the user;

[0014] The U-shaped network is part of the diffusion model and consists of 16 Transformer layers and convolutional layers. The Transformer layer includes a self-attention layer, a cross-attention layer, and a feedforward layer. In the self-attention layer, the latent features, i.e., image features, are used as query vectors, key vectors, and value vectors. In the cross-attention layer, the latent features, i.e., image features, are used as query vectors, and text features are used as key vectors and value vectors. When performing the cross-attention operation, the information of the two modalities of image and text is fused, and while injecting the information of the reference object, the editability of the text to the generated image is ensured, and the background of the generated image is made consistent with the given background image. In the forward process of the U-shaped network, the latent features are gradually downsampled by 8 times, and then upsampled by 8 times, and finally a 512×512 image is obtained through a variational self-decoder.

[0015] Step 5: During the training process, the loss function of mask boundary attenuation is adopted; the training process uses two identification vectors to represent the reference object, eliminating the influence of background information on the reference object ID, and only needs to calculate the loss of the noise inside the mask to reduce the gap between the predicted noise and the real noise; in the forward process of the diffusion model, the total downsampling multiple reaches 64 times, which results in a maximum offset of 64 pixels for a pixel in the input image, which seriously affects the accuracy of the reference object boundary; in view of the above situation, in order to ensure the accuracy of the reference object ID and eliminate the interference of background information, first use a 3×3 convolution kernel to perform three consecutive dilation convolutions on the mask to expand the area of ​​the mask; then perform a Gaussian blur operation, and select a 3×3 Gaussian kernel to perform a convolution operation on the mask. The closer to the edge of the mask, the more 0 elements in the convolution operation, the smaller the mean, that is, the smaller the weight, to achieve the purpose of attenuating the boundary weight of the binary mask. Finally, the weight of the internal loss function of the mask is 1, and the weight of the mask boundary loss function gradually decays from 1 to 0;

[0016] During the reasoning process, text protection and text prior mechanisms are used to ensure the editability of the text; considering the global attention in the text Transformer model, the identification vector will affect the normal expression of the text, so a text prior method is used to ensure the meaning; the text without the identification vector is used as the learning target to reduce the gap between the text with the identification vector; the vectors of adjectives and verbs in the text without the identification vector are recorded, and the positions of the same adjectives and verbs in the text with the identification vector are found, and the vectors at the position are replaced with the recorded adjectives and verbs in the text without the identification vector. The vectors of adjectives and verbs are used to express the required attributes normally. At the same time, considering that the attributes of the reference object are included in the identification vector, when these inherent attributes are modified using text, they will conflict with the components in the identification vector. Therefore, the text protection is performed by vector orthogonal decomposition. First, the attributes that need to be edited are found, and then the identification vector is decomposed. The word vector corresponding to the attribute that needs to be edited is used as the basis vector, and the identification vector is decomposed into components perpendicular to the basis vector and components parallel to the basis vector. The parallel components are considered to be conflicting components and are removed. The vertical components are considered to be components that do not affect the attributes that need to be edited and are retained. Through such operations, both words can be expressed normally, ensuring the ID while ensuring editability.

[0017] The beneficial effects of the present invention are as follows: it can be more widely applied to actual scenes, the identity consistency of the reference object is high, and the attributes of the reference object, such as shape, color, material, etc., are highly editable. The present invention can solve the problem that in the existing customized image method, the reference object cannot be edited using text as a condition at the user-specified position, greatly expanding the application scope of the diffusion model in the field of image editing, and has certain practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a system block diagram of the customized image method based on the diffusion model.

[0019] Figure 2 This is a network structure diagram of the customized image method based on the diffusion model.

[0020] Figure 3 It is the correspondence diagram between text protection and text prior mechanism during reasoning. DETAILED DESCRIPTION

[0021] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0022] The system block diagram of the customized image method based on the diffusion model of the present invention is as follows: Figure 1 The specific contents are as follows:

[0023] (1) Data preparation:

[0024] Since the image of the reference object contains background interference, in order to avoid the influence of background information, we first use the existing image segmentation model to obtain the image of the reference object under the white background and the corresponding mask.

[0025] (2) Image preprocessing and feature extraction:

[0026] During the training process, data enhancement is performed on the pictures of the reference objects. Specifically, some data enhancements such as rotation, border filling, and enlargement of the mask area are performed on the pictures. For the images input to the image encoder, the image encoder is used to encode the images and extract semantic features. First, the image is scaled to 224×224 and input into the image encoder; then, features are extracted from the image, and the global feature vector is selected as the final image feature vector. The feature vector is then passed to a multi-layer perceptron, which contains 3 convolutional layers and a normalization layer to further mine the multi-dimensional information of the reference object. Finally, the feature vector is converted into an identification word vector exclusive to the reference object through the reshape operation;

[0027] (3) Text feature extraction:

[0028] After obtaining the identification vector of the reference object, the text will be input into the text encoder to obtain the word vector. Subsequently, the identification vector of the reference object is concatenated with the word vector to generate a fused text vector, which is then passed to the text Transformer model to finally generate text features.

[0029] In the text transformer, a masked attention mechanism is used to capture global contextual relationships. To ensure temporal consistency, when processing words earlier in the sequence, words later in the sequence will be masked to prevent future information from affecting the attention of the current word, thereby ensuring that the temporal rules of text features are correctly preserved.

[0030] (4) Image-text feature fusion:

[0031] The bounding box position provided by the user is converted into a mask form. First, a tensor with all 0s is initialized, and the pixel in the middle of the bounding box provided by the user is assigned a value of 255 to obtain a mask, which is then merged with the noise image and the background image. Then, a variational autoencoder is used to map these merged data to the latent space, and an 8x downsampled latent feature is extracted from it. This latent feature, i.e., the image feature, is input into a U-shaped network together with the text feature obtained above. The network fuses the information of the text and image modalities through a cross-attention mechanism.

[0032] In the training phase, the noise image is obtained by adding Gaussian noise to a picture of the reference object after data augmentation; the background image is obtained by data augmentation of any image containing the reference object, and multiplying it by 1 minus the mask. In the inference phase, the noise image is noise that fully conforms to the Gaussian distribution, while the background image is directly specified by the user. The mask is converted from the bounding box coordinates provided by the user, and users can also specify more detailed mask methods.

[0033] In the U-type network, the latent features are used as the query vector Query, while the key vector Key and the value vector Value are provided by the text features. The latent features interact with the text features to perform information fusion. The latent features are transformed by the convolution layer φ to obtain the Query, and the text features are transformed by two different convolution layers ψ and θ to obtain the Key and Value. The key and value feature vectors are dot-multiplied and softmax calculated to obtain a d×d dimensional feature map, which is then dot-multiplied with the feature vector output by the query to complete feature modulation, and finally added to the latent features to output the latent space features after information fusion. This process not only introduces image prior information and injects information about the reference object, but also fuses it with the text to ensure the editability of the generated image, while making the generated background image consistent with the given background image.

[0034] (5) Post-processing:

[0035] like Figure 3 As shown in the figure, in order to ensure the editability of the text, text protection and text prior mechanisms are used in the inference stage. Two texts are input respectively, one with identification vectors s1 and s2 and the other without identification vectors, and they are sent to the text transformer to record the text features corresponding to the attribute words to be edited. The vectors of adjectives and verbs in the text with identification vectors s1 and s2 are replaced with the vectors corresponding to the adjectives and verbs in the text without identification vectors. Then, the vector decomposition operation is performed, and the attribute word vector to be edited is used as the basis vector to establish a vector space constructed with the basis vector. In this space, s1 and s2 are decomposed into vertical components and parallel components in the direction of the basis vector. The parallel component represents the part that conflicts with the attribute to be edited, and we remove it and only keep the vertical component. Finally, after the variational self-decoder, the 64×64 potential features output by the U-type network are upsampled to obtain a 512×512 image.

[0036] The network training process uses the Dreambooth dataset, where each reference object has 3-4 images. The inpainting network PowerPaint trained on the Stable Diffusion v1.5 model is used as the basic framework of the entire model. The image encoder used is Open-CLIP ViT / H-14, and SAM is used as the segmenter to obtain accurate masks. For the input single image, we use boundary padding, random resizing and random padding as data augmentation. The learning rate and batch size are set to 0.00005 and 1. During training, the image resolution is processed to 512×512, and the parameters of the network optimization are only those in the multi-layer perceptron, which contains 3 convolutional layers and regularization layers. It takes about 1000 steps to memorize the reference objects. In the inference stage, we use two markers to represent the reference object. By default, the size without classifier guidance is set to 7.5. All runs are performed on a single 4090 GPU.

[0037] During the experiment, visualization results are mainly used to evaluate network performance. The experimental results are as follows:

[0038] Using the baseline, without adding MLP fine-tuning, text prior, and text protection mechanism, the objects in the edited images have very low consistency with the reference objects. They belong to the same category semantically, but the details such as texture and color are completely different from those of the reference objects.

[0039] With MLP fine-tuning and without text priors and text protection mechanisms, the objects in the edited images are highly consistent with the reference objects, but are in an overfitting state, that is, for texts with certain editing actions, colors, and shapes, the generated images cannot follow the instructions of the text;

[0040] By adding MLP fine-tuning, text prior and text protection mechanisms, the objects in the edited image are highly consistent with the reference objects. At the same time, for text with given editing actions, colors and shapes, the generated image completely follows the instructions of the text, achieving identity consistency while also retaining the editability of the text.

Claims

1. A customized image method based on a diffusion model, characterized in that: The customized image method consists of three parts, including image encoder, text encoder and U-shaped network. The specific steps are as follows: Step 1: Given several images of reference objects, use an image segmenter to separate their foreground images; Step 2: After the size of the foreground image extracted in step 1 is transformed to 224×224, it is sent to the image encoder, that is, the image part of the CLIP model, to extract the image features of the foreground image, and the global vector is selected as the feature vector; the feature vector is then sent to a multi-layer perceptron with 3 convolutional layers, and the feature vector is mapped into two word vectors with a dimension of 768 as the identification vectors s1 and s2 of the reference object, which are used to learn more dimensional information in the reference object; Step 3: Send the text to the text encoder to obtain the word vector, concatenate it with the identification vector of the reference object obtained in step 2 as the text vector, and then send it to the text Transformer model to obtain the text feature; The specific operation of obtaining word vectors is as follows: First, a full English text is passed through a word segmenter to convert English words into corresponding IDs, that is, one or several numbers, and then the corresponding IDs are converted into word vectors with a dimension of 768 using a dictionary; In the text Transformer model, the masked attention mechanism is used to perform global attention on the text vector; Step 4: The specific location of the edit in the user-specified image is given in the form of a bounding box, that is, four point coordinates, which are processed into a mask and spliced ​​together with the noise image and the background image; for the spliced ​​image, the variational autoencoder is used to map it to the latent space, and the 8-fold downsampled potential features are extracted. The attention operation is performed in the U-shaped network with the text features obtained in step 2 to achieve the fusion of text modality and image modality; during the training process, the noise image is obtained by data augmentation of any image containing the reference object and then adding Gaussian noise; the background image is obtained by data augmentation of any image containing the reference object and multiplying it by 1 minus the mask; during the inference process, the noise image is a tensor that completely obeys the Gaussian distribution, the background image is specified by the user, and the mask is converted from the position in the form of a bounding box given by the user, and it also supports the form of refined masks input by the user; The U-shaped network is part of the diffusion model and consists of 16 Transformer layers and convolutional layers. The Transformer layer includes a self-attention layer, a cross-attention layer, and a feedforward layer. In the self-attention layer, the latent features, i.e., image features, are used as query vectors, key vectors, and value vectors. In the cross-attention layer, the latent features, i.e., image features, are used as query vectors, and text features are used as key vectors and value vectors. When performing the cross-attention operation, the information of the two modalities of image and text is fused, and while injecting the information of the reference object, the editability of the text to the generated image is ensured, and the background of the generated image is made consistent with the given background image. In the forward process of the U-shaped network, the latent features are gradually downsampled by 8 times, and then upsampled by 8 times, and finally a 512×512 image is obtained through a variational self-decoder. Step 5: During the training process, the loss function of mask boundary attenuation is adopted; the training process uses two identification vectors to represent the reference object, eliminating the influence of background information on the reference object ID, and only needs to calculate the loss of the noise inside the mask to reduce the gap between the predicted noise and the real noise; in the forward process of the diffusion model, the total downsampling multiple reaches 64 times, which results in a maximum offset of 64 pixels for a pixel in the input image, which seriously affects the accuracy of the reference object boundary; in view of the above situation, in order to ensure the accuracy of the reference object ID and eliminate the interference of background information, first use a 3×3 convolution kernel to perform three consecutive dilation convolutions on the mask to expand the area of ​​the mask; then perform a Gaussian blur operation, and select a 3×3 Gaussian kernel to perform a convolution operation on the mask. The closer to the edge of the mask, the more 0 elements in the convolution operation, the smaller the mean, that is, the smaller the weight, to achieve the purpose of attenuating the boundary weight of the binary mask. Finally, the weight of the internal loss function of the mask is 1, and the weight of the mask boundary loss function gradually decays from 1 to 0; During the reasoning process, text protection and text prior mechanisms are used to ensure the editability of the text. Considering the global attention in the text Transformer model, the identification vector will affect the normal expression of the text, so a text prior method is used to ensure the meaning. The text without the identification vector is used as the learning target to reduce the gap between the text after the identification vector is added and the text. The vectors of adjectives and verbs in the text without the identification vector are recorded, and the positions of the same adjectives and verbs are found in the text with the identification vector. The vectors at the positions are replaced with the recorded vectors of the adjectives and verbs in the text without the identification vector, so that the required attributes can be expressed normally. At the same time, considering that the identification vector contains the attributes of the reference object, when using the text to modify these inherent attributes, they will conflict with the components in the identification vector, so the vector orthogonal decomposition method is used to protect the text. First, find the attributes that need to be edited, and then decompose the identification vector. The word vector corresponding to the attribute that needs to be edited is used as the basis vector, and the identification vector is decomposed into components perpendicular to the basis vector and components parallel to the basis vector. The parallel components are considered to be conflicting components and are removed. The vertical component is considered to be a component that does not affect the attribute to be edited and is retained.

Images