Multi-modal scene fusion method based on large diffusion model

Abstract

The invention belongs to the technical field of image generation, and particularly discloses a multi-modal scene fusion method based on a large diffusion model. Firstly, singular value decomposition is carried out on a weight matrix of a diffusion model backbone network to update the weight matrix of the backbone network; then, inputting the text cue word into the diffusion model to generate a text condition image; performing VAE decoding and preprocessing on the text condition image to obtain a preprocessed text condition image; inputting the preprocessed text condition image into a pre-trained visual classifier to predict the category, and obtaining a prediction label; updating loss is calculated according to the predicted label and the real label, parameters of the diffusion model are updated, and fine adjustment of the diffusion model is achieved; and finally, generating a scene fusion image by using the fine-tuned diffusion model. Scene fusion is divided into two stages, in the first stage, a combined noise image is generated by sampling a background image and a foreground image, and in the second stage, a scene fusion image is generated through iteration of the combined noise image. According to the method, image generation is guided through the image, seamless fusion of the foreground image and the background image is achieved, meanwhile, image generation is guided through control information, and controllability of scene fusion is enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision image generation, and in particular relates to a multimodal scene fusion method based on a diffusion large model. Background Art

[0002] In recent years, large-scale text-to-image models have achieved remarkable success in text-driven image generation. However, the inherent ambiguity of natural language makes it difficult to convey precise and subtle visual details. Even using highly detailed text prompts cannot precisely control image generation. Therefore, enhancing the controllability of generated images has always been an important research topic in image generation.

[0003] Traditional methods achieve controllability of generated images through personalized learning, adding additional control conditions, etc. For personalized learning, the existing method is to use a small number of sample image data to fine-tune the pre-trained diffusion model so that it can learn the visual information corresponding to the new concept. However, since the features of the generated image only come from a few sample images, the generated image will face the problem of poor diversity, and there are also problems such as long training time. For adding additional control conditions, existing methods generally enhance the control effect by adding text conditions or image conditions. The ambiguity of the text description will cause the generated result to be inconsistent with the text description when using text-guided scene fusion. The image-guided scene fusion can make the generated image highly similar to the reference image due to image guidance, but the posture, direction, etc. of the reference image are still lacking in controllability. Therefore, how to optimize the personalized generation effect of the diffusion model so that it can efficiently learn the corresponding semantic information and achieve more controllable scene fusion is a technical problem that needs to be solved. Summary of the invention

[0004] In view of the deficiencies in the prior art, the technical problem that the present invention intends to solve is to propose a multimodal scene fusion method based on a diffusion large model.

[0005] The present invention solves the technical problem by adopting the following technical solution: A multimodal scene fusion method based on a diffusion large model, characterized in that the method comprises the following steps: Step 1: Perform singular value decomposition on the weight matrix of the backbone network of the diffusion model to update the weight matrix of the backbone network; Step 2: Fine-tune the diffusion model and use the pre-trained visual classifier for supervision; Input the text prompt words into the diffusion model to generate a text-conditioned image; the text-conditioned image is decoded and preprocessed by VAE to obtain a preprocessed text-conditioned image; the preprocessed text-conditioned image is input into the pre-trained visual classifier to predict the category and obtain the predicted label; the update loss is calculated based on the predicted label and the true label, and the parameters of the diffusion model are updated to achieve fine-tuning of the diffusion model; Step 3: Generate scene fusion image using the fine-tuned diffusion model; Input the background image, the target position binary image, the foreground image, the foreground object binary image and the text prompt word into the fine-tuned diffusion model to generate a background noise image and a foreground noise image; merge the background noise image with the foreground noise image to obtain a merged noise image; Input the conditional control image into the pre-trained control network to obtain the latent vector of the conditional control image; for the t-th iteration, input the latent vector of the conditional control image, the text conditional vector, and the latent vector of the generated image of the t-1-th iteration into the pre-trained control network to obtain the conditional control vector of the t-th iteration; The generated image latent vector and text conditional vector of the t-1th iteration are input into the backbone network of the diffusion model, and the generated feature vector is added to the conditional control vector of the tth iteration to obtain the generated image latent vector of the tth iteration; after multiple iterations, the final generated image latent vector is obtained; the final generated image latent vector is decoded by VAE to obtain a scene fusion image.

[0006] Furthermore, in step 3, the background part of the generated image is replaced by the reconstructed background image in the set iterations; the reconstructed background image is represented as: (5) In the formula, represents the reconstructed background image, is the generated image potential vector of the t-th iteration, I mask is the target position binary image, is the background noise image, represents dot product, represents the maximum number of iterations, is the threshold value.

[0007] Furthermore, the control network consists of the encoder layer and the middle layer of the diffusion model backbone network, and the backbone network of the diffusion model is UNet.

[0008] Furthermore, during the fine-tuning of the diffusion model, the variational autoencoder is frozen, the category label is set to a rare word token, and the rare word token is combined with the common prompt word to obtain the text prompt word.

[0009] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention uses a two-stage image generation method to achieve scene fusion, involving multiple modes such as foreground image, background image, text and conditional control image. In the first stage, noise is applied to the foreground image and the background image respectively and then merged. The merged noise image is used as the starting noise for the image generation in the second stage, which makes full use of the advantages of image-guided generation and can seamlessly merge the foreground image into the background image. In the second stage, the control network extracts control information from the conditional control image provided by the user, and injects the control information into the generation process to guide image generation, so that the model can perform personalized learning more efficiently, while enhancing the controllability of scene fusion.

[0010] 2. Use pre-trained visual classifiers to supervise the fine-tuning of the diffusion model, without the need for traditional few-sample datasets, so that the model can learn the required knowledge from richer semantic information. Perform singular value decomposition on the weight matrix of the backbone network to reduce training parameters, and use the gradient skipping strategy to greatly improve training efficiency.

[0011] 3. In the process of scene fusion, in order to reduce the impact on the background, the reconstructed background image is used to replace the background part of the generated image based on the background restoration mechanism to improve the background deviation of the generated image and reduce the loss of background information. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is the overall framework diagram of the present invention; Figure 2 is a flow chart of model fine-tuning of the present invention; Figure 3 is a flow chart of generating an image of the present invention; Figure 4 It is a visual comparison result between the model fine-tuning method of the present invention and the mainstream fine-tuning method. DETAILED DESCRIPTION

[0013] Specific embodiments are given below in conjunction with the accompanying drawings. The specific embodiments are only used to clearly and completely introduce the technical solutions of the present invention, and are not intended to limit the protection scope of the present application.

[0014] The present invention provides a multi-modal scene fusion method based on a diffusion large model (hereinafter referred to as the method, see Figures 1 to 4 ), including the following steps: Step 1: Perform singular value decomposition on the weight matrix of the diffusion model backbone network UNet; The Stable Diffusion model (SD) is a core model in the field of AI painting. It can perform image generation tasks such as text-generated images and image-generated images. It includes a backbone network (UNet), a variational autoencoder (VAE), and a text encoder (CLIP).

[0015] Flatten the weight matrix of the backbone network UNet from three dimensions to two dimensions; perform singular value decomposition on the flattened weight matrix; and update the weight matrix of the backbone network UNet according to the singular values ​​using formula (1); (1) In the formula, represents the updated weight matrix, Represents the deviation before and after the weight matrix is ​​updated, , are all orthogonal matrices, represents a list of singular values ​​in descending order, Represents the ReLU activation function.

[0016] Step 2: Use the pre-trained visual classifier to supervise the fine-tuning of the diffusion model; For fine-tuning of the diffusion model, freeze the variational autoencoder and set the class label to a rare word S * The rare word S * The tag is combined with the general prompt word to obtain the text prompt word P, such as A Photo of a S * tigercat; Input the text prompt word P into the diffusion model and iteratively generate the text conditional image U(P); The text conditional image U(P) is decoded by VAE, and then preprocessed to adjust the image size and format, so that the image is suitable for the visual classifier to obtain the preprocessed text conditional image; the preprocessed text conditional image is input into the pre-trained visual classifier to predict the category of the text conditional image U(P) and obtain the predicted label; according to the predicted label and the true label, the update loss is calculated through the loss function of formula (2), and the parameters of the backbone network UNet and the text encoder CLIP are updated to achieve fine-tuning of the diffusion model.

[0017] (2) In the formula, represents the update loss, represents the cross entropy loss, is a pre-trained visual classifier, represents VAE decoding and preprocessing operations, represents the one-hot encoding of the target category, Embedding vectors representing rare word tokens.

[0018] In order to shorten the training time, the gradient jumping strategy is used to fine-tune the diffusion model, that is, the gradient propagation reaches a certain depth before back propagation is performed to optimize the parameters of the diffusion model. The pre-trained visual classifier contains rich semantic information for category distinction and can be regarded as an expert model in different fields.

[0019] Step 3: Generate a scene fusion image using the fine-tuned diffusion model; the scene fusion is divided into two stages, the first stage is to generate a merged noise image, and the second stage is to iteratively perform scene fusion; The background image I and the target position binary image I mask , foreground image r, foreground object binary image r mask The text prompt word P is input into the fine-tuned diffusion model. The model is based on the target position binary image I mask and the foreground object binary image r mask The background image and foreground image are sampled separately and noise is applied iteratively to generate a background noise image. and foreground noise image ; Merge the background noise image with the foreground noise image to obtain a merged noise image ; The conditional control image I c Input into the pre-trained control network to obtain the latent vector of the conditional control image ; The generated image latent vector for the first iteration is the merged noise image The latent vector of the conditional control image is , the text conditional vector c, and the generated image latent vector at the t-1th iteration Input into the pre-trained control network to obtain the conditional control vector of the tth iteration , the generation process is expressed as: (3) In the formula, represents the pre-trained control network; The generated image potential vector of the t-1th iteration and the text condition vector c are input into the UNet network, and the generated feature vector is consistent with the conditional control vector of the tth iteration Add together to get the generated image potential vector of the tth iteration , the process is expressed as: (4) In the formula, Represents the UNet network; The above generation process is still based on text-to-image generation, which will cause a large deviation between the generated image and the original background image. Therefore, the present invention adopts a threshold To adjust the generation process, the background part of the generated image is replaced by the reconstructed background image in the set iterations to improve the background deviation of the generated image; the reconstruction process of the background image is expressed as: (5) In the formula, represents the reconstructed background image, represents dot product, Indicates the maximum number of iterations; According to the above process, the final generated image latent vector is obtained after multiple iterations; the final generated image latent vector is decoded by VAE to obtain a scene fusion image, so that the foreground image can be seamlessly fused to the target position of the background image according to the conditional control of the image's posture, direction and other information.

[0020] The above control network consists of the encoder layer and the intermediate layer of the diffusion model backbone network UNet, which is used to generate conditional control vectors.

[0021] In order to verify the fine-tuning effect of the diffusion model, on the one hand, the SD model was fine-tuned on all categories of the ImageNet and CUB datasets, generating 50 images for each category. In addition, the basic SD model was used [1] (The model has not been fine-tuned) 50 images are directly generated for all categories of the two datasets. The distribution difference between the generated images and the real images is measured by the two indicators of FID and KID. At the same time, the pre-trained visual classifier (VIT) is used to predict all the generated images to determine whether the generated images can accurately generate images of a specific category, and the prediction accuracy is measured by the two indicators of Top-1 and Top-5. The test results are shown in Table 1.

[0022]

[0023] As shown in Table 1, compared with the SD model directly generating images, the FID and KID indicators of the fine-tuned SD model are lower, indicating that the generated images of the fine-tuned SD model are closer to the real images. In terms of the prediction accuracy of the visual classifier, the fine-tuned SD model can predict the accuracy, and the improvement of the prediction accuracy on the CUB dataset is very obvious.

[0024] On the other hand, the fine-tuning method of the present invention is compared with the existing mainstream fine-tuning methods (including DreamBooth [2] 、Textual [3] and Lora [4] ) for comparison. For visualization of the comparison results, see Figure 4 . As can be seen from the figure, the SD model will generate incorrect results when facing ambiguous words, such as tiger cat and fox sparrow, and there are certain problems in fine-grained representation, which makes the key features of the generated image different from the real image. The mainstream fine-tuning method has solved the problem of ambiguous word generation errors to a certain extent, but there are still errors in the fine-grained representation of certain categories, and the training time is relatively long. The fine-tuned SD model of the present invention effectively avoids the ambiguous word problem, and at the same time can use the pre-trained visual classifier to effectively learn the fine-grained representation of the category, so that the generated image is closer to the real image.

[0025] In order to verify the effect of the method of the present invention, a test is conducted on the TF-ICON dataset, which contains 237 sets of images consisting of background, foreground and their binary images. The method of the present invention and the existing scene synthesis method (including SDEdit (0.4)) are used respectively. [5] 、SDEdit(0.6)、Blended [6] 、Paint [7] and DIB [8] ) for image synthesis, and the CLIP-Score and LPIPS indicators are used to evaluate the quality of image generation. The test results are shown in Table 2. CLIP-Score uses the pre-trained CLIP model to score the generated image and its corresponding text prompt to evaluate the matching degree of text and image, or calculate the similarity between two images. The higher the CLIP-Score score, the greater the image similarity. LPIPS uses a deep learning model to evaluate the similarity of two images. The smaller the LPIPS, the greater the image similarity.

[0026]

[0027] In Table 2, LPIPS(BG) refers to the binary image I mask For the foreground mask of the generated image, the same operation is performed on the original background image, and LPIPS is used to evaluate the similarity of the background. Similarly, LPIPS(FG) refers to the binary image r of the foreground object. mask The background mask of the generated image is applied to the original foreground image, and the similarity of the foreground is evaluated using LPIPS. CLIP(Image) scores the similarity between the foreground area of ​​the generated image and the foreground area of ​​the reference image, and CLIP(text) scores the generated image and its corresponding text description using the CLIP model.

[0028] SDEdit uses stochastic differential equations (SDE) for image synthesis and editing. SDE provides a mechanism to introduce randomness in the image generation process, which can help generate more diverse images; for image editing, it includes color adjustment, shape modification, etc. Blended is based on text-guided image synthesis and introduces a new editing method, that is, editing images through a diffusion process, with text driving the editing direction. Paint uses a diffusion model to implement example-based image editing. It improves the performance of the model by training a diffusion model conditioned on example images and introducing content bottlenecks and radical enhancements, allowing users to guide editing through example images, so as to perform image processing more intuitively and achieve fine-grained control and seamless fusion of image content. DIB first uses the proposed Poisson gradient loss, style loss, and content loss to synthesize a preliminary mixed image; then, the preliminary mixed image is further transformed to have a style more similar to the background target image to better match the target image. These methods either use non-diffusion methods or diffusion methods with a single control condition in one stage to achieve scene fusion, and their effects have certain problems and lack of controllability.

[0029] The method of the present invention adopts a two-stage image generation method. In the first stage, noise is applied to the foreground image and the background image respectively and then merged to obtain a merged noise image, which is used as the starting noise for the second stage image generation, making full use of the advantages of the image-guided generation method. In the second stage, the image is controlled by text conditions and additional conditions, and control is applied during the image generation process, thereby affecting the image generation process. In addition, in order to reduce the impact on the background during the second stage image generation process, a background restoration mechanism is used to reduce the loss of background information. Finally, the control network can be used to flexibly control the controllable effect of scene fusion, so that the foreground of the generated image is more in line with the requirements.

[0030] The above methods are from: [1]Rombach R, Blattmann A, Lorenz D, et al. High-resolution images synthesis with latent diffusion models[C] / / Proceedings of the IEEE / CVFconference on computer vision and pattern recognition. 2022: 10684-10695. [2]Nataniel Ruiz et al. “Dreambooth: Fine tuning text-to-imagediffusion models for subject-driven generation”.In: Proceedings of the IEEE / CVF conference on com-puter vision and pattern recognition. 2023, pp. 22500–22510. [3]Rinon Gal et al. “An image is worth one word: Per-sonalizing text-to-image generation using textual in-version”. In: arXiv preprint arXiv:2208.01618 (2022). [4]Edward J Hu et al. “Lora: Low-rank adaptation of large languagemodels”. In: arXiv preprint arXiv:2106.09685 (2021). [5]Meng C, He Y, Song Y, et al. Sdedit: Guided image synthesis andediting with stochastic differential equations[J]. arXiv preprint arXiv:2108.01073, 2021. [6]Avrahami O, Fried O, Lischinski D. Blended latent diffusion[J].ACM transactions on graphics (TOG), 2023, 42(4): 1-11. [7]Yang B, Gu S, Zhang B, et al. Paint by example: Exemplar-basedimage editing with diffusion models[C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition. 2023: 18381-18391. [8]Zhang L, Wen T, Shi J. Deep image blending[C] / / Proceedings of the IEEE / CVF winter conference on applications of computer vision. 2020: 231-240. Any matters not described in the present invention are applicable to the prior art.

Claims

1. A multimodal scene fusion method based on a diffusion large model, characterized in that: The method comprises the following steps: Step 1: Perform singular value decomposition on the weight matrix of the backbone network of the diffusion model to update the weight matrix of the backbone network; Step 2: Fine-tune the diffusion model and use the pre-trained visual classifier for supervision; Input the text prompt words into the diffusion model to generate a text-conditioned image; the text-conditioned image is decoded and preprocessed by VAE to obtain a preprocessed text-conditioned image; the preprocessed text-conditioned image is input into the pre-trained visual classifier to predict the category and obtain the predicted label; the update loss is calculated based on the predicted label and the true label, and the parameters of the diffusion model are updated to achieve fine-tuning of the diffusion model; Step 3: Generate scene fusion image using the fine-tuned diffusion model; Input the background image, the target position binary image, the foreground image, the foreground object binary image and the text prompt word into the fine-tuned diffusion model to generate a background noise image and a foreground noise image; merge the background noise image with the foreground noise image to obtain a merged noise image; Input the conditional control image into the pre-trained control network to obtain the latent vector of the conditional control image; for the t-th iteration, input the latent vector of the conditional control image, the text conditional vector, and the latent vector of the generated image of the t-1-th iteration into the pre-trained control network to obtain the conditional control vector of the t-th iteration; The generated image latent vector and text conditional vector of the t-1th iteration are input into the backbone network of the diffusion model, and the generated feature vector is added to the conditional control vector of the tth iteration to obtain the generated image latent vector of the tth iteration; after multiple iterations, the final generated image latent vector is obtained; the final generated image latent vector is decoded by VAE to obtain a scene fusion image.

2. The multimodal scene fusion method based on the diffusion large model according to claim 1 is characterized in that: In step 3, the background part of the generated image is replaced by the reconstructed background image in the set iterations; the reconstructed background image is represented as: (5) In the formula, represents the reconstructed background image, is the generated image potential vector of the t-th iteration, I mask is the target position binary image, is the background noise image, represents dot product, represents the maximum number of iterations, is the threshold value.

3. The multimodal scene fusion method based on the diffusion large model according to claim 1 or 2, characterized in that: The control network consists of the encoder layer and the middle layer of the diffusion model backbone network, and the backbone network of the diffusion model is UNet.

4. The multimodal scene fusion method based on the diffusion large model according to claim 1 is characterized in that: During the diffusion model fine-tuning process, the variational autoencoder is frozen, the category label is set to the token of a rare word, and the token of the rare word is combined with the common prompt word to obtain the text prompt word.

Images