Text-driven subject personalized completion method based on image restoration model

Abstract

The invention discloses a text-driven subject personalized completion method based on an image restoration model, and belongs to the field of image generation and editing. The method comprises the steps of 1, personalized fine-tuning of an image restoration model; and step 2, image completion of the two stages. According to the method, the problem of main body feature attribute coupling is solved by further optimizing a DreamBooth-based personalized fine tuning method, vector decomposition is proposed to further solve the coupling problem, a two-stage image personalized completion framework is proposed to improve the image restoration quality, and finally, the high-quality text-driven main body personalized completion method is realized.

Description

Technical Field

[0001] The present invention belongs to the field of image generation and editing, and in particular, relates to a text-driven subject personalized completion method based on an image restoration model. Background Art

[0002] With the continuous development of diffusion models, image generation technology based on diffusion models has made significant progress and further promoted the evolution of the image editing field. Among them, subject-driven personalized image generation has attracted much attention because it can generate realistic images of specified subjects in various scenes. Existing methods DreamBooth and IP-Adapter mainly use different adaptation strategies to enable the pre-trained text-to-image diffusion model to achieve a variety of editing effects while maintaining the identity of the object. However, most of these methods focus on the reconstruction of the entire scene, and when users want to insert specific objects in specific areas of a given scene, they face greater challenges. In order to meet practical needs, such as rendering of renderings, poster design, and virtual try-on, subject-driven image completion has gradually become an important task.

[0003] For this task, existing methods such as AnyDoor and MimicBrush extract sample information from reference images to conditionally constrain the pre-trained diffusion model. However, these reference visual features often contain too specific content, which limits the generation ability of the pre-trained text-to-image model and makes it difficult to flexibly edit the desired effects or attributes of the target object. To overcome this limitation, some recent works such as LAR-Gen have attempted to improve the effect by combining image and text-guided attention mechanisms. However, these methods still face challenges in the trade-off between training data efficiency and visual editing quality. Summary of the invention

[0004] This paper proposes a generation method based on a diffusion model for text-driven subject image completion, which effectively alleviates the trade-off between data and accuracy. In this method, by introducing parameter-efficient image customization techniques (such as DreamBooth) for model adaptation, the goal can be achieved with a small number of template images. However, the direct application of such methods still has significant limitations in achieving accurate object insertion and modification. For example, the denoising process in the repair model may introduce global noise, interfere with the integration of local information, and affect the synthesis quality of the target area. In addition, the small sample fine-tuning strategy couples the appearance of the subject to a single identity marker ([sks]), which easily leads to overfitting of the appearance, thereby weakening the editing effect of text instructions.

[0005] In view of the above problems, the present invention proposes an innovative generation framework to achieve high precision and efficiency at the attribute level. The present invention introduces a separated repair framework, which divides the repair process into two stages: local content generation and global context coordination, which respectively enhances the precise integration of local objects and the overall visual coordination consistency. At the same time, the method introduces an attribute decoupling mechanism to solve the problem of model overfitting through text descriptions and image pairs of diversified subject attributes. In addition, the text attribute replacement module introduced in the reasoning stage of the present invention uses an orthogonal decomposition strategy to separate interference information from text guidance, further improving the editing quality of the target object.

[0006] The technical solution adopted by the present invention is:

[0007] A text-driven subject personalized completion method based on an image restoration model, the steps are as follows:

[0008] Step 1: Personalized fine-tuning of image restoration model;

[0009] (1.1) Obtain a set of images with a common target subject and text descriptions corresponding to the images, wherein each image and its corresponding text description serve as a subject data pair, constituting training data for fine-tuning the image restoration model, so that the image restoration model can learn the characteristics of the target subject. The text description corresponding to the image refers to a concise description of the target subject containing an identity tag.

[0010] (1.2) Based on the target question raised by the user, a multimodal large language model is used to describe the attributes of the target subject in the set of images with a common target subject described in step (1.1), and a key-value pair including all feature descriptions is obtained. Based on the key-value pair, the multimodal large language model is used again, and based on the target question further raised by the user, more regular text about the target subject description is obtained. The regular text is a detailed description of some target subject features in the key-value pair without identity tags.

[0011] (1.3) With the help of the text-controlled image generation model, a specified number of regular images are generated based on the regular text obtained in step (1.2) to obtain regular data pairs including regular text and regular images.

[0012] (1.4) Obtain masks of all images in the subject data pair and the regular data pair with the help of the image segmentation model, and add the masks to the corresponding subject data pair and the regular data pair for fine-tuning the image restoration model. All images include the regular images obtained in step (1.3) and the images of the target subject in step (1.1).

[0013] (1.5) Use the masked subject data pairs and regular data pairs obtained in step (1.4) to fine-tune the image restoration model. Use different sampling probabilities for different subject data pairs and regular data pairs, and fine-tune the image restoration model based on the Lora method to learn the characteristics of the target subject.

[0014] Preferably, the image restoration model adopts a diffusion restoration model:

[0015] Randomly add noise to all images at time step t, send the noisy images to the UNet of the diffusion repair model, pass through the initial convolution layer, and at the same time, randomly expand the mask obtained in step (1.4) and encode it together with the corresponding image. Add the encoded whole to the result of the initial convolution layer, and perform subsequent downsampling and upsampling processing of UNet to predict the added noise. During the training process of the diffusion repair model, it is necessary to linearly transform the key parameter matrix of the attention layer of the diffusion repair model based on the lora method. Sum value linear transformation parameter matrix Perform fine-tuning training; it is necessary to calculate the reconstruction loss and use the expanded mask to weight the loss to update the parameters of LoRa.

[0016] Step 2: Two-stage image completion;

[0017] (2.1) During the image completion process, the input of the fine-tuned image inpainting model includes the background image to be completed, the binary mask image corresponding to the background image, the completion area specified by the user on the binary mask image, and the personalized text description containing the identity mark of the target subject set by the user. The completion area is used to complete the target subject. The area of ​​the specified size is cropped from the same position in the background image and the binary mask image that is larger than the completion area to obtain the local image and the local binary mask; the personalized text description containing the identity mark of the target subject set by the user, the background image to be completed, the binary mask image corresponding to the background image, the local image and the local binary mask are sent to the fine-tuned image inpainting model.

[0018] (2.2) Encode the personalized text description containing identity tags set by the user for the target subject using a text encoder , use the text encoder to encode the key-value pairs obtained in step (1.2) , then apply vector decomposition to decouple the features and obtain the decoupled feature vector. The decoupling formula is as follows:

[0019]

[0020] in, represents the magnitude of a vector;

[0021] (2.3) Encode the local image to obtain the latent variable of the local image, initialize the random noise of the same dimension with the standard normal distribution, and perform forward noise addition of the diffusion repair model on the latent variable of the local image to time step T, where T is the total number of noise addition steps of the diffusion repair model, to obtain the latent variable of the noisy local image; then, the latent variable of the noisy local image and the decoupled feature vector obtained in step (2.2) are simultaneously sent to UNet for denoising to obtain the initial prediction noise;

[0022] (2.4) The diffusion repair model uses the initial predicted noise obtained in step (2.3) to denoise the latent variable of the noisy local image before denoising in step (2.3) to obtain the latent variable of the denoised local image; next, the random noise of the same dimension is initialized using the standard normal distribution in step (2.3), and the latent variable of the local image obtained in step (2.3) is forward-noised to the current time step to obtain the latent variable of the noisy local image at the current time step; the latent variable of the denoised local image and the latent variable of the noisy local image at the current time step are fused using a local binary mask to obtain the latent variable of the noisy local image that needs to be input into UNet, and the latent variable is sent to UNet for denoising together with the decoupled feature vector obtained in step (2.2) to obtain the predicted noise of the current time step as the denoising input of the diffusion repair model for the next time step; repeat this step until , and get the prediction noise of the last time step. is a manually set hyperparameter, and T is the total number of noise addition steps of the diffusion repair model. The fusion formula is as follows:

[0023]

[0024] in represents the latent variable of the denoised local image, is a local binary mask, represents the latent variable of the noisy local image at time step t; Represents an element-wise matrix multiplication operation.

[0025] (2.5) In the diffusion inpainting model, the latent variables of the denoised local image after fusion at the last time step are thoroughly denoised using the predicted noise at the last time step, and the direct prediction is The clean latent variable is then decoded using the decoder to obtain a local clean image. Next, according to the cropped area in step (2.1), the local clean image is fused with the input background image to obtain a global clean image, and then the encoder is used to encode the latent variable corresponding to the global clean image.

[0026] (2.6) Encode the global clean image obtained in step (2.5) and the binary mask image corresponding to the background image input in step (2.1) together and send them to UNet. At the same time, in the diffusion repair model, use the standard normal distribution in step (2.3) to initialize the random noise of the same dimension, and add the noise to the latent variable corresponding to the global clean image obtained in step (2.5). , we get the latent variable of the clean image after adding noise, and then send it to UNet to get the global prediction noise.

[0027] (2.7) Using the global prediction noise obtained in step (2.6), denoise the latent variable of the clean image after adding noise in step (2.6) to obtain the latent variable of the denoised global image;

[0028] (2.8) The diffusion repair model uses the standard normal distribution in step (2.3) to initialize random noise of the same dimension, and adds noise to the latent variable corresponding to the global clean image obtained in step (2.5) to the current time step to obtain the latent variable of the noisy global image at the current time step; the latent variable of the denoised global image obtained in step (2.7) and the latent variable of the noisy global image at the current time step are fused using the binary mask image corresponding to the background image to obtain the latent variable of the noisy global image that needs to be input into UNet, and the latent variable is sent to UNet for denoising together with the decoupled feature vector obtained in step (2.2) to obtain the predicted noise of the current time step, and the predicted noise is used to denoise the latent variable of the noisy global image input into UNet to obtain the latent variable of the denoised global image at the next time step, which is used as the denoising input of the diffusion repair model at the next time step; repeat this step until , we get the latent variable of the global picture without noise at the final time step. The fusion formula is as follows:

[0029]

[0030] in represents the latent variable of the denoised global picture, is the binary mask image corresponding to the background image, A latent variable representing the noisy global picture at time step t.

[0031] (2.9) Use the image decoder of the diffusion inpainting model to decode the latent variable of the noise-free global image obtained in step (2.8) to obtain the personalized completed image required by the user.

[0032] Beneficial effects of the present invention:

[0033] 1. The present invention realizes accurate object insertion and attribute-level modification through text guidance, overcoming the problems of global noise interference and insufficient local information integration in the object editing process of existing methods, and significantly improving the practical application capabilities in multiple scenarios.

[0034] 2. The present invention innovatively proposes a separated image restoration framework, which divides the image completion process into two stages: local content generation and global context harmony, which not only enhances the local feature details of the object, but also improves the overall visual consistency.

[0035] 3. The present invention effectively alleviates the problem of object appearance overfitting by introducing an attribute decoupling mechanism and a text attribute replacement module, and supports diversified attribute editing and higher-quality text-driven object editing.

[0036] 4. The present invention outperforms mainstream methods in multiple quantitative indicators such as CLIP-T, CLIP-I and DINO in identity preservation and attribute editing tasks. It also performs outstandingly in the subjective evaluation of user surveys and can generate identity-preserving and context-harmonious repair results.

[0037] 5. Compared with the few-sample fine-tuning methods such as DreamBooth, the DreamMix of the present invention avoids the phenomena of filling irrelevant content and losing main features; compared with other large-scale training methods, the present invention achieves high-quality editing effects with less data requirements, and has significant advantages in identity preservation and attribute editing effects, and has broad potential for technical promotion. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is the overall framework diagram of the present invention. DETAILED DESCRIPTION

[0039] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0040] The invention is a text-driven subject personalized completion method based on the image restoration model, and proposes a technical solution called DreamMix. DreamMix designs a personalized fine-tuning method for feature decoupling, and designs a vector decomposition method for further decoupling, and designs a two-stage image restoration framework to improve image quality.

[0041] For feature learning of the image subject, similar to DreamBooth, DreamMix fine-tunes the image restoration model based on the Lora method to enable the model to learn the features of the subject. On this basis, in order to weaken the coupling of features, firstly, based on the target question raised by the user, a multimodal large language model is used to describe the attributes of the image subject, and a key-value pair of attribute description is obtained. Then, a variety of description sentences are generated with the help of the multimodal large language model and the key-value pair as regular text. Next, with the help of the text-controlled image generation model, a regular image corresponding to the regular text is generated. Then, in the fine-tuning stage, the subject image and text description, the regular image and text description are used to train the image restoration model at the same time, where the regular text does not contain the attribute word [sks] to avoid the word [sks] learning the decoupled features. In the fine-tuning process, unlike other methods that use randomly generated masks, this method uses the subject image generated by the image segmentation model and the mask of the regular image subject for model fine-tuning.

[0042] In order to further weaken the attribute coupling, DreamMix performs vector decomposition on text features. Specifically, for the input text, the text feature vector is first obtained through the text encoder, and then the attribute associated with the input text is selected from the attribute key-value pairs obtained from the multimodal large language model and sent to the text encoder to obtain its corresponding feature vector. Then, the orthogonal decomposition formula is applied to obtain the decoupled feature vector, which is used as the text condition to control the generation of the image restoration model.

[0043] In order to improve the quality of image restoration and enhance the detailed features of the subject and the overall harmony of the image, this method proposes a two-stage image restoration framework. First, in order to improve the detailed features of the subject, the background image is partially cropped according to the completion area specified by the user on the binary mask image, and a certain step of image restoration is performed. Next, single-step denoising is performed and decoded with an image decoder to obtain a clean local image, and then the clean local image is reintegrated into the background image, and then encoded with an image encoder, and the remaining steps of image restoration are performed on the global image. Figure 1 As shown in FIG. 1 , a text-driven subject personalized completion method based on an image restoration model is shown in FIG. 1 . The steps are as follows:

[0044] Step 1: Personalized fine-tuning of image restoration model;

[0045] (1.1) Obtain a set of images with a common target subject and text descriptions corresponding to the images, wherein each image and its corresponding text description serve as a subject data pair, constituting training data for fine-tuning the image restoration model, so that the image restoration model can learn the characteristics of the target subject. The text description corresponding to the image refers to a concise description of the target subject containing an identity tag.

[0046] (1.2) Based on the target question raised by the user, a multimodal large language model is used to describe the attributes of the target subject in the set of images with a common target subject described in step (1.1), and a key-value pair including all feature descriptions is obtained. Based on the key-value pair, the multimodal large language model is used again, and based on the target question further raised by the user, more regular text about the target subject description is obtained. The regular text is a detailed description of some target subject features in the key-value pair without identity tags (see Figure 1 (a) in the figure.

[0047] (1.3) With the help of the text-controlled image generation model, a specified number of regular images are generated based on the regular text obtained in step (1.2) to obtain regular data pairs including regular text and regular images.

[0048] (1.4) Obtain masks of all images in the subject data pair and the regular data pair with the help of the image segmentation model, and add the masks to the corresponding subject data pair and regular data pair for fine-tuning the image restoration model. All images include the regular image obtained in step (1.3) and the image of the target subject in step (1.1) (see Figure 1 (b) in the figure.

[0049] (1.5) Use the masked subject data pairs and regular data pairs obtained in step (1.4) to fine-tune the image restoration model. Use different sampling probabilities for different subject data pairs and regular data pairs, and fine-tune the image restoration model based on the Lora method to learn the characteristics of the target subject.

[0050] Preferably, the image restoration model adopts a diffusion restoration model:

[0051] Randomly add noise to all images at time step t, send the noisy images to the UNet of the diffusion repair model, pass through the initial convolution layer, and at the same time, randomly expand the mask obtained in step (1.4) and encode it together with the corresponding image. Add the encoded whole to the result of the initial convolution layer, and perform subsequent downsampling and upsampling processing of UNet to predict the added noise. During the training process of the diffusion repair model, it is necessary to linearly transform the key parameter matrix of the attention layer of the diffusion repair model based on the lora method. Sum value linear transformation parameter matrix Perform fine-tuning training; it is necessary to calculate the reconstruction loss and use the expanded mask to weight the loss to update the parameters of LoRa.

[0052] Step 2: Two-stage image completion;

[0053] (2.1) During the image completion process, the input of the fine-tuned image inpainting model includes the background image to be completed, the binary mask image corresponding to the background image, the completion area specified by the user on the binary mask image, and the personalized text description containing the identity mark of the target subject set by the user. The completion area is used to complete the target subject. The area of ​​the specified size is cropped from the same position in the background image and the binary mask image that is larger than the completion area to obtain the local image and the local binary mask; the personalized text description containing the identity mark of the target subject set by the user, the background image to be completed, the binary mask image corresponding to the background image, the local image and the local binary mask are sent to the fine-tuned image inpainting model.

[0054] (2.2) Encode the personalized text description containing identity tags set by the user for the target subject using a text encoder , use the text encoder to encode the key-value pairs obtained in step (1.2) , then apply vector decomposition to decouple the features and obtain the decoupled feature vector (see Figure 1 (c) in the figure), the decoupling formula is as follows:

[0055]

[0056] in, represents the magnitude of a vector;

[0057] (2.3) Encode the local image to obtain the latent variable of the local image, initialize the random noise of the same dimension with the standard normal distribution, and perform forward noise addition of the diffusion repair model on the latent variable of the local image to time step T, where T is the total number of noise addition steps of the diffusion repair model, to obtain the latent variable of the noisy local image; then, the latent variable of the noisy local image and the decoupled feature vector obtained in step (2.2) are simultaneously sent to UNet for denoising to obtain the initial prediction noise;

[0058] (2.4) The diffusion repair model uses the initial predicted noise obtained in step (2.3) to denoise the latent variable of the noisy local image before denoising in step (2.3) to obtain the latent variable of the denoised local image; next, the random noise of the same dimension is initialized using the standard normal distribution in step (2.3), and the latent variable of the local image obtained in step (2.3) is forward-noised to the current time step to obtain the latent variable of the noisy local image at the current time step; the latent variable of the denoised local image and the latent variable of the noisy local image at the current time step are fused using a local binary mask to obtain the latent variable of the noisy local image that needs to be input into UNet, and the latent variable is sent to UNet for denoising together with the decoupled feature vector obtained in step (2.2) to obtain the predicted noise of the current time step as the denoising input of the diffusion repair model for the next time step; repeat this step until , and get the prediction noise of the last time step. is a manually set hyperparameter, and T is the total number of noise addition steps of the diffusion repair model. The fusion formula is as follows:

[0059]

[0060] in represents the latent variable of the denoised local image, is a local binary mask, represents the latent variable of the noisy local image at time step t; Represents an element-wise matrix multiplication operation.

[0061] (2.5) In the diffusion inpainting model, the latent variables of the denoised local image after fusion at the last time step are thoroughly denoised using the predicted noise at the last time step, and the direct prediction is The clean latent variable is then decoded using the decoder to obtain a local clean image. Next, according to the cropped area in step (2.1), the local clean image is fused with the input background image to obtain a global clean image, and then the encoder is used to encode the latent variable corresponding to the global clean image.

[0062] (2.6) Encode the global clean image obtained in step (2.5) and the binary mask image corresponding to the background image input in step (2.1) together and send them to UNet. At the same time, in the diffusion repair model, use the standard normal distribution in step (2.3) to initialize the random noise of the same dimension, and add the noise to the latent variable corresponding to the global clean image obtained in step (2.5). , we get the latent variable of the clean image after adding noise, and then send it to UNet to get the global prediction noise.

[0063] (2.7) Using the global prediction noise obtained in step (2.6), denoise the latent variable of the clean image after adding noise in step (2.6) to obtain the latent variable of the denoised global image;

[0064] (2.8) The diffusion repair model uses the standard normal distribution in step (2.3) to initialize random noise of the same dimension, and adds noise to the latent variable corresponding to the global clean image obtained in step (2.5) to the current time step to obtain the latent variable of the noisy global image at the current time step; the latent variable of the denoised global image obtained in step (2.7) and the latent variable of the noisy global image at the current time step are fused using the binary mask image corresponding to the background image to obtain the latent variable of the noisy global image that needs to be input into UNet, and the latent variable is sent to UNet for denoising together with the decoupled feature vector obtained in step (2.2) to obtain the predicted noise of the current time step, and the predicted noise is used to denoise the latent variable of the noisy global image input into UNet to obtain the latent variable of the denoised global image at the next time step, which is used as the denoising input of the diffusion repair model at the next time step; repeat this step until , we get the latent variable of the global picture without noise at the final time step. The fusion formula is as follows:

[0065]

[0066] in represents the latent variable of the denoised global picture, is the binary mask image corresponding to the background image, A latent variable representing the noisy global picture at time step t.

[0067] (2.9) Use the image decoder of the diffusion inpainting model to decode the latent variable of the noise-free global image obtained in step (2.8) to obtain the personalized completed image required by the user (see Figure 1 (d) in the figure.

[0068] Example:

[0069] In this embodiment, the framework of the present invention is built based on the Fooocus repair model, and the DreamBench personalized image dataset is used for fine-tuning. The LoRA with a rank of 4 is applied to and Attention layer. During fine-tuning, images were resized to 1024×1024, using the AdamW optimizer with a learning rate of 1e-4, and the fine-tuning time per object was approximately 20 minutes on a single RTX 4090 GPU.

[0070] The present invention compares with existing excellent methods in two tasks: personalized image generation and text-driven attribute editing. Specifically, the present invention selects 30 subjects from DreamBench and designs 4000 image-mask pairs in COCO-val2017 for testing. The comparison results of various indicators are as follows:

[0071]

[0072] Among them, CLIP-I and DINO evaluate the image similarity between the generated image and the subject, and CLIP-T calculates the correlation between the generated image and the text condition.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A text-driven subject personalized completion method based on an image restoration model, characterized in that: Here are the steps: Step 1: Personalized fine-tuning of image restoration model; Step 2: Two-stage image completion; The step 1 specifically includes: (1.1) Obtain a set of images with a common target subject and text descriptions corresponding to the images, wherein each image and its corresponding text description serve as a subject data pair, constituting training data for a fine-tuned image restoration model, so that the image restoration model can learn the characteristics of the target subject; the text description corresponding to the image refers to a concise description of the target subject containing an identity tag; (1.2) Based on the target question raised by the user, a multimodal large language model is used to describe the attributes of the target subject in the set of images with a common target subject described in step (1.1), and a key-value pair including all feature descriptions is obtained; based on the key-value pair, the multimodal large language model is used again, and based on the target question further raised by the user, more regular text about the target subject description is obtained; the regular text is a detailed description of some target subject features in the key-value pair without identity tags; (1.3) using a text-controlled image generation model, generating a specified number of regular images based on the regular text obtained in step (1.2) to obtain regular data pairs including regular text and regular images; (1.4) Obtain masks of all images in the subject data pair and the regular data pair with the help of the image segmentation model, and add the masks to the corresponding subject data pair and the regular data pair for fine-tuning the image restoration model; all images include the regular image obtained in step (1.3) and the image of the target subject in step (1.1); (1.5) Using the masked subject data pairs and regular data pairs obtained in step (1.4), fine-tune the image restoration model. Different sampling probabilities are used for different subject data pairs and regular data pairs. The image restoration model is fine-tuned based on the Lora method to learn the characteristics of the target subject. The step 2 comprises: (2.1) During the image completion process, the input of the fine-tuned image inpainting model includes the background image to be completed, the binary mask image corresponding to the background image, the completion area specified by the user on the binary mask image, and the personalized text description containing the identity mark of the target subject set by the user; the completion area is used to complete the target subject, and the area of ​​the specified size is cropped from the same position in the background image and the binary mask image that is contained in and larger than the completion area to obtain a local image and a local binary mask; the personalized text description containing the identity mark of the target subject set by the user, the background image to be completed, the binary mask image corresponding to the background image, the local image, and the local binary mask are sent to the fine-tuned image inpainting model; (2.2) Encode the personalized text description containing identity tags set by the user for the target subject using a text encoder , use the text encoder to encode the key-value pairs obtained in step (1.2) , then apply vector decomposition to decouple the features and obtain the decoupled feature vector. The decoupling formula is as follows: in, represents the magnitude of a vector; (2.3) Encode the local image to obtain the latent variable of the local image, initialize the random noise of the same dimension with the standard normal distribution, and perform forward noise addition of the diffusion repair model on the latent variable of the local image to time step T, where T is the total number of noise addition steps of the diffusion repair model, to obtain the latent variable of the noisy local image; then, the latent variable of the noisy local image and the decoupled feature vector obtained in step (2.2) are simultaneously sent to UNet for denoising to obtain the initial prediction noise; (2.4) The diffusion repair model uses the initial predicted noise obtained in step (2.3) to denoise the latent variable of the noisy local image before denoising in step (2.3) to obtain the latent variable of the denoised local image; next, the random noise of the same dimension is initialized using the standard normal distribution in step (2.3), and the latent variable of the local image obtained in step (2.3) is forward-noised to the current time step to obtain the latent variable of the noisy local image at the current time step; the latent variable of the denoised local image and the latent variable of the noisy local image at the current time step are fused using a local binary mask to obtain the latent variable of the noisy local image that needs to be input into UNet, and the latent variable is sent to UNet for denoising together with the decoupled feature vector obtained in step (2.2) to obtain the predicted noise of the current time step as the denoising input of the diffusion repair model for the next time step; repeat this step until , and get the prediction noise of the last time step; where is a manually set hyperparameter, T is the total number of noise addition steps of the diffusion repair model; the fusion formula is as follows: in represents the latent variable of the denoised local image, is a local binary mask, represents the latent variable of the noisy local image at time step t; Represents the element-by-element multiplication operation of a matrix; (2.5) In the diffusion inpainting model, the latent variables of the denoised local image after fusion at the last time step are thoroughly denoised using the predicted noise at the last time step, and the direct prediction is The clean latent variable is then decoded by the decoder to obtain a local clean image. Next, according to the cropped area in step (2.1), the local clean image is fused with the input background image to obtain a global clean image, and then the encoder is used to encode the latent variable corresponding to the global clean image. (2.6) Encode the global clean image obtained in step (2.5) and the binary mask image corresponding to the background image input in step (2.1) together and send them to UNet; at the same time, in the diffusion repair model, use the standard normal distribution in step (2.3) to initialize the random noise of the same dimension, and add the noise to the latent variable corresponding to the global clean image obtained in step (2.5) , get the latent variable after adding noise to the clean image, and then send it to UNet to get the global prediction noise; (2.7) Using the global prediction noise obtained in step (2.6), denoise the latent variable of the clean image after adding noise in step (2.6) to obtain the latent variable of the denoised global image; (2.8) The diffusion repair model uses the standard normal distribution in step (2.3) to initialize random noise of the same dimension, and adds noise to the latent variable corresponding to the global clean image obtained in step (2.5) to the current time step to obtain the latent variable of the noisy global image at the current time step; the latent variable of the denoised global image obtained in step (2.7) and the latent variable of the noisy global image at the current time step are fused using the binary mask image corresponding to the background image to obtain the latent variable of the noisy global image that needs to be input into UNet, and the latent variable is sent to UNet for denoising together with the decoupled feature vector obtained in step (2.2) to obtain the predicted noise of the current time step, and the predicted noise is used to denoise the latent variable of the noisy global image input into UNet to obtain the latent variable of the denoised global image at the next time step, which is used as the denoising input of the diffusion repair model at the next time step; repeat this step until , get the latent variable of the global picture without noise at the final time step; the fusion formula is as follows: in represents the latent variable of the denoised global picture, is the binary mask image corresponding to the background image, The latent variable representing the noisy global picture at time step t; (2.9) Use the image decoder of the diffusion inpainting model to decode the latent variable of the noise-free global image obtained in step (2.8) to obtain the personalized completed image required by the user.

2. According to the text-driven subject personalized completion method based on the image restoration model of claim 1, it is characterized in that: The image restoration model in step 1 adopts a diffusion restoration model: randomly add noise to all images to time step t, send the noisy images to the UNet of the diffusion restoration model, pass through the initial convolution layer, and at the same time, dilate the mask obtained in step (1.4) with a random size and encode it together with the corresponding image, add the encoded whole to the result after the initial convolution layer, and perform subsequent downsampling and upsampling processing of UNet to predict the added noise; During the training of the diffusion repair model, it is necessary to linearly transform the key parameter matrix of the attention layer of the diffusion repair model based on the lora method Sum value linear transformation parameter matrix Perform fine-tuning training; it is necessary to calculate the reconstruction loss and use the expanded mask to weight the loss to update the parameters of LoRa.

Images