Image enhancement method, device, apparatus, and storage medium

Abstract

This application provides an image enhancement method, apparatus, device, and storage medium. The method includes: acquiring an image, a mask, and a text prompt; wherein the mask is used to indicate the enhancement location; determining a first noise based on the mask; and performing multi-step iterative denoising based on the image and text prompt, starting with the first noise, to obtain a high-quality restored image. The method provided by this application determines the first noise based on the mask; and performs multi-step iterative denoising based on the image and text prompt, starting with the first noise, to obtain a high-quality restored image. This significantly enhances the alignment between the generated content and the text prompt without sacrificing visual plausibility, thereby obtaining a high-quality, high-fidelity image restoration result.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image enhancement method, apparatus, device, and storage medium. Background Technology

[0002] Text-guided image restoration technology aims to enable users to generate entirely new content or modify existing content within a specified area of ​​an image by entering text prompts.

[0003] The current mainstream technical solution mainly involves fine-tuning the pre-trained diffusion model. In implementation, in order to enhance the alignment of prompt words, an excessive focus is placed on text information, while neglecting the integration of the generated content with the surrounding image environment. This results in the repaired result appearing abrupt and unnatural, and the overall visual rationality and quality of the image are degraded.

[0004] Therefore, existing methods cannot effectively achieve both the alignment of prompt words and the visual plausibility of generated content at the same time. Summary of the Invention

[0005] To address one of the aforementioned technical deficiencies, this application provides an image enhancement method, apparatus, device, and storage medium.

[0006] A first aspect of this application provides an image enhancement method, the method comprising:

[0007] Retrieve the image, mask, and text hints; the mask is used to indicate the enhancement location.

[0008] Based on the mask, determine the first noise;

[0009] Starting with the first noise level, multi-step iterative denoising is performed based on the image and text prompts to obtain a high-quality restored image.

[0010] Optionally, the first noise is determined based on the mask, including:

[0011] The second noise z is randomly sampled from the standard normal distribution N(0,1). T ;

[0012] In the first denoising step, by extending the U-Net model, the second noise z is optimized using gradient descent based on the mask M′ and with the joint loss as the objective. T The mean μ and standard deviation σ of the sampling distribution are used to determine the first noise.

[0013] Optionally, the joint loss is L joint =L c +L s :

[0014] Among them, L c For the loss of cross attention, Ls This is due to a loss of self-attention.

[0015] Optionally,

[0016] Where v is the attention identifier, v = c for cross-attention, and v = s for self-attention; h′ is the length of the feature space of the extended U-Net model, w′ is the width of the feature space of the extended U-Net model, i is the length index of the feature space of the extended U-Net model, and j is the width index of the feature space of the extended U-Net model; M′ ij Let i be the value at position i,j in the mask. Let i be the value of i,j in the attention map v.

[0017] Optionally, the first noise z′ T =μ′+σ′z T ;

[0018] Where μ′ is the mean value after processing. lr is the learning rate. For the mean gradient, L joint The loss is the combined loss; σ′ is the standard deviation after treatment. The standard deviation gradient is given.

[0019] Optionally, starting with the first noise, multi-step iterative denoising is performed based on the image and text prompts to obtain a high-quality restored image, including:

[0020] Starting with the first noise, multi-step iterative denoising is performed based on image and text prompts to obtain the latent variable z0 in the last iteration;

[0021] The z0 image is decoded using a decoder to obtain a high-quality restored image.

[0022] Optionally, in any iteration of denoising, the guided noise is used... After completing the denoising process in the current step, we obtain the latent variable z for the next iteration. t-1 ;

[0023] Where t is the iteration step identifier;

[0024]

[0025] Where, ∈ θ (z t ,t,c,z m M′) represents the predicted original noise, z t The third noise in the t-th iteration is the first noise in the first iteration, and the third noise in the t-th iteration is the latent variable for the next iteration obtained in the (t+1)-th iteration.

[0026] c is for text prompt, z m M is the image, and M′ is the mask. This is a coefficient used in the diffusion process to adjust the change of guiding intensity with time step;

[0027] γ c The guiding coefficient for text alignment. Let r be the gradient of the text alignment term. c (z t c) represents the reward model for text alignment items; γ m The guiding coefficient for visual rationality item. r is the gradient of the visual rationality term. m (z t ,z m ) represents the reward model for visual rationality items; γ q The leading coefficient for human preference terms. Let r be the gradient of the human preference term. q (z t c) is a reward model for human preference items.

[0028] A second aspect of this application provides an image enhancement apparatus, the apparatus comprising:

[0029] The acquisition module is used to acquire images, masks, and text prompts; the mask is used to indicate the enhancement location.

[0030] The determination module is used to determine the first noise based on the mask;

[0031] The generation module is used to perform multi-step iterative denoising based on the first noise level and image and text prompts to obtain a high-quality restored image.

[0032] A third aspect of this application provides an electronic device, comprising:

[0033] Memory;

[0034] Processor; and

[0035] Computer programs;

[0036] The computer program is stored in the memory and configured to be executed by the processor to implement the method described in the first aspect above.

[0037] In a fourth aspect, this application provides a computer-readable storage medium having a computer program stored thereon; the computer program is executed by a processor to implement the method described in the first aspect above.

[0038] This application provides an image enhancement method, apparatus, device, and storage medium. The method includes: acquiring an image, a mask, and a text prompt; wherein the mask is used to indicate the enhancement location; determining a first noise based on the mask; and performing multi-step iterative denoising based on the image and text prompt, starting with the first noise, to obtain a high-quality restored image. The method provided by this application determines the first noise based on the mask; and performs multi-step iterative denoising based on the image and text prompt, starting with the first noise, to obtain a high-quality restored image. This significantly enhances the alignment between the generated content and the text prompt without sacrificing visual plausibility, thereby obtaining a high-quality, high-fidelity image restoration result. Attached Figure Description

[0039] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0040] Figure 1 A schematic flowchart of an image enhancement method provided in an embodiment of this application;

[0041] Figure 2 A schematic diagram illustrating the implementation principle of an image enhancement method provided in this application embodiment;

[0042] Figure 3 This is a schematic diagram of the structure of an image enhancement device provided in an embodiment of this application;

[0043] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0044] To make the technical solutions and advantages of the embodiments of this application clearer, the exemplary embodiments of this application will be described in further detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not an exhaustive list of all embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0045] In developing this application, the inventors discovered that text-guided image inpainting technology aims to enable users to generate entirely new content or modify existing content within a specified area of ​​an image by inputting text prompts. Current mainstream solutions primarily involve fine-tuning a pre-trained diffusion model, for example, by concatenating noise, masks, and feature maps of known image regions as input to fine-tune the network. To improve training efficiency, subsequent work employed a dual-branch architecture to handle conditional information about known regions. Furthermore, to enhance the alignment between the inpainted content and the text prompts, some methods use object-related masks and descriptions for training. Simultaneously, there are methods that do not require training but improve prompt alignment by resetting attention scores during the inference phase. However, none of these methods can effectively achieve both prompt alignment and visual plausibility of the generated content simultaneously.

[0046] Existing technologies generally face a core challenge: the difficulty of simultaneously ensuring both prompt alignment and visual plausibility. Specifically, existing technologies suffer from the following drawbacks: First, there is the problem of prompt neglect. Many fine-tuning-based methods use randomly generated masks and irrelevant global image descriptions during training. This causes the model to tend to guess and fill in content based on the image context rather than strictly following the user-provided text prompts, resulting in generated content that does not match the prompts. Second, visual quality deteriorates. Some methods, in order to enhance prompt alignment, overemphasize textual information, neglecting the integration of the generated content with the surrounding image environment. This leads to abrupt and unnatural restoration results, degrading the overall visual plausibility and quality of the image.

[0047] To address the aforementioned problems, this application provides an image enhancement method, apparatus, device, and storage medium. The method includes: acquiring an image, a mask, and a text prompt; wherein the mask is used to indicate the enhancement location; determining a first noise based on the mask; and performing multi-step iterative denoising based on the image and text prompt, starting with the first noise, to obtain a high-quality restored image. The method provided in this application determines the first noise based on the mask; and performs multi-step iterative denoising based on the image and text prompt, starting with the first noise, to obtain a high-quality restored image. This significantly enhances the alignment between the generated content and the text prompt without sacrificing visual rationality, thereby obtaining a high-quality, high-fidelity image restoration result.

[0048] See Figure 1 This embodiment provides an image enhancement method, the principle of which is as follows: Figure 2 As shown, the method includes:

[0049] 101, obtain image z m Mask M′ and text prompt c.

[0050] For example, to obtain the image z used as input. mMask M′ and text prompt c.

[0051] Wherein, image z m The image is of low quality.

[0052] The mask M′ is used to indicate the enhancement location. Therefore, the mask M′ represents the repair conditions given by the user. In addition, M′ is also a mask that matches the feature space size.

[0053] 102. Based on the mask, determine the first noise.

[0054] Step 102 is the pre-denoising stage of the image enhancement method provided in this embodiment, namely the prior-guided noise optimization stage as shown on the left side of Figure 2. This stage is a preprocessing stage, and its core task is to obtain and optimize the initial noise (i.e., the first noise z′) for the denoising process. T ).

[0055] The purpose of step 102 is to generate an optimized initial noise (i.e., the first noise z′) that is more conducive to image inpainting. T The quality of the generated result is strongly correlated with the attention distribution (including cross-attention and self-attention) in the early stages of generation (such as the first denoising step t=981). A good repair will have its attention highly focused on the mask region to be repaired; conversely, a failed repair will have its attention scattered. Based on this, step 102 can be implemented by extending the U-Net model, as follows:

[0056] 102-1, the second noise z randomly sampled from the standard normal distribution N(0,1). T .

[0057] In step 102-2, by extending the U-Net model, the second noise z is optimized using gradient descent based on the mask M′ and with the joint loss as the objective. T The mean μ and standard deviation σ of the sampling distribution are used to determine the first noise z′. T .

[0058] The joint loss is L joint =L c +L s :

[0059] Among them, L c For the loss of cross attention, L s This is due to a loss of self-attention.

[0060]

[0061] Where v is the attention identifier, v = c for cross-attention, and v = s for self-attention. h′ is the length of the feature space of the extended U-Net model, w′ is the width of the feature space of the extended U-Net model, i is the length index of the feature space of the extended U-Net model, and j is the width index of the feature space of the extended U-Net model. M′ ij Let i be the value at position i,j in the mask. Let i be the value of i,j in the attention map v. c For a cross-attention graph, A s For self-attention.

[0062] First noise z′ T =μ′+σ′z T .

[0063] Where μ′ is the mean value after processing. lr is the learning rate (e.g., lr = 0.01). For the mean gradient, L joint This represents the combined loss. σ′ is the standard deviation after treatment. The standard deviation gradient is given.

[0064] That is, in step 102-2, the extended model U-Net extracts its internal cross-attention map A in the first denoising step. c And self-attention map A s Based on the mask M′, calculate the loss for cross-attention. and loss of self-attention Loss of cross attention L c and loss of self-attention L s The penalty is to distract attention outside the masked area, and the reward is to focus attention within the masked area.

[0065] Joint loss L joint Loss due to cross attention L c Loss of self-attention L s Added together, that is, L joint =L c +L s With L joint To achieve the goal, the mean μ and standard deviation σ of the initial noise sampling distribution are optimized using gradient descent. The mean μ is then processed to obtain the processed mean. The standard deviation σ is processed to obtain the processed standard deviation. Finally, the first noise z′ is obtained. T =μ′+σ′z T .

[0066] In step 102, the standard noise (i.e., the second noise z) is first randomly sampled. T The noise (i.e., the first noise z′) is generated by a forward propagation and a mask M′ representing the user-given repair conditions, and through a forward propagation and backward optimization, outputs a "customized" optimized noise (i.e., the first noise z′) that is more suitable for the current repair task. T ).

[0067] 103. Starting with the first noise, multi-step iterative denoising is performed based on the image and text prompts to obtain a high-quality restored image.

[0068] Step 103 is the denoising stage of the image enhancement method provided in this embodiment, namely the decomposed training-free guidance stage as shown on the right side of Figure 2. This stage uses the initial noise (i.e., the first noise z′) of the optimized denoising process output from the first stage (i.e., step 102) as the basis. T Starting from t=T, in each iterative denoising step from t=T to 1, the predicted noise is guided and corrected in multiple ways, outputting fully denoised latent variables, and decoded into the final repaired image. The goal of the second stage (i.e., step 103) is to continuously guide the generation path throughout the subsequent denoising process to ensure the visual rationality and overall quality of the final result.

[0069] The output of the first stage (i.e., step 102) is directly used as the input of the second stage (i.e., step 103). The two stages are linked together and jointly ensure the quality of the final result.

[0070] Step 103 can be achieved through the following steps:

[0071] 103-1, with the first noise z′ T Starting from image z m The text prompt c is subjected to multi-step iterative denoising to obtain the latent variable z0 in the last iteration.

[0072] In step 103-1, the image z obtained in step 101 will be used as a basis. m The user-specified mask M′ and text prompt c, from the first noise z′ T The process begins by gradually denoising and restoring the clear latent variable z0. Figure 2 The middle is represented as z′ T →z T-1 →…→The process of z0; z t Let represent the latent variable of any denoising step t; M′ represents the mask that matches the size of the latent space; ∈ θ This indicates the noise predicted by the model in this step.

[0073] In any iteration of denoising, the guided noise After completing the denoising process in the current step, we obtain the latent variable z for the next iteration. t-1 .

[0074] Where t is the iteration step identifier.

[0075]

[0076] Where, ∈ θ (z t ,t,c,z m M′) represents the predicted original noise, z t The third noise in step t is the first noise in step 1. The third noise in step t is the latent variable for the next iteration obtained in step t+1.

[0077] c is for text prompt, z m M is the image, and M′ is the mask. This is a coefficient used in the diffusion process to adjust the change of guiding intensity with time step.

[0078] γ c The guiding coefficient for text alignment. Let r be the gradient of the text alignment term. c (z t c) represents the reward model for text alignment items. m The guiding coefficient for visual rationality item. r is the gradient of the visual rationality term. m (z t ,z m ) represents the reward model for visual rationality items. γ q The leading coefficient for human preference terms. Let r be the gradient of the human preference term. q (z t c) is a reward model for human preference items.

[0079] Step 103-1 can be implemented using a diffusion model of the score function. In step 103-1, the text-guided image restoration conditional generation task p(z) will be used. t |c,z m The conditional probability of a sub-objective can be expressed as:

[0080] p(z t |c,z m )∝p(c|z t )·p(z m |z t )·p quality (z t )·p(zt )

[0081] The set is given text c and image z. m Under the condition of generating latent variable z t The probability can be decomposed into the product of three guiding terms and a base term: text alignment term p(c|z) t Ensure the generated content conforms to the text prompts; visual plausibility item p(z) m |z t Ensure that the generated content seamlessly integrates with existing content in terms of texture and structure; human preference item p quality (z t Improve the overall aesthetic quality of the generated images.

[0082] The diffusion model of the score function can utilize the gradient of the conditional log probability. To correct for noise, in step 103-1, an existing differentiable reward model r, corresponding to each objective, is introduced. c ,r m ,r q To approximate text alignment, visual plausibility, and human preference, respectively. The computable reward function r(z) is used. t ) and the incalculable logarithmic probability logp(·|z t The relationship can be viewed as logp(·|z) t )∝r(z t Thus, the gradient of the log probability can be obtained through the gradient of the reward function. To approximate. Furthermore, by adjusting the original prediction noise ∈ θ Gradient correction enables probabilistic guidance.

[0083] Therefore, for each denoising step from t=T to 1, the following steps are performed:

[0084] In each denoising step t, the final guided noise used for denoising It is calculated using the following formula:

[0085]

[0086] In this formula, ∈ θ (z t ,t,c,z m ,M′)(i.e. Figure 2 ∈ θ The first term is the predicted raw noise, and the following three terms are correction terms. Among them, This indicates the corresponding reward model score for the current latent variable z. t The gradient indicates the direction in which the reward score increases the fastest; γ c,γ m ,γ q These are the weighting coefficients for each guidance, used to control the intensity of different guidance objectives; This is a coefficient in the diffusion process used to adjust the guide intensity as a function of time. The noise after using this guide... After completing the denoising in the current step, we obtain the latent variable z for the next step. t-1 This ensures that the final result is highly aligned with the prompt words while also possessing excellent visual plausibility and image quality.

[0087] 103-2, z0 is decoded by the decoder to obtain a high-quality restored image of the image.

[0088] After the denoising loop is completed (t=0), the final latent variable z0 is output. In step 103-2, z0 is decoded by the decoder to obtain the final repaired image.

[0089] The image enhancement method provided in this embodiment is a fine-tuning-free image restoration method. The image enhancement method provided in this embodiment does not require retraining or fine-tuning of the underlying diffusion model. Instead, it optimizes and guides the generation process through two core stages: before and during denoising.

[0090] The image enhancement method provided in this embodiment can be based on image z m The user-specified mask M′ and text prompt c are used to obtain the final high-quality restored image. The image enhancement method provided in this embodiment requires no training and is also tune-free, eliminating the need for time-consuming and laborious fine-tuning or retraining of the base model.

[0091] The image enhancement method provided in this embodiment can ultimately form a plug-and-play module that can be flexibly combined with various existing pre-trained diffusion repair models.

[0092] Furthermore, the image enhancement method provided in this embodiment can significantly improve the alignment of prompt words. Through an innovative "prior-guided noise optimization" process, this method addresses the initial noise at the very beginning of the generation process, effectively solving the problem of prompt word ignoring commonly found in existing technologies and ensuring that the model can accurately generate the user-specified content.

[0093] Furthermore, the image enhancement method provided in this embodiment ensures and enhances visual plausibility. Through a "decomposition-based, training-free guided" process, the image enhancement method provided in this embodiment generates target decomposition and uses a specialized reward model for fine-tuning. This effectively avoids the drawback of sacrificing image quality for the sake of alignment, significantly improving the integration of the repaired area with the background and the realism and aesthetics of the overall image.

[0094] Furthermore, the image enhancement method provided in this embodiment balances alignment and visual plausibility, resulting in superior overall performance. This method successfully resolves the contradiction between prompt alignment and visual plausibility. Through the collaborative work of two stages (i.e., steps 102 and 103), the final generated image is both faithful to the text description and possesses a high degree of visual realism and coherence, surpassing the overall performance of existing technologies.

[0095] This embodiment provides an image enhancement method that acquires an image, a mask, and text prompts. The mask indicates the enhancement location. Based on the mask, a first noise is determined. Using the first noise as a starting point, multi-step iterative denoising is performed based on the image and text prompts to obtain a high-quality restored image. The method provided in this embodiment determines the first noise based on the mask and performs multi-step iterative denoising based on the image and text prompts to obtain a high-quality restored image. This significantly enhances the alignment between the generated content and the text prompts without sacrificing visual plausibility, thereby achieving a high-quality, high-fidelity image restoration result.

[0096] Based on the same inventive concept as image enhancement methods, this embodiment provides an electronic device, see [link to relevant documentation]. Figure 3 The device includes:

[0097] The acquisition module 301 is used to acquire images, masks, and text prompts. The mask is used to indicate the enhancement location.

[0098] The determination module 302 is used to determine the first noise based on the mask.

[0099] The generation module 303 is used to perform multi-step iterative denoising based on the first noise as a starting point, and based on the image and text prompts, to obtain a high-quality restored image.

[0100] The determining module 302 is used to randomly sample the second noise z from the standard normal distribution N(0,1). T .

[0101] In the first denoising step, by extending the U-Net model, the second noise z is optimized using gradient descent based on the mask M′ and with the joint loss as the objective. T The mean μ and standard deviation σ of the sampling distribution are used to determine the first noise.

[0102] The joint loss is L joint =L c +L s :

[0103] Among them, L c For the loss of cross attention, L s This is due to a loss of self-attention.

[0104] in,

[0105] Where v is the attention identifier, v = c for cross-attention, and v = s for self-attention. h′ is the length of the feature space of the extended U-Net model, w′ is the width of the feature space of the extended U-Net model, i is the length index of the feature space of the extended U-Net model, and j is the width index of the feature space of the extended U-Net model. M′ ij Let i be the value at position i,j in the mask. Let i be the value of i,j in the attention map v.

[0106] Among them, the first noise z′ T =μ′+σ′z T .

[0107] Where μ′ is the mean value after processing. lr is the learning rate. For the mean gradient, L joint This represents the combined loss. σ′ is the standard deviation after treatment. The standard deviation gradient is given.

[0108] Starting with the first noise level, multi-step iterative denoising is performed based on the image and text prompts to obtain a high-quality restored image, including:

[0109] Starting with the first noise, multi-step iterative denoising is performed based on the image and text prompts to obtain the latent variable z0 in the last iteration.

[0110] The z0 image is decoded using a decoder to obtain a high-quality restored image.

[0111] In any iteration of denoising, the guided noise After completing the denoising process in the current step, we obtain the latent variable z for the next iteration. t-1 .

[0112] Where t is the iteration step identifier.

[0113]

[0114]

[0115] Where, ∈ θ (z t ,t,c,z m M′) represents the predicted original noise, z t The third noise in step t is the first noise in step 1. The third noise in step t is the latent variable for the next iteration obtained in step t+1.

[0116] c is for text prompt, z m M is the image, and M′ is the mask. This is a coefficient used in the diffusion process to adjust the change of guiding intensity with time step.

[0117] γ c The guiding coefficient for text alignment. Let r be the gradient of the text alignment term. c (z t c) represents the reward model for text alignment items. m The guiding coefficient for visual rationality item. r is the gradient of the visual rationality term. m (z t ,z m ) represents the reward model for visual rationality items. γ q The leading coefficient for human preference terms. Let r be the gradient of the human preference term. q (z t c) is a reward model for human preference items.

[0118] The device provided in this embodiment determines the first noise based on the mask; using the first noise as a starting point, it performs multi-step iterative denoising based on the image and text prompts to obtain a high-quality restored image. It can significantly enhance the alignment between the generated content and the text prompts without sacrificing visual rationality, thereby obtaining a high-quality, high-fidelity image restoration result.

[0119] Based on the same inventive concept as image enhancement methods, this embodiment provides an electronic device, which, as... Figure 4 As shown, it includes: a memory 401, a processor 402, and a computer program.

[0120] The computer program is stored in memory 401 and configured to be executed by processor 402 to implement the above-described image enhancement method.

[0121] Specifically,

[0122] Retrieve the image, mask, and text hints. The mask is used to indicate the enhancement location.

[0123] The first noise is determined based on the mask.

[0124] Starting with the first noise level, multi-step iterative denoising is performed based on the image and text prompts to obtain a high-quality restored image.

[0125] The first noise, determined based on the mask, includes:

[0126] The second noise z is randomly sampled from the standard normal distribution N(0,1). T .

[0127] In the first denoising step, by extending the U-Net model, the second noise z is optimized using gradient descent based on the mask M′ and with the joint loss as the objective. T The mean μ and standard deviation σ of the sampling distribution are used to determine the first noise.

[0128] The joint loss is L joint =L c +L s :

[0129] Among them, L c For the loss of cross attention, L s This is due to a loss of self-attention.

[0130] in,

[0131] Where v is the attention identifier, v = c for cross-attention, and v = s for self-attention. h′ is the length of the feature space of the extended U-Net model, w′ is the width of the feature space of the extended U-Net model, i is the length index of the feature space of the extended U-Net model, and j is the width index of the feature space of the extended U-Net model. M′ ij Let i be the value at position i,j in the mask. Let i be the value of i,j in the attention map v.

[0132] Among them, the first noise z′ T =μ′+σ′z T .

[0133] Where μ′ is the mean value after processing. lr is the learning rate. For the mean gradient, L joint This represents the combined loss. σ′ is the standard deviation after treatment. The standard deviation gradient is given.

[0134] Starting with the first noise level, multi-step iterative denoising is performed based on the image and text prompts to obtain a high-quality restored image, including:

[0135] Starting with the first noise, multi-step iterative denoising is performed based on the image and text prompts to obtain the latent variable z0 in the last iteration.

[0136] The z0 image is decoded using a decoder to obtain a high-quality restored image.

[0137] In any iteration of denoising, the guided noise After completing the denoising process in the current step, we obtain the latent variable z for the next iteration. t-1 .

[0138] Where t is the iteration step identifier.

[0139]

[0140] Where, ∈ θ (z t ,t,c,z m M′) represents the predicted original noise, z t The third noise in step t is the first noise in step 1. The third noise in step t is the latent variable for the next iteration obtained in step t+1.

[0141] c is for text prompt, z m M is the image, and M′ is the mask. This is a coefficient used in the diffusion process to adjust the change of guiding intensity with time step.

[0142] γ c The guiding coefficient for text alignment. Let r be the gradient of the text alignment term. c (z t c) represents the reward model for text alignment items. m The guiding coefficient for visual rationality item. r is the gradient of the visual rationality term. m (z t ,z m ) represents the reward model for visual rationality items. γ q The leading coefficient for human preference terms. Let r be the gradient of the human preference term. q (z t c) is a reward model for human preference items.

[0143] The electronic device provided in this embodiment has a computer program executed by a processor to determine the first noise based on a mask; starting from the first noise, it performs multi-step iterative denoising based on the image and text prompts to obtain a high-quality restored image. It can significantly enhance the alignment between the generated content and the text prompts without sacrificing visual rationality, thereby obtaining a high-quality, high-fidelity image restoration result.

[0144] Based on the same inventive concept as the image enhancement method, this embodiment provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the above-described image enhancement method.

[0145] Specifically,

[0146] Retrieve the image, mask, and text hints. The mask is used to indicate the enhancement location.

[0147] The first noise is determined based on the mask.

[0148] Starting with the first noise level, multi-step iterative denoising is performed based on the image and text prompts to obtain a high-quality restored image.

[0149] The first noise, determined based on the mask, includes:

[0150] The second noise z is randomly sampled from the standard normal distribution N(0,1). T .

[0151] In the first denoising step, by extending the U-Net model, the second noise z is optimized using gradient descent based on the mask M′ and with the joint loss as the objective. T The mean μ and standard deviation σ of the sampling distribution are used to determine the first noise.

[0152] The joint loss is L joint =L c +L s :

[0153] Among them, L c For the loss of cross attention, L s This is due to a loss of self-attention.

[0154] in,

[0155] Where v is the attention identifier, v = c for cross-attention, and v = s for self-attention. h′ is the length of the feature space of the extended U-Net model, w′ is the width of the feature space of the extended U-Net model, i is the length index of the feature space of the extended U-Net model, and j is the width index of the feature space of the extended U-Net model. M′ ij Let i be the value at position i,j in the mask. Let i be the value of i,j in the attention map v.

[0156] Among them, the first noise z′ T =μ′+σ′z T .

[0157] Where μ′ is the mean value after processing. lr is the learning rate. For the mean gradient, L joint This represents the combined loss. σ′ is the standard deviation after treatment. The standard deviation gradient is given.

[0158] Starting with the first noise level, multi-step iterative denoising is performed based on the image and text prompts to obtain a high-quality restored image, including:

[0159] Starting with the first noise, multi-step iterative denoising is performed based on the image and text prompts to obtain the latent variable z0 in the last iteration.

[0160] The z0 image is decoded using a decoder to obtain a high-quality restored image.

[0161] In any iteration of denoising, the guided noise After completing the denoising process in the current step, we obtain the latent variable z for the next iteration. t-1 .

[0162] Where t is the iteration step identifier.

[0163]

[0164] Where, ∈ θ (z t ,t,c,z m M′) represents the predicted original noise, z t The third noise in step t is the first noise in step 1. The third noise in step t is the latent variable for the next iteration obtained in step t+1.

[0165] c is for text prompt, z m M is the image, and M′ is the mask. This is a coefficient used in the diffusion process to adjust the change of guiding intensity with time step.

[0166] γ c The guiding coefficient for text alignment. Let r be the gradient of the text alignment term. c (z t c) represents the reward model for text alignment items. m The guiding coefficient for visual rationality item. r is the gradient of the visual rationality term. m (z t ,z m ) represents the reward model for visual rationality items. γ q The leading coefficient for human preference terms. Let r be the gradient of the human preference term. q (z t c) is a reward model for human preference items.

[0167] The computer-readable storage medium provided in this embodiment has a computer program thereon that is executed by a processor to determine a first noise based on a mask; starting with the first noise, multi-step iterative denoising is performed based on the image and text prompts to obtain a high-quality restored image. This significantly enhances the alignment between the generated content and the text prompts without sacrificing visual rationality, thereby obtaining a high-quality, high-fidelity image restoration result.

[0168] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0169] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0170] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0171] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0172] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0173] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0174] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An image enhancement method characterized by, The method comprises: obtaining an image, a mask and a text prompt; wherein the mask is used to indicate an enhancement position; determining a first noise according to the mask; taking the first noise as a starting point, performing multi-step iterative denoising based on the image and the text prompt to obtain a high-quality repaired image of the image; wherein the determining a first noise according to the mask comprises: a second noise randomly sampled from a standard normal distribution ;​ In the first denoising step, the extended model U-Net is used based on the mask. The second noise is optimized using gradient descent with the joint loss as the objective. mean of the sampling distribution with standard deviation Identify the first noise; The joint loss is : wherein, is the loss for cross attention, is the loss for self attention; wherein the taking the first noise as a starting point, performing multi-step iterative denoising based on the image and the text prompt to obtain a high-quality repaired image of the image comprises: Taking the first noise as a starting point, multi-step iterative denoising is performed based on images, text prompts, and the like to obtain the hidden variable of the last step iteration ; by a decoder to obtain a high-quality inpainted image of the image. decoded, to obtain a high-quality inpainted image of the image.

2. The method of claim 1, wherein, ； in, For attention markers, This is a time of cross attention. This is self-attention; To extend the feature space of the U-Net model, To expand the feature space of the U-Net model, To extend the long index of the feature space of the U-Net model, To extend the feature space of the U-Net model with a wide index; In the mask The value at that location, For attention graph Chinese correspondence The value of .

3. The method of claim 1, wherein, the first noise ; wherein, is the processed mean, , is the learning rate, is the mean gradient, is the joint loss; is the processed standard deviation, , is the standard deviation gradient.

4. The method of claim 1, wherein, In any step of the iterative denoising, the noise The denoising process of the current step is completed to obtain the latent variable of the next iteration ; wherein is the iteration step identification; ； wherein, is the original noise, the first step iteration, the third noise of the first step iteration is the first noise, the third noise of the second step iteration is the second noise, and the third noise of the third step iteration is the third noise, the third noise of the second step iteration is the second noise, and the third noise of the third step iteration is the third noise, the third step iteration obtains the latent variable of the next step iteration; for text cues, for images, for masks, for coefficients in the diffusion process, used to adjust the change in steering strength with time step; a guiding coefficient for a text alignment item, a gradient for a text alignment item, a reward model for a text alignment item; a guiding coefficient for a visual plausibility item, a gradient for a visual plausibility item, a reward model for a visual plausibility item; a guiding coefficient for a human preference item, a gradient for a human preference item, a reward model for a human preference item.

5. An image enhancement device, characterized by The device comprises: an obtaining module configured to obtain an image, a mask and a text prompt; wherein the mask is used to indicate an enhancement position; a determining module configured to determine a first noise according to the mask; a generating module configured to take the first noise as a starting point, perform multi-step iterative denoising based on the image and the text prompt to obtain a high-quality repaired image of the image; wherein the determining module is configured to determine the first noise from a mean and a standard deviation of a standard normal distribution based on the mask by gradient descent optimization of a joint loss of a sampling distribution and a standard deviation The joint loss is : wherein, is the loss for cross attention, is the loss for self attention; wherein the taking the first noise as a starting point, performing multi-step iterative denoising based on the image and the text prompt to obtain a high-quality repaired image of the image comprises: Taking the first noise as a starting point, multi-step iterative denoising is performed based on images, text prompts, and the like to obtain the hidden variable of the last step iteration ; by a decoder to obtain a high-quality inpainted image of the image. decoded, to obtain a high-quality inpainted image of the image.

6. An electronic device, comprising: comprises: a memory; a processor; and a computer program; wherein the computer program is stored in the memory and configured to be executed by the processor to implement the method of any one of claims 1-4. having a computer program stored thereon; the computer program is executed by a processor to implement the method of any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, ​

Images