Image target removal method and device and electronic equipment

Abstract

The invention provides an image target removal method and apparatus, and an electronic device. The method comprises the steps of obtaining a to-be-processed image and a corresponding foreground target mask; inputting the to-be-processed image and the foreground target mask into a pre-training diffusion model; wherein the pre-training diffusion model comprises an auto-encoder module and a noise prediction module; encoding the to-be-processed image through an encoder in the self-encoder module to obtain an initial hidden variable corresponding to the to-be-processed image; performing forward diffusion noise adding processing on the initial hidden variable to obtain a noise-containing hidden variable; based on the foreground target mask, executing a reverse diffusion sampling process on the noise-containing hidden variable through a noise prediction module, and obtaining and correcting a self-attention map of each time step; and guiding a reverse diffusion sampling process based on the corrected self-attention map of each time step and a self-attention redirection guiding mode so as to realize target removal of the to-be-processed image. According to the invention, the target removal quality and stability can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image object removal method, device and electronic device. Background Art

[0002] As an important task in the field of computer vision, the target removal task aims to automatically remove the specified target or object from the image while maintaining coherence and naturalness with the surrounding environment. This task also has a wide range of practical applications. For example, in image editing software, users can use the target removal function to delete unwanted objects or areas in the image, thereby improving the aesthetics and visual effects of the image, which has important applications in advertising, graphic design, photography and other fields.

[0003] Traditional object removal methods are based on blocks or CNNs. Although these methods can achieve certain results in some cases, they have some limitations, especially in maintaining the consistency and naturalness of the removed area with the surrounding environment. In addition, for complex scenes and large-scale removal, these methods have limited processing capabilities and often fail to achieve satisfactory results.

[0004] Recently, with the rapid development of generative models, more and more generative models have been applied to target removal tasks, among which the more common ones are methods based on generative adversarial networks and methods based on diffusion models. However, these methods also have some shortcomings. The training speed of generative adversarial networks is slow and unstable, and they are prone to problems such as mode collapse (reduced diversity of images generated by the generator) or mode oscillation (instability of the training dynamics between the generator and the discriminator). Although methods based on diffusion models can achieve target removal, due to the randomness of the diffusion model itself and the inaccurate guidance operation, these methods may cause artifacts in the final result or incomplete target removal. Summary of the invention

[0005] The purpose of the present application is to provide an image object removal method, device and electronic device, which encodes the image to be processed and performs forward diffusion noise processing through a pre-trained diffusion model, and performs a reverse diffusion sampling process based on a foreground object mask, and outputs an image after the object is removed in conjunction with a self-attention redirection guidance method. The pre-trained diffusion model does not require additional fine-tuning, which greatly reduces the training overhead and can improve the quality and stability of object removal.

[0006] In a first aspect, the present application provides an image target removal method, the method comprising: obtaining an image to be processed and a corresponding foreground target mask; inputting the image to be processed and the foreground target mask into a pre-trained diffusion model; wherein the pre-trained diffusion model comprises an autoencoder module and a noise prediction module; encoding the image to be processed through the encoder in the autoencoder module to obtain an initial latent variable corresponding to the image to be processed; performing forward diffusion and noise addition processing on the initial latent variable to obtain a noisy latent variable; based on the foreground target mask, performing a reverse diffusion sampling process on the noisy latent variable through the noise prediction module to obtain and correct the self-attention map of each time step; based on the corrected self-attention map of each time step, and the self-attention redirection guidance method, guiding the reverse diffusion sampling process to achieve target removal of the image to be processed.

[0007] Furthermore, the above-mentioned step of performing forward diffusion and noise addition processing on the initial latent variable to obtain the noisy latent variable includes: setting the total number of diffusion steps and the variance strategy; calculating the forward diffusion coefficient according to the total number of diffusion steps and the variance strategy; and obtaining the noisy latent variable after adding noise to the total number of diffusion steps according to the forward diffusion coefficient.

[0008] Furthermore, the above-mentioned noise prediction module is a U-net network architecture; based on the foreground target mask, the noise prediction module performs a reverse diffusion sampling process on the noisy hidden variables, and the steps of obtaining and correcting the self-attention map of each time step include: for each layer and each time step of the model, the following steps are performed: according to the noisy hidden variables, the self-attention similarity matrix corresponding to the current time step of the current layer of the model is determined; according to the foreground target mask, the self-attention similarity matrix is ​​corrected to obtain a corrected foreground target similarity matrix and a corrected background similarity matrix; according to the corrected foreground target similarity matrix, the corrected background similarity matrix, and the corresponding Value matrix, the corrected foreground attention output result and the corrected background attention output result are determined; the corrected foreground attention output result and the corrected background attention output result are integrated to obtain the corrected self-attention map corresponding to the current time step of the current layer.

[0009] Furthermore, the step of correcting the self-attention similarity matrix according to the foreground target mask to obtain a corrected foreground target similarity matrix and a corrected background similarity matrix includes: obtaining the corrected foreground target similarity matrix and the corrected background similarity matrix according to the following first specified formula: ; ; ; in, represents the corrected foreground target similarity matrix; represents the corrected background similarity matrix; Indicates infinity; represents the flattened foreground object mask; Represents the pixel set of the foreground target area; Represents a set of pixels in the background area; Represents the model Tier The self-attention similarity matrix corresponding to the time step; The step of determining a modified foreground target attention output result and a modified background attention output result according to the modified foreground target similarity matrix, the modified background similarity matrix, and the corresponding Value matrix comprises: determining the modified foreground target attention output result and the modified background attention output result according to the following second specified formula: ; ; in, Represents the corrected foreground target attention output result; Represents the corrected background attention output result; Represents the model Tier The Value matrix corresponding to the time step; Represents the corrected foreground target self-attention map; Represents the corrected background self-attention map; softmax() is the normalization function; The step of integrating the corrected foreground target attention output result and the corrected background attention output result to obtain a corrected self-attention map corresponding to the current time step of the current layer includes: according to the following third specified formula, the corrected self-attention map corresponding to the current time step of the current layer: ; in, Represents the transposed flattened foreground object mask.

[0010] Furthermore, the above-mentioned self-attention map based on the corrected time step and the self-attention redirection guidance method guides the reverse diffusion sampling process to achieve the step of removing the target of the image to be processed, including: taking the first time step as the current time step, taking the noisy latent variable as the current predicted latent variable, and performing the following prediction correction fusion steps: predicting the current predicted latent variable of the current time step through the noise prediction module to obtain the model prediction noise; based on the corrected self-attention map of the current time step, correcting the model prediction noise to obtain the corrected prediction noise; according to the corrected prediction noise, the forward diffusion coefficient , calculate the predicted latent variable of the next time step and the noisy latent variable of the next time step in the reverse diffusion sampling process; fuse the predicted latent variable within the foreground target mask and the noisy latent variable outside the mask to retain the background information, obtain the fused latent variable, take the next time step as the current time step, and use the fused latent variable as the current predicted latent variable again, continue to execute the prediction correction fusion step, until all time steps are traversed, and use the fused latent variable of the last time step as the latent variable output result; decode the latent variable output result through the decoder in the autoencoder module to obtain the output image after the target is removed.

[0011] Furthermore, the step of correcting the model prediction noise based on the corrected self-attention map of the current time step to obtain the corrected prediction noise includes: correcting the model prediction noise according to the following fourth specified formula to obtain the corrected prediction noise: ; in, represents the predicted latent variable at the tth time step in the reverse diffusion sampling process, ; Indicates t model prediction noise at time steps; Remove the guided coefficient for the target; AAS() means to perform the reverse diffusion sampling process on the noisy latent variable through the noise prediction module to obtain and correct t The entire processing of the self-attention map of time steps; Indicates the corrected t Corrected prediction noise for the time step.

[0012] Furthermore, the step of calculating the predicted latent variable of the next time step and the noisy latent variable of the next time step in the reverse diffusion sampling process according to the corrected prediction noise and the forward diffusion coefficient includes: calculating the predicted latent variable of the next time step according to the following fifth specified formula: ; in, , which represents the reverse diffusion sampling process t The forward diffusion coefficient at the time step; Indicates i Variance strategy at the moment; Represents the next time step in the reverse diffusion sampling process t Forward diffusion coefficient at -1; express t Corrected prediction noise for the time step; The next time step in the reverse diffusion sampling process t -1 predicted latent variable; According to the sixth specified formula below, calculate the noisy latent variable for the next time step: ; in , Indicates a zero mean matrix and a unit covariance matrix The standard Gaussian distribution of is a random Gaussian noise randomly drawn from it; is the latent variable representation of the input image; Indicates the next time step t Noisy latent variable at -1.

[0013] In the second aspect, the present application also provides an image target removal device, which includes: an image mask acquisition module, used to acquire the image to be processed and the corresponding foreground target mask; a model input module, used to input the image to be processed and the foreground target mask into a pre-trained diffusion model; wherein the pre-trained diffusion model includes an autoencoder module and a noise prediction module; an encoding module, used to encode the image to be processed through the encoder in the autoencoder module to obtain the hidden variables corresponding to the image to be processed; a denoising module, used to perform forward diffusion denoising on the hidden variables to obtain noisy hidden variables; a reverse diffusion sampling module, used to perform a reverse diffusion sampling process on the noisy hidden variables through the noise prediction module based on the foreground target mask, to obtain and correct the self-attention map of each time step; a diffusion guidance module, used to guide the reverse diffusion sampling process based on the corrected self-attention map of each time step and the self-attention redirection guidance method, so as to achieve target removal of the image to be processed.

[0014] In a third aspect, the present application further provides an electronic device, comprising a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the method described in the first aspect above.

[0015] In a fourth aspect, the present application further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method described in the first aspect above.

[0016] In the image target removal method, device and electronic device provided by the present application, firstly, the image to be processed and the corresponding foreground target mask are obtained; then the image to be processed and the foreground target mask are input into the pre-trained diffusion model; wherein the pre-trained diffusion model includes an autoencoder module and a noise prediction module; the image to be processed is encoded by the encoder in the autoencoder module to obtain the initial latent variable corresponding to the image to be processed; the initial latent variable is forward diffused and denoised to obtain the noisy latent variable; based on the foreground target mask, the noisy latent variable is subjected to a reverse diffusion sampling process by the noise prediction module to obtain and correct the self-attention map of each time step; based on the corrected self-attention map of each time step, and the self-attention redirection guidance method, the reverse diffusion sampling process is guided to achieve the target removal of the image to be processed. In this method, the image to be processed is encoded and forward diffused and denoised by the pre-trained diffusion model, and the reverse diffusion sampling process is performed based on the foreground target mask, and the image after the target is removed is output in combination with the self-attention redirection guidance method, and the pre-trained diffusion model does not require additional fine-tuning, which greatly reduces the training overhead and can improve the quality and stability of target removal. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A flowchart of an image object removal method provided in an embodiment of the present application; Figure 2 A pre-training model architecture diagram provided for an embodiment of the present application; Figure 3 A schematic diagram of a forward and reverse diffusion process provided in an embodiment of the present application; Figure 4 A schematic diagram of a model processing process provided in an embodiment of the present application; Figure 5 A structural block diagram of an image object removal device provided in an embodiment of the present application; Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The technical solution of the present application will be described clearly and completely below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0020] In the prior art, although the methods based on diffusion models can achieve target removal, due to the randomness of the diffusion model itself and the inaccurate guidance operation, these methods may cause artifacts in the final result or incomplete target removal.

[0021] Based on this, the embodiments of the present application provide an image target removal method, device and electronic device, which encode the image to be processed and perform forward diffusion noise processing through a pre-trained diffusion model, and perform a reverse diffusion sampling process based on a foreground target mask, and output the image after the target is removed in conjunction with a self-attention redirection guidance method. The pre-trained diffusion model does not require additional fine-tuning, which greatly reduces the training overhead and can improve the quality and stability of target removal.

[0022] To facilitate understanding of this embodiment, a method for removing an image object disclosed in an embodiment of the present application is first introduced in detail. Figure 1 A flowchart of an image object removal method provided in an embodiment of the present application, the method comprising the following steps: Step S102, obtaining the image to be processed and the corresponding foreground object mask; Step S104, inputting the image to be processed and the foreground object mask into a pre-trained diffusion model; wherein the pre-trained diffusion model includes an autoencoder module and a noise prediction module; Step S106, encoding the image to be processed through the encoder in the autoencoder module to obtain an initial latent variable corresponding to the image to be processed; Step S108, performing forward diffusion noise adding processing on the initial latent variable to obtain a noisy latent variable; Step S110, based on the foreground target mask, the noise prediction module performs a reverse diffusion sampling process on the noisy latent variable to obtain and correct the self-attention map of each time step; Step S112, based on the corrected self-attention map of each time step and the self-attention redirection guidance method, the reverse diffusion sampling process is guided to achieve target removal of the image to be processed.

[0023] In the image target removal method provided in the embodiment of the present application, the input of the pre-trained diffusion model is the image to be processed and the corresponding foreground target mask to be removed, and the output of the model is the image after the specified foreground target is removed. In the above-mentioned pre-trained diffusion model, the redirected self-attention is used to guide the reverse diffusion sampling process to achieve the guidance method of image target removal, which surpasses the existing most advanced methods in terms of the quality and stability of target removal. In this embodiment, the model does not require additional fine-tuning, which greatly reduces the overhead required for training. Since this embodiment works according to the principle of the diffusion model itself, it can be implemented in various pre-trained diffusion model architectures and checkpoints, and has excellent scalability. Unlike training-based methods that can only be applied to specific network architectures, this embodiment can be applied to new pre-trained diffusion models generated iteratively, and corresponding updates are performed to further improve the target removal effect using a more powerful model.

[0024] The following is a detailed description of the model processing process: Step 1, using the encoder in the autoencoder module of the pre-trained diffusion model to encode the image to be processed, and obtain the initial latent variable corresponding to the image to be processed; In this embodiment, the pre-trained diffusion model used is Stable Diffusion (SD), which is a type of pre-trained latent variable graph diffusion model. Figure 2 As shown, it mainly includes the autoencoder module and the noise prediction network. Given an input image with three RGB channels in pixel space , the encoder in the autoencoder module Encode it into a latent variable representation in the latent variable space (i.e., the initial latent variable) ,in .

[0025] Step 2, perform forward diffusion noise processing on the initial latent variables to obtain noisy latent variables; like Figure 3 As shown, the forward diffusion process is designed to Step 2 adds a small amount of Gaussian noise to the input image to represent the latent variables of the input image. Converted to random Gaussian noise . The forward diffusion process can be defined as follows: ; in is the variance strategy, is the identity matrix. It can be deduced that: ; in and .

[0026] The specific steps of adding noise to latent variables in the forward diffusion process are as follows: Step 2.1, set the total number of diffusion steps and variance strategies ; Step 2.2, according to the total number of diffusion steps and variance strategies , calculate the forward diffusion coefficient ; Step 2.3, according to the forward diffusion coefficient, the total number of diffusion steps of the initial latent variable is added with noise to obtain the noise The noisy latent variable after step , ,in Indicates a zero mean matrix and a unit covariance matrix The standard Gaussian distribution of is a random Gaussian noise randomly drawn from it.

[0027] Step 3: Based on the foreground target mask, the noise prediction module performs a reverse diffusion sampling process on the noisy latent variable to obtain and correct the self-attention map of each time step; In this embodiment, the noise prediction module of the pre-trained diffusion model Most of them are based on the U-net network architecture, which consists of three parts: downsampling encoder ,bottleneck , upsampling decoder The self-attention module plays an important role in the U-net network. It uses the attention mechanism to aggregate features, thereby achieving more detailed control over the details of image generation. At the same time, it can also maintain control over the overall structure and content to ensure the consistency and integrity of the generated image. The self-attention map can be obtained from the U-net model ,in Represents the model Tier The similarity matrix corresponding to the time step, the specific steps to modify the self-attention map of each time step are as follows: For each layer of the model at each time step, the following steps are performed: Step 3.1, according to the noisy latent variable Determine the self-attention similarity matrix corresponding to the current time step of the current layer of the model Specifically, given a latent variable ,in Represent the height, width and channel dimensions respectively, then the corresponding query matrix , key matrix and value matrix Through a learnable linear layer , and We obtain is the matrix dimension. Similarity matrix Can be defined as: ; ; Step 3.2, according to the foreground target mask, the self-attention similarity matrix is ​​corrected to obtain a corrected foreground target similarity matrix and a corrected background similarity matrix; The corrected foreground target similarity matrix and the corrected background similarity matrix are obtained according to the following first specified formula: ; ; ; in, represents the corrected foreground target similarity matrix; represents the corrected background similarity matrix; Indicates infinity; represents the flattened foreground object mask; A set of pixels in the foreground target area; Represents a set of pixels in the background area; Represents the model Tier The self-attention similarity matrix corresponding to the time step; Step 3.3, determining a corrected foreground attention output result and a corrected background attention output result according to the corrected foreground target similarity matrix, the corrected background similarity matrix, and the corresponding Value matrix; According to the following second specified formula, the corrected foreground target attention output result and the corrected background attention output result are determined: ; ; in, Represents the corrected foreground target attention output result; Represents the corrected background attention output result; Represents the model Tier The Value matrix corresponding to the time step; Represents the corrected foreground target self-attention map; Represents the corrected background self-attention map; softmax() is the normalization function; In step 3.4, the corrected foreground attention output results and the corrected background attention output results are integrated to obtain the corrected self-attention map corresponding to the current time step of the current layer.

[0028] According to the third specified formula below, the corrected self-attention map corresponding to the current time step of the current layer is: ; in, represents the flattened foreground target mask after transposition. This process can ensure that the above correction operations are performed in the corresponding foreground target area and background area respectively.

[0029] The above method of modifying self-attention is called "attention activation and suppression", abbreviated as AAS in English, and the U-net network processed by the AAS method is represented as .

[0030] Step 4: Based on the corrected self-attention map of each time step and the self-attention redirection guidance method, the reverse diffusion sampling process is guided to achieve target removal of the image to be processed; Specifically, take the first time step as the current time step, take the noisy latent variable as the current prediction latent variable, and perform the following prediction correction fusion steps, such as Figure 4 As shown: Step 4.1, predict the current prediction latent variable of the current time step through the noise prediction module to obtain the model prediction noise; Step 4.2, based on the corrected self-attention map of the current time step, correct the model prediction noise to obtain the corrected prediction noise; According to the fourth specified formula below, the model prediction noise is corrected to obtain the corrected prediction noise: ; in, represents the predicted latent variable at the tth time step in the reverse diffusion sampling process, ; Indicates t model prediction noise at time steps; Remove the guided coefficient for the target; AAS() means to perform the reverse diffusion sampling process on the noisy latent variable through the noise prediction module to obtain and correct t The entire processing of the self-attention map of time steps; Indicates the corrected t Corrected prediction noise for the time step.

[0031] Step 4.3, based on the corrected prediction noise and the forward diffusion coefficient, calculate the prediction latent variable of the next time step and the noisy latent variable of the next time step in the reverse diffusion sampling process; like Figure 3 As shown, from Initially, the reverse diffusion sampling process aims to Iterative sampling obtains true samples. However It is difficult to calculate, so we need to use the parameter A deep neural network posterior estimate of To fit it. The joint distribution is called the reverse diffusion process, which is defined as The starting Markov chain that learns the Gaussian transition probability distribution:

[0032] The mean It can be obtained by deep neural network training, and the variance is generally fixed to the same variance as the forward diffusion process distribution In order to control the randomness of the reverse diffusion sampling process and accelerate sampling, deterministic DDIM sampling modifies the reverse diffusion process into the following form:

[0033] Therefore, the predicted latent variable for the next time step is calculated according to the fifth specified formula as follows: ; in, , which represents the reverse diffusion sampling process t The forward diffusion coefficient at the time step; Indicates i Variance strategy at the moment; Represents the next time step in the reverse diffusion sampling process t Forward diffusion coefficient at -1; express t Corrected prediction noise for the time step; The next time step in the reverse diffusion sampling process t -1 predicted latent variable; According to the sixth specified formula below, calculate the noisy latent variable for the next time step: ; in , Indicates a zero mean matrix and a unit covariance matrix The standard Gaussian distribution of is a random Gaussian noise randomly drawn from it; is the latent variable representation of the input image; Indicates the next time step t Noisy latent variable at -1.

[0034] Step 4.4, fuse the predicted latent variables within the foreground target mask and the noisy latent variables outside the mask to retain the background information and obtain the fused latent variables; For example, according to the formula , the predicted latent variables within the foreground target mask and the noisy latent variables outside the mask are fused to obtain the fused latent variables.

[0035] Step 4.5, take the next time step as the current time step, and use the fused latent variable as the current prediction latent variable again, and continue to execute the prediction correction fusion step until all time steps are traversed, and then use the fused latent variable of the last time step as the latent variable output result; when Repeat the previous steps 4.1-4.4 until the hidden variable output result is obtained .

[0036] Step 4.6, through the decoder in the autoencoder module Decode the hidden variable output result to get the output image after the target is removed .

[0037] The embodiment of the present application also provides an image target removal method, which is a fine-tuning-free image target removal method based on a diffusion model, and generally includes the following steps: 1) Encode the input image into latent variables using the autoencoder module of the pre-trained diffusion model; 2) Forward diffuse and denoise the latent variables; 3) Input the noisy latent variables into the denoising module of the pre-trained diffusion model to perform a reverse diffusion sampling process, obtain and modify the self-attention map of each time step; 4) Construct a "self-attention redirection guidance" method to guide the reverse diffusion sampling process to achieve target removal. This embodiment can achieve effective and stable target removal for a given image using a pre-trained diffusion model without the need for additional fine-tuning, and can be applied to different pre-trained diffusion models.

[0038] Based on the above method embodiment, the present application embodiment also provides an image object removal device, see Figure 5As shown, the device includes: an image mask acquisition module 502, which is used to acquire the image to be processed and the corresponding foreground target mask; a model input module 504, which is used to input the image to be processed and the foreground target mask into a pre-trained diffusion model; wherein the pre-trained diffusion model includes an autoencoder module and a noise prediction module; an encoding module 506, which is used to encode the image to be processed through the encoder in the autoencoder module to obtain the hidden variables corresponding to the image to be processed; a denoising module 508, which is used to perform forward diffusion denoising on the hidden variables to obtain noisy hidden variables; a reverse diffusion sampling module 510, which is used to perform a reverse diffusion sampling process on the noisy hidden variables through the noise prediction module based on the foreground target mask, and obtain and correct the self-attention map of each time step; a diffusion guidance module 512, which is used to guide the reverse diffusion sampling process based on the corrected self-attention map of each time step and the self-attention redirection guidance method, so as to achieve the target removal of the image to be processed.

[0039] Furthermore, the noise adding module 508 is used to set the total diffusion steps and the variance strategy; calculate the forward diffusion coefficient according to the total diffusion steps and the variance strategy; and obtain the noisy latent variable after adding the total diffusion steps to the initial latent variable according to the forward diffusion coefficient.

[0040] Furthermore, the above-mentioned noise prediction module is a U-net network architecture; the reverse diffusion sampling module 510 is used to perform the following steps for each layer and each time step of the model: determine the self-attention similarity matrix corresponding to the current time step of the current layer of the model according to the noisy hidden variables; according to the foreground target mask, correct the self-attention similarity matrix to obtain a corrected foreground target similarity matrix and a corrected background similarity matrix; according to the corrected foreground target similarity matrix, the corrected background similarity matrix, and the corresponding Value matrix, determine the corrected foreground attention output result and the corrected background attention output result; integrate the corrected foreground attention output result and the corrected background attention output result to obtain the corrected self-attention map corresponding to the current time step of the current layer.

[0041] Furthermore, the reverse diffusion sampling module 510 is used to obtain a modified foreground target similarity matrix and a modified background similarity matrix according to the following first specified formula: ; ; ; in, represents the corrected foreground target similarity matrix; represents the corrected background similarity matrix; Indicates infinity; represents the flattened foreground object mask; A set of pixels in the foreground target area; Represents a set of pixels in the background area; Represents the model Tier The self-attention similarity matrix corresponding to the time step; According to the following second specified formula, the corrected foreground target attention output result and the corrected background attention output result are determined: ; ; in, Represents the corrected foreground target attention output result; Represents the corrected background attention output result; Represents the model Tier The Value matrix corresponding to the time step; Represents the corrected foreground target self-attention map; Represents the corrected background self-attention map; softmax() is the normalization function; According to the third specified formula below, the corrected self-attention map corresponding to the current time step of the current layer is: ; in, Represents the transposed flattened foreground object mask.

[0042] Furthermore, the diffusion guidance module 512 is used to take the first time step as the current time step and the noisy latent variable as the current predicted latent variable, and perform the following prediction correction fusion steps: predict the current predicted latent variable of the current time step through the noise prediction module to obtain the model prediction noise; based on the corrected self-attention map of the current time step, correct the model prediction noise to obtain the corrected prediction noise; calculate the predicted latent variable of the next time step and the noisy latent variable of the next time step in the reverse diffusion sampling process according to the corrected prediction noise and the forward diffusion coefficient; fuse the predicted latent variable within the foreground target mask and the noisy latent variable outside the mask to retain the background information to obtain the fused latent variable, take the next time step as the current time step, and use the fused latent variable again as the current predicted latent variable, continue to perform the prediction correction fusion steps, until all time steps are traversed, and the fused latent variable of the last time step is used as the latent variable output result; decode the latent variable output result through the decoder in the autoencoder module to obtain the output image after the target is removed.

[0043] Furthermore, the diffusion guiding module 512 is used to correct the model prediction noise according to the following fourth specified formula to obtain the corrected prediction noise: ; in, represents the predicted latent variable at the tth time step in the reverse diffusion sampling process, ; Indicates t model prediction noise at time steps; Remove the guided coefficient for the target; AAS() means to perform the reverse diffusion sampling process on the noisy latent variable through the noise prediction module to obtain and correct t The entire processing of the self-attention map of time steps; Indicates the corrected t Corrected prediction noise for the time step.

[0044] Furthermore, the diffusion guidance module 512 is used to calculate the predicted latent variable of the next time step according to the following fifth specified formula: ; in, , which represents the reverse diffusion sampling process t The forward diffusion coefficient at the time step; Indicates i Variance strategy at the moment; Represents the next time step in the reverse diffusion sampling process t Forward diffusion coefficient at -1; express t Corrected prediction noise for the time step; The next time step in the reverse diffusion sampling process t -1 predicted latent variable; According to the sixth specified formula below, calculate the noisy latent variable for the next time step: ; in , Indicates a zero mean matrix and a unit covariance matrix The standard Gaussian distribution of is a random Gaussian noise randomly drawn from it; is the latent variable representation of the input image; Indicates the next time step t Noisy latent variable at -1.

[0045] The device provided in the embodiment of the present application has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the embodiment of the device, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0046] The present application also provides an electronic device, such as Figure 6As shown, it is a schematic diagram of the structure of the electronic device, wherein the electronic device includes a processor 61 and a memory 60, the memory 60 stores computer executable instructions that can be executed by the processor 61, and the processor 61 executes the computer executable instructions to implement the above method.

[0047] exist Figure 6 In the illustrated embodiment, the electronic device further includes a bus 62 and a communication interface 63 , wherein the processor 61 , the communication interface 63 and the memory 60 are connected via the bus 62 .

[0048] Among them, the memory 60 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 63 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. can be used. The bus 62 can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 62 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0049] The processor 61 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 61. The above processor 61 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor 61 reads the information in the memory and completes the steps of the method of the above embodiment in combination with its hardware.

[0050] An embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the above method. The specific implementation can be found in the aforementioned method embodiment, which will not be repeated here.

[0051] The computer program products of the methods, devices, and electronic devices provided in the embodiments of the present application include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.

[0052] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0053] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0054] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0055] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for removing an image object, characterized in that: The method comprises: Get the image to be processed and the corresponding foreground target mask; Inputting the image to be processed and the foreground object mask into a pre-trained diffusion model; wherein the pre-trained diffusion model includes an autoencoder module and a noise prediction module; The image to be processed is encoded by the encoder in the autoencoder module to obtain an initial latent variable corresponding to the image to be processed; Performing forward diffusion noise adding processing on the initial latent variable to obtain a noisy latent variable; Based on the foreground object mask, the noise prediction module performs a reverse diffusion sampling process on the noisy latent variable to obtain and correct the self-attention map of each time step; Based on the corrected self-attention map of each time step and the self-attention redirection guidance method, the reverse diffusion sampling process is guided to achieve the target removal of the image to be processed.

2. The method according to claim 1, characterized in that The step of performing forward diffusion noise addition processing on the initial latent variable to obtain a noisy latent variable includes: Set the total number of diffusion steps and variance strategy; Calculating a forward diffusion coefficient according to the total number of diffusion steps and the variance strategy; According to the forward diffusion coefficient, the total diffusion steps are added to the initial latent variable to obtain a noisy latent variable.

3. The method according to claim 1, characterized in that The noise prediction module is a U-net network architecture; based on the foreground target mask, the noise prediction module performs a reverse diffusion sampling process on the noisy latent variable to obtain and correct the self-attention map of each time step, including: For each layer of the model at each time step, the following steps are performed: Determine the self-attention similarity matrix corresponding to the current time step of the current layer of the model according to the noisy hidden variable; According to the foreground target mask, the self-attention similarity matrix is ​​modified to obtain a modified foreground target similarity matrix and a modified background similarity matrix; Determine a corrected foreground attention output result and a corrected background attention output result according to the corrected foreground target similarity matrix, the corrected background similarity matrix, and the corresponding Value matrix; The corrected foreground attention output result and the corrected background attention output result are integrated to obtain a corrected self-attention map corresponding to the current time step of the current layer.

4. The method according to claim 3, characterized in that The step of correcting the self-attention similarity matrix according to the foreground target mask to obtain a corrected foreground target similarity matrix and a corrected background similarity matrix comprises: The corrected foreground target similarity matrix and the corrected background similarity matrix are obtained according to the following first specified formula: ； ； ； in, represents the corrected foreground target similarity matrix; represents the corrected background similarity matrix; Indicates infinity; represents the flattened foreground object mask; Represents the pixel set of the foreground target area; Represents a set of pixels in the background area; Represents the model Tier The self-attention similarity matrix corresponding to the time step; The step of determining a modified foreground target attention output result and a modified background attention output result according to the modified foreground target similarity matrix, the modified background similarity matrix, and the corresponding Value matrix comprises: According to the following second specified formula, the corrected foreground target attention output result and the corrected background attention output result are determined: ； ； in, Represents the corrected foreground target attention output result; Represents the corrected background attention output result; Represents the model Tier The Value matrix corresponding to the time step; Represents the corrected foreground target self-attention map; Represents the corrected background self-attention map; softmax() is the normalization function; The step of integrating the corrected foreground target attention output result and the corrected background attention output result to obtain a corrected self-attention map corresponding to the current time step of the current layer includes: According to the third specified formula below, the corrected self-attention map corresponding to the current time step of the current layer is: ； in, Represents the transposed flattened foreground object mask.

5. The method according to claim 1, characterized in that Based on the corrected self-attention map of each time step and the self-attention redirection guidance method, the reverse diffusion sampling process is guided to achieve the step of removing the target of the image to be processed, including: Taking the first time step as the current time step and the noisy latent variable as the current prediction latent variable, perform the following prediction correction fusion steps: The current prediction latent variable of the current time step is predicted through the noise prediction module to obtain the model prediction noise; Based on the corrected self-attention map at the current time step, correcting the model prediction noise to obtain a corrected prediction noise; Calculate the predicted latent variable of the next time step and the noisy latent variable of the next time step in the reverse diffusion sampling process according to the corrected prediction noise and the forward diffusion coefficient; The predicted latent variable in the foreground target mask and the noisy latent variable outside the mask are fused to retain the background information to obtain the fused latent variable, the next time step is used as the current time step, and the fused latent variable is used again as the current predicted latent variable, and the prediction correction fusion step is continued until all time steps are traversed, and the fused latent variable of the last time step is used as the latent variable output result; The latent variable output result is decoded by the decoder in the autoencoder module to obtain an output image after the target is removed.

6. The method according to claim 5, characterized in that The step of correcting the model prediction noise based on the corrected self-attention map at the current time step to obtain the corrected prediction noise includes: According to the following fourth specified formula, the model prediction noise is corrected to obtain the corrected prediction noise: ； in, represents the predicted latent variable at the tth time step in the reverse diffusion sampling process, ; Indicates t model prediction noise at time steps; Remove the guide coefficient for the target; AAS() indicates that the noise prediction module performs the reverse diffusion sampling process on the noisy latent variable to obtain and correct t The entire processing of the self-attention map of time steps; Indicates the corrected t Corrected prediction noise for the time step.

7. The method according to claim 5, characterized in that The step of calculating the predicted latent variable of the next time step and the noisy latent variable of the next time step in the reverse diffusion sampling process according to the corrected prediction noise and the forward diffusion coefficient includes: According to the fifth specified formula below, calculate the predicted latent variable for the next time step: ； in, , which indicates the first t The forward diffusion coefficient at the time step; Indicates i Variance strategy at the moment; Represents the next time step in the reverse diffusion sampling process t Forward diffusion coefficient at -1; express t Corrected prediction noise for the time step; The next time step in the reverse diffusion sampling process t -1 predicted latent variable; According to the sixth specified formula below, calculate the noisy latent variable for the next time step: ； in , Indicates a zero mean matrix and a unit covariance matrix The standard Gaussian distribution of is a random Gaussian noise randomly drawn from it; is the latent variable representation of the input image; Indicates the next time step t Noisy latent variable at -1.

8. An image object removal device, characterized in that: The device comprises: An image mask acquisition module is used to acquire the image to be processed and the corresponding foreground object mask; A model input module, used for inputting the image to be processed and the foreground object mask into a pre-trained diffusion model; wherein the pre-trained diffusion model includes an autoencoder module and a noise prediction module; An encoding module, used for encoding the image to be processed through the encoder in the autoencoder module to obtain a hidden variable corresponding to the image to be processed; A noise adding module, used for performing forward diffusion noise adding processing on the latent variable to obtain a noisy latent variable; A reverse diffusion sampling module is used to perform a reverse diffusion sampling process on the noisy latent variable through the noise prediction module based on the foreground target mask to obtain and correct the self-attention map of each time step; The diffusion guidance module is used to guide the reverse diffusion sampling process based on the corrected self-attention map of each time step and the self-attention redirection guidance method to achieve the target removal of the image to be processed.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method according to any one of claims 1 to 7.

Images