Methods and systems for removing objects in image editing

Abstract

This invention belongs to the field of computer vision and image processing, specifically relating to a method and system for removing objects in image editing. The first aspect of this invention provides a method for removing objects in image editing, comprising the following steps: acquiring an image to be processed and a task instruction; inputting the image to be processed into a variational autoencoder to obtain a latent representation according to the task instruction; inputting the latent representation into a pre-trained diffusion model to obtain a mask for the image to be processed; dilating the mask using two Gaussian blurs with different blur radii to obtain an optimized mask; and mixing the optimized mask with the image to be processed to complete the image editing with object removal. By decoupling the removal and repair problems, this invention significantly improves the removal effect and image quality of the model for object removal tasks.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing, and specifically relates to a method and system for removing objects in image editing. Background Technology

[0002] With the rapid development of the internet and digital technologies, images have gradually become the most widespread form of information dissemination online. Every day, people from all walks of life create countless new images for marketing and promotion, many of which have high reusability and can greatly improve image production efficiency. This has made image editing technology a research hotspot in the field of computer vision. Object removal, as a core subtask of image editing, aims to precisely erase specified objects from an image and fill the target area with visually coherent and semantically reasonable content, thereby obtaining a cleaner image for reuse. With the massive increase in the volume and quality of images, traditional methods based on Fourier transform and pixel block filling are no longer sufficient to meet practical needs. Therefore, generative model-based image editing methods have emerged. These methods aim to utilize the generative model's ability to perceive contextual regions, allowing the area after object removal to blend harmoniously with the background semantics. This has broad application prospects in professional fields such as social media, image editing, and e-commerce. For example, in the photo editing industry, individuals can upload pictures, select passersby, and then remove objects to make the background of their selfies cleaner, thus improving image quality. In the social media industry, users can quickly prepare copy based on existing images through simple editing. In e-commerce, by removing objects from product images, marketing posters can be reused quickly, greatly improving the efficiency of product categorization and the user's shopping experience.

[0003] Current image editing methods based on generative models primarily rely on diffusion models, which generate high-quality restoration results through a progressive denoising process. The core idea is to iteratively optimize noisy images in the latent space to ultimately restore a clean image. However, during the pre-training phase, the datasets and training strategies used to improve the model's perception and understanding of images are not effectively designed to guide the model in generating context-consistent restoration content based on textual or mask cues. Furthermore, collecting large datasets suitable for object removal tasks is impractical. To address this, some researchers have proposed noise intensity-based methods, editing specific regions by controlling the intensity and distribution of noise. Since these methods are often unstable, some researchers have proposed task-cue-based methods. The model learns through multi-task learning, utilizing task cues to adapt to different restoration needs. This approach allows the model to learn the commonalities and differences between different tasks during training. This method is similar to cue learning in natural language processing but is applied to the field of image editing.

[0004] While existing diffusion-based object removal methods have made significant progress in image inpainting, they still face several limitations in practical applications. Firstly, the training task of the diffusion model itself is not geared towards removal, leading to incomplete removal and poor background consistency. Existing methods struggle to achieve high-fidelity object removal and fail to restore the true texture structure of the background when dealing with complex semantic scenes such as text overlay, perspective distortion, or dense occlusion. For scenes with high dynamic range or complex lighting, the restored area is prone to color casts or inconsistencies in brightness. Furthermore, when the target object has a strong correlation with the background, such as shadows or reflections, existing methods often only remove the object itself, ignoring its correlation effects, resulting in a disconnect between the restored area and its surroundings. To reduce data requirements, existing methods typically use small datasets to fine-tune pre-trained models. However, the distribution difference between such data and pre-trained data can cause significant model bias, leading to artifacts, edge blending failures, and catastrophic forgetting. Finally, some existing methods rely on precise text guidance, using text-specific masks to eliminate regions. While novel, these methods are not practical and require a high level of user expertise. The system requires detailed text instructions from the user, but ordinary users often struggle to generate effective prompts. When instructions and image content are ambiguous, the model may only partially perform the task. Insufficient cross-modal robustness leads to failures in semantic alignment between text and image. When undescribed distracting elements exist in the image, the model may incorrectly remove non-target regions or ignore key targets. Summary of the Invention

[0005] Image editing is a broad field, and object removal is a key subtask; however, currently there are no models specifically focused on object removal. Most image editing models use mixed datasets for multi-task training. Therefore, this invention proposes using a small-scale, specific dataset tailored to the object removal task. This method mainly consists of preprocessing, coarse-grained object removal, fine-grained object inpainting, and post-processing.

[0006] To address the inherent shortcomings of existing object removal methods based on diffusion models in terms of effectiveness, efficiency, and generalization, this invention proposes a method for removing objects in image editing based on an innovative two-stage framework.

[0007] The first aspect of the present invention provides a method for removing objects in image editing, the steps of which include:

[0008] Acquire the image to be processed and the task instructions;

[0009] According to the task instructions, the image to be processed is input into a variational autoencoder to obtain a latent representation;

[0010] The latent representation is input into a pre-trained diffusion model to obtain a mask for the image to be processed;

[0011] The mask is dilated by two Gaussian blurring operations with different blur radii to obtain an optimized mask.

[0012] The optimized mask and the image to be processed are blended to complete the image editing for removing the object.

[0013] Furthermore, the task instructions include a specified masked area and fixed text prompts.

[0014] Furthermore, in response to the specified mask area being smaller than a set threshold, the image to be processed is cropped and input into a variational autoencoder.

[0015] Furthermore, the pre-trained diffusion model is an SD-Inpainting model, which is fine-tuned using a dynamic loss function.

[0016] Furthermore, the dataset used for fine-tuning includes solid color RGB images.

[0017] Furthermore, the pre-trained diffusion model employs a classifier-guided strategy during the inference process.

[0018] Furthermore, the two Gaussian blurs with different blur radii include:

[0019] The first Gaussian blur uses a larger blur radius for initial smoothing;

[0020] The second Gaussian blur uses a smaller blur radius to refine the edges.

[0021] A second aspect of the present invention provides a system for removing objects in image editing, comprising:

[0022] The data acquisition module is used to acquire the images to be processed and task instructions;

[0023] The instruction processing module is used to input the image to be processed into the variational autoencoder to obtain a latent representation according to the task instruction;

[0024] An image input module is used to input the latent representation into a pre-trained diffusion model to obtain a mask for the image to be processed;

[0025] The Gaussian blur module is used to dilate the mask by applying two Gaussian blurs with different blur radii to obtain an optimized mask.

[0026] The mask blending module is used to blend the optimized mask with the image to be processed, thereby completing the image editing for removing the object.

[0027] A third aspect of the present invention provides an electronic device including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the methods described above.

[0028] A fourth aspect of the present invention provides a storage medium for storing a computer program, which, when executed by a computer, implements the above-described method.

[0029] The beneficial effects of this invention are as follows:

[0030] This invention addresses the shortcomings of existing object removal methods by proposing a novel solution aimed at improving the accuracy of object removal and the quality of background restoration, especially when large-scale, high-quality datasets are difficult to obtain. By decoupling the removal and restoration problems, this invention significantly improves the model's removal performance and image quality for object removal tasks.

[0031] Specifically, the effects of this invention are manifested in the following aspects:

[0032] 1. Accuracy and Background Restoration Quality in Object Removal Tasks: Through a two-stage model decoupling process, this invention significantly improves both removal and restoration performance without relying on large-scale datasets or extensive training. It combines different denoising intensities and inference steps in the two-stage diffusion models. The object removal task is decoupled into two independent stages: "coarse-grained erasure" and "fine-grained restoration." The first stage focuses on thoroughly removing the target object, suppressing generation tendencies by fine-tuning the pre-trained model. The second stage utilizes the general restoration capabilities of the original model to restore background texture and semantic consistency. This design avoids a single model simultaneously handling conflicting objectives—removal and generation—significantly improving task focus.

[0033] 2. Enhanced Image Blending Effect: The core of the blurring process lies in generating a smooth transition effect through two progressive Gaussian blurring steps and mask dilation, which is then applied to image blending and seamless cloning. This method not only improves the visual quality of the image but also effectively avoids problems such as boundary artifacts. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the two-stage decoupling framework of the method of the present invention.

[0035] Figure 2 This is a schematic diagram of the fine-tuning method proposed in this invention. Detailed Implementation

[0036] The present invention will now be described in further detail with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0037] The first aspect of this embodiment provides a training process for a method of removing objects in image editing, including three steps: preprocessing, model training, and postprocessing.

[0038] The method of this invention employs a diffusion model. To more clearly illustrate the high-fidelity object removal method using a two-stage diffusion model in this invention, the framework is as follows: Figure 1 As shown in the accompanying drawings, the following description will be provided in conjunction with specific implementation details. This embodiment uses a small dataset. The removal model P is fine-tuned, and a test image I is used for preprocessing. This preprocessing is then performed through a two-stage decoupling framework, and finally, the original image is restored using two progressive Gaussian blurs. The dataset contains N samples, each containing three images: K=1 represents the original image, K=2 represents the removed ground truth, and K=3 represents the grayscale image of the masked region. This invention employs an innovative task decoupling strategy, dividing the object removal task into two independent but collaborative sub-tasks: the first stage focuses on completely removing the target object, while the second stage aims to seamlessly restore the background to achieve natural integration with surrounding content. By fine-tuning the pre-trained diffusion model, it focuses on object removal in the first stage, while the native model performs high-quality background restoration in the second stage. This design not only reduces the performance limitations of a single model in complex scenes but also significantly improves the visual consistency and semantic rationality of the generated images.

[0039] Step 1. Preprocessing. Use a small dataset. This dataset contains N samples, each containing 3 images. When K=1, it represents the original image; when K=2, it represents the removed ground truth; and when K=3, it represents the grayscale image of the masked region. To ensure that the color of the restored image is similar to that of the restored image, the dataset includes a large number of solid color RGB images to enhance the model's color perception ability during subsequent training and testing.

[0040] Because the diffusion model has strict requirements on input size, and direct scaling can cause irreversible damage to image quality, some degree of data preprocessing is essential. When the area of ​​the mask region is less than 30% of a set threshold of 512x512, cropping is performed. The width and height of the current cropped region are calculated; if the left or top side lacks sufficient fill, it expands to the right or downwards; if the right or bottom side lacks sufficient fill, it expands to the left or upwards. This ensures that the cropped region both includes the target region and meets the model's input size requirements. Through cropping and padding, the distortion of original image pixel values ​​caused by scaling is minimized, providing a high-quality base image for subsequent model processing.

[0041] Step 2. Model Training. First, save the embedded text "Remove it" in pt format, omitting the repeated loading of the text encoder. Bucketing is performed based on the proportion of different mask areas.

[0042] like Figure 2 As shown, different weighted mask L2 loss ratios are used for training and fine-tuning. The loss weights are dynamically adjusted during training fine-tuning. Specifically, let the area of ​​the mask region be 1 / 2. Based on the L2 loss formula mentioned above, the normalized dynamic loss function is constructed according to the following steps: First, separate the loss of the masked and unmasked regions.

[0043] Masking area loss:

[0044] Loss in non-masked areas:

[0045] Calculate the normalized weights, with the mask region weights being... The weight of the non-masked region is

[0046] Obviously, The normalization requirement is satisfied, and the final dynamic loss function is:

[0047]

[0048] Fine-tuning the SD-Inpainting model involved using a classifier-guided strategy during inference to adjust its behavior in the denoising process, significantly reducing the likelihood of generating new objects. Specifically, this was done as follows: This allows it to focus on erasing objects within the target area. Where z t Let τ represent the latent representation of the original image at time step t with added noise, where ∈ represents noise, m represents the binary mask of the target removal region, t represents the time step, y represents the fixed text prompt "Remove it", and τ represents the noisy representation of the original image at time step t. θ (·) denotes the CLIP text encoder. The removal phase involves 25 denoising steps, and the repair phase involves 20 denoising steps (intensity set to 0.5). For each sample, the ground truth (without objects), the single-channel black and white mask, and the original image with the objects masked are fed into the VAE encoder to obtain 4-channel, 1-channel, and 4-channel outputs, which are then concatenated and fed into the model. The original text is fused with image features through a cross-attention mechanism to predict the current noise.

[0049] A well-pre-trained model can significantly reduce the resources required for training. SD-Inpainting achieves a good balance between model size and performance, so this invention chooses it as the base model. However, because the model is fed too much cluttered data during pre-training, its removal performance is unsatisfactory, and images processed by it often exhibit artifacts and distortions. In this invention, by utilizing the model's catastrophic forgetting mechanism to a certain extent, the model is made to overfit a small-scale object removal dataset, greatly reducing the probability of incomplete removal, artifact distortions, and replacement of new objects in the model's output images when performing object removal tasks.

[0050] The weighted loss function for the masked region ensures that the model equally considers the restoration quality of both masked and unmasked regions. This strategy effectively suppresses artifact generation, improves edge blending naturalness, and reduces the number of training epochs required. This invention fixes the text embedding during the inference phase, thus omitting the text encoder loading process and significantly saving GPU memory and inference time. Furthermore, different inference steps and denoising intensities are set for the two-stage diffusion model: the first stage uses 25 inference steps to ensure object removal effectiveness, while the second stage uses 20 inference steps with a denoising intensity of 0.5 to achieve the optimal balance between object removal and background restoration. This efficient inference framework makes this method more practical and scalable in real-world applications.

[0051] Step 3. Post-processing. Dilate the mask to increase its safety margins and prevent overly sharp edges. Apply Gaussian blur. The mask undergoes multiple smoothing processes to gradually reduce hard transitions at the boundaries. The first Gaussian blur (R = 15px) eliminates hard edges, while the second blur (R / 2 = 7px) refines the transition band. Simultaneously, a mask dilation operation (dilation radius 5px) expands the edge processing range, avoiding residue caused by inaccuracies in hand-drawn masks. (Based on X) final = X init *(1-mask) + X output *mask, where X final This refers to the image returned to the user after post-processing. X output This refers to the image obtained after the second stage, X. init This refers to the original input image. The optimized mask blends the cropped area with the original input sample image and the original image, ensuring smooth edges and no obvious boundaries.

[0052] Images processed by diffusion models often experience a certain degree of quality degradation due to lossy compression and decoding of information during the inference process by the encoder and decoder. This invention, based on robust seamless fusion using repeating Gaussian blur, effectively overcomes the poor fusion effect of edited images. Addressing the inaccuracies of hand-drawn masks, such as edge jitter and missing markings, an adaptive progressive Gaussian algorithm is designed to dynamically expand the mask boundaries to semantically coherent regions, avoiding the removal of residual images or excessive erasure, thus resulting in a more consistent and seamless fusion of the edited image.

[0053] Adobe Photoshop's Gaussian Blur filter only supports a single radius setting, and the BlurView library in Android development only provides basic parameter configurations. This single-parameter mode cannot resolve the contradictions in complex edge processing: if the radius is too large, it will lead to excessive blurring of the edges (loss of background texture); if the radius is too small, obvious boundary artifacts (jaws or color cast) will remain. However, this invention solves the problem that a single parameter cannot balance blur intensity and detail preservation through parameter combination optimization (two different radii + mask dilation) and stage collaborative design.

[0054] The second aspect of this embodiment provides a practical use case of the object removal system in image editing of the present invention.

[0055] In the UI interface of the system provided by this invention, the user uploads an image I, uses the drawing pen provided by the UI, and uses the mouse to draw and select the area A where the object to be eliminated exists. The backend will automatically generate a mask image m based on A. The removal model receives I and m and outputs O1. The repair model receives m and O1 and outputs O2.

[0056] Although specific embodiments of the invention have been disclosed for illustrative purposes to aid in understanding and implementing the invention, those skilled in the art will understand that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the invention and the appended claims. Therefore, the invention should not be limited to the content disclosed in the preferred embodiments, and the scope of protection claimed by the invention is defined by the claims.

Claims

1. A method for removing objects in image editing, comprising the following steps: Acquire the image to be processed and the task instructions; In response to the specified mask area in the task instruction being smaller than a set threshold, the image to be processed is cropped; According to the task instructions, the cropped image to be processed is input into the variational autoencoder to obtain the latent representation; The latent representation is input into a pre-trained diffusion model finely tuned by a dynamic loss function to perform a first-stage coarse-grained removal of the target object, resulting in the first-stage removal result. The first-stage removal result is input into the original pre-trained diffusion model to perform a second-stage fine-grained repair on the background, resulting in the second-stage repair result. The specified mask region is dilated and then subjected to two Gaussian blurring processes with different blur radii to obtain an optimized mask. The first Gaussian blur uses a larger blur radius for initial smoothing, while the second Gaussian blur uses a smaller blur radius to refine the edges. The optimized mask, the second-stage repair result, and the image to be processed are mixed to complete the image editing of the removed object.

2. The method according to claim 1, characterized in that, The task instructions include a specified masked area and a fixed text prompt.

3. The method according to claim 1, characterized in that, The pre-trained diffusion model is the SD-Inpainting model.

4. The method according to claim 3, characterized in that, The dataset used for fine-tuning includes solid color RGB images.

5. The method according to claim 3, characterized in that, The pre-trained diffusion model employs a classifier-guided strategy during the inference process.

6. A system for removing objects in image editing, comprising: The data acquisition module is used to acquire the images to be processed and task instructions; In response to the specified mask area in the task instruction being smaller than a set threshold, the image to be processed is cropped; The instruction processing module is used to input the cropped image to be processed into the variational autoencoder to obtain the latent representation according to the task instruction; The image input module is used to input the latent representation into a pre-trained diffusion model finely tuned by a dynamic loss function, and to perform a first-stage coarse-grained removal of the target object to obtain the first-stage removal result. The first-stage removal result is input into the original pre-trained diffusion model to perform a second-stage fine-grained repair on the background, resulting in the second-stage repair result. The Gaussian blur module is used to dilate the specified mask region and perform two Gaussian blur processes with different blur radii to obtain an optimized mask; wherein, the first Gaussian blur uses a larger blur radius for initial smoothing; the second Gaussian blur uses a smaller blur radius for edge refinement; The mask blending module is used to blend the optimized mask, the second-stage repair result, and the image to be processed to complete the image editing of the removed object.

7. An electronic device comprising a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 5.

8. A storage medium storing a computer program, which, when executed by a computer, implements the method of any one of claims 1 to 5.

Images