Image elimination and repair method, electronic device, chip system and storage medium

Abstract

The application provides an image erasing and repairing method, an electronic device, a chip system and a storage medium, and relates to the technical field of image processing. In response to a user operation on a first image, a to-be-erased object is determined, the first image and a mask of the to-be-erased object are input into an image erasing model, and after processing, the to-be-erased object is erased and the erased area is repaired as background texture. The image erasing model used in the application can obtain positive text and negative text according to the first image, and perform model inference based on the positive text and the negative text to regenerate image features in the erased area. The model inference tends to generate background pixel information corresponding to the positive text in the erased area and suppress the generation of foreground object pixel information corresponding to the negative text. Therefore, after the model inference, no other portrait or object image and the like is generated in the erased area, thereby improving the effect of image erasing and repairing.

Description

[0001] This application claims priority to Chinese Patent Application No. 202410858225.4, filed with the State Intellectual Property Office of China on June 27, 2024, entitled "Image Processing Method, Electronic Device, Chip System and Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of image processing technology, and in particular to an image removal and restoration method, electronic device, chip system and storage medium. Background Technology

[0003] When users take photos with their mobile phones, passersby or other objects may appear in the photos. In such cases, users often want to remove the people or objects from the photos, which can be achieved using image removal and restoration models. However, in some situations, after removing the person or object, other people or objects may be generated at the removal location, failing to meet the user's needs and degrading the user experience. Summary of the Invention

[0004] This application provides an image removal and restoration method, electronic device, chip system, and storage medium, which solves the problem of removing a person's image and then regenerating a deformed image in a multi-person scene, thus improving the image removal and restoration experience.

[0005] In a first aspect, this application provides an image removal and restoration method, the method comprising: responding to a user's first operation on a first image, determining an object to be removed in the first image; obtaining a first mask image based on the first image, wherein the mask in the first mask image is a mask of the object to be removed; inputting the first image and the first mask image into a first image removal model, performing image processing through the first image removal model, and outputting a second image; wherein the second image does not include the object to be removed and the area after the object to be removed is restored to a background texture. The step of performing image processing through the first image removal model includes: processing the first image based on the first mask image to obtain a first incomplete image, in which the object to be removed is removed; obtaining positive text and negative text based on the first image, wherein the positive text is used to enhance the generation of background texture and the negative text is used to suppress the generation of foreground object texture; performing model inference based on the positive text and negative text, such that the removed area in the first incomplete image is restored to a background texture.

[0006] The image removal and restoration method provided in this application responds to a user's operation on a first image, determines the object to be removed in the first image, and inputs the first image and a first mask image into an image removal model for processing. After processing, the object to be removed in the image is removed, and the removed area is restored to background texture. This application improves the image removal model, which can obtain positive and negative text from the first image and perform model inference based on the positive and negative text to regenerate image features in the removed area. Specifically, the model inference tends to generate background pixel information corresponding to positive text in the removed area while suppressing the generation of foreground object pixel information corresponding to negative text. Therefore, after model inference, no other foreground object images such as portraits or objects are generated in the removed area, thereby improving the image removal and restoration effect.

[0007] In some scenarios, multiple human figures exist around the area to be eliminated. In such cases, if model inference is performed based on the images surrounding the area to be eliminated, a human figure may be regenerated in the eliminated area, resulting in poor restoration effects. By using the image elimination and restoration method provided in this application, the selected object can be accurately eliminated without generating other human figures or objects in the elimination area.

[0008] It's important to note that after the object to be removed is eliminated, other images or objects may be generated in the removed area. This stems from the image removal and restoration mechanism of the image removal model. The image removal model consists of two stages: first, the user-selected object is removed from the image, and then the image is regenerated in the removed area (i.e., image restoration). Regarding the principle of image restoration, the image removal model infers from the image surrounding the removal area and then regenerates the image in the removed area. The goal of the image removal model is to regenerate the background texture in the removed area.

[0009] In the embodiments of this application, "eliminate" can also be replaced with "erase," "wipe," "remove," or other terms that are the same as or similar to "eliminate." The mask image can also be called a mask image, a masking image, or a masking pattern image, and the mask region can also be called a mask region or a masking pattern region.

[0010] In this embodiment, the first operation can be a selection operation or a smearing operation. For example, the first operation can be a user sliding their finger around the image to be eliminated or dragging the elimination cursor on the first image, thereby forming a selected area of ​​regular or irregular shape. The selected area can be a regular shape such as a rectangle or an ellipse, or it can be an irregular shape, depending on the actual use case.

[0011] In some possible implementations, in response to a user's selection operation on the first image, the closed region formed by the movement trajectory of the selection operation is determined as the selected region; or, in response to a user's smearing operation on the first image, the smeared region is determined as the selected region.

[0012] With this solution, users can directly select an area on a photo and remove the image of a person or object in that area with one click. The background texture is also restored in the removed area, improving the removal and repair effect.

[0013] This application provides the following two user scenarios:

[0014] User Scenario 1: In response to user input, the electronic device enters the gallery or photo album interface. The gallery interface displays the first image. After the user clicks the edit option in the gallery interface, the electronic device displays options such as doodle and AI removal. When the AI ​​removal option is selected, in response to the user's selection or smearing action on a certain area of ​​the first image, the electronic device removes the object (e.g., a person or object) in the selected area and performs restoration processing on the removed area to ensure that the removed area is consistent with the background image, thus improving the image removal effect.

[0015] User Scenario 2: With the camera app enabled, in response to the user clicking the shutter button on the shooting interface, the electronic device captures a first image and displays a thumbnail of the first image on the shooting interface. In response to the user's interaction with the thumbnail, the electronic device navigates to the gallery interface, where the first image is displayed. After the user clicks the edit option in the gallery interface, the electronic device displays options such as drawing and AI removal. When the AI ​​removal option is selected, in response to the user's selection or drawing action on a specific area of ​​the first image, the electronic device removes objects (such as people or objects) from the selected area and performs retouching processing on the removed area to ensure that the removed area remains consistent with the background image, improving the image removal effect.

[0016] In some possible implementations, the above-mentioned model inference based on the positive text and the negative text includes:

[0017] Based on the following equation, using a classifier-free approach, the predicted image noise value is calculated based on the positive and negative text:

[0018] Npred=Nneg+α*(Npos-Nneg),

[0019] Wherein, Npos represents the background noise prediction value obtained based on the positive text prediction, Nneg represents the foreground noise prediction value obtained based on the negative text prediction, α is a hyperparameter, and Npred represents the image noise prediction value obtained by the first image cancellation model at each time step.

[0020] Based on the image noise prediction value Npred, multiple noise sampling is performed to obtain a multi-noise sampled image;

[0021] The second image is obtained based on the multi-noise sampled image.

[0022] The above approach allows for the use of a classifier-free method, based on positive and negative text, to guide the model in generating background information, suppressing the regeneration of foreground objects and thus improving the image removal and restoration effect.

[0023] In some possible implementations, obtaining positive and negative text based on the first image includes: performing panoramic segmentation on the first image to obtain background category information and foreground category information; generating positive text based on the background category information and background text, and generating negative text based on the foreground category information and foreground text; wherein, the positive text is used to enhance the background texture corresponding to the background category information, and the negative text is used to suppress the foreground object texture corresponding to the foreground category information.

[0024] The background text and foreground text are the texts learned by the first image removal model during training.

[0025] The above approach allows for model inference based on positive and negative text. The model inference tends to generate background pixel information corresponding to positive text in the eliminated area, while suppressing the generation of foreground object pixel information corresponding to negative text. Therefore, through model inference, no other foreground object images such as portraits or objects will be generated in the eliminated area, thereby improving the image elimination and restoration effect.

[0026] In some possible implementations, before inputting the first image and the first mask image into the first image elimination model, the method further includes: performing panoramic segmentation processing on the image samples to obtain a panoramic segmentation result, the panoramic segmentation result including a background region and multiple foreground instance regions; randomly selecting a sub-region of the background region as a background sample mask and generating background text corresponding to the background sample mask; using the background sample mask and the background text as a training set and the sample image as a supervision signal to train the model; and randomly selecting an instance region of the multiple foreground instance regions as a foreground sample mask and generating foreground text corresponding to the foreground sample mask; using the foreground sample mask and the foreground text as a training set and the sample image as a supervision signal to train the model.

[0027] The random selection of the background sample mask can be achieved by: selecting a region within the background area as the background sample mask; or selecting a region within the overlapping area of ​​the foreground instance and the background area, and using the background area within the selected region as the background sample mask. In this selected region, the area of ​​the background sample mask is larger than the area of ​​the foreground instance.

[0028] This application's solution trains the image removal model using sample images, background sample masks, and background text, and also trains it using sample images, foreground sample masks, and foreground text, enabling the image removal model to learn background and foreground text. This way, when applying the image removal model to image processing, it tends to generate background pixel information corresponding to positive text while suppressing the generation of foreground object pixel information corresponding to negative text. This prevents the generation of other foreground object images, such as portraits or objects, in the removed area, thereby improving the image removal and restoration effect.

[0029] It should be noted that in this embodiment, the input image is decomposed into foreground object regions and background regions. When the mask M covers the foreground object region (i.e., mask M is a foreground mask), foreground text input (denoted as Obj) is added for training. When the mask M covers the background region (i.e., mask M is a background mask), background text input (denoted as Ctxt) is added for training. In this way, the model can be guided to generate background information during the model inference stage through classifier-free guidance (CFG), thus suppressing the problem of foreground object regeneration.

[0030] In some possible implementations, before processing the first image based on the first mask image, the method further includes: identifying a shadow region in the first image surrounding the object to be eliminated; and performing dilation processing on the mask in the first mask image, wherein the dilated mask includes the shadow region.

[0031] In this case, the above-mentioned processing of the first image based on the first mask image to obtain the first incomplete image, in which the object to be eliminated is eliminated, includes: processing the first image based on the mask after the dilation process to obtain the first incomplete image, in which the object to be eliminated and the shadow area are eliminated.

[0032] Experimental tests revealed that when there is a shadow area around the object to be removed in the first image, the shadow area will affect the image restoration to a certain extent. If the mask of the object to be removed does not include the shadow area, other foreground objects such as portraits or objects may be generated in the removed area.

[0033] The above-described scheme of this application performs dilation processing on the mask of the object to be eliminated, so that the mask after dilation includes the shadow area. This can eliminate both the object to be eliminated and the shadow area. Then, when performing image restoration on the eliminated area, the shadow area can be avoided from affecting the image restoration, thereby avoiding the generation of other foreground object images such as portraits or objects in the eliminated area. Therefore, the effect of image elimination and restoration can be improved.

[0034] In some possible implementations, the dilation process on the mask in the first mask image includes: performing panoramic segmentation on the first image to obtain a foreground object mask and a background mask; performing a first dilation process on the initial mask in the first mask image to obtain a first mask; finding the intersection of the first mask and the background mask to obtain a second mask; finding the difference between the second mask and the initial mask to obtain a ring-shaped third mask; performing pixel clustering on the third mask to divide the region where the third mask is located into multiple pixel blocks; comparing the average brightness of each pixel block in the multiple pixel blocks with the average brightness of the first image to obtain a processed third mask, wherein the average brightness of the pixel blocks contained in the processed third mask is less than the average brightness of the first image; and summing the processed third mask and the initial mask to obtain a dilated mask.

[0035] In this application, when there is a shadow area around the object to be eliminated, the initial mask is first inflated using a preset inflated kernel size. During the mask inflatation process, the inflated mask cannot cover the foreground object area. The intersection of the inflated mask and the background mask is calculated, and the intersection result (i.e., the second mask) does not include other foreground objects besides the object to be eliminated. This avoids the accidental deletion of foreground objects.

[0036] Then, an inflated mask (a ring-shaped third mask) is determined around the outer edge of the mask for the object to be eliminated. Based on brightness contrast, a suitable inflated mask is selected. For example, if the average brightness of a pixel block is greater than or equal to the average brightness of the image, the pixel block is not a shadow area and does not meet the requirement; if the average brightness of a pixel block is less than the average brightness of the image, the pixel block is a shadow area and meets the requirement. The suitable inflated mask, together with the initial mask, constitutes the final inflated mask. This embodiment employs adaptive mask expansion based on brightness contrast, ensuring that the inflated mask includes possible shadow areas near the object to be eliminated, thus suppressing the regeneration of foreground objects caused by shadows.

[0037] In some possible implementations, restoring the eliminated region in the first incomplete image to the background texture includes: identifying a foreground object in the first image that has features similar to the object to be eliminated; processing the heatmap of the region where the object to be eliminated is located based on the mask of the foreground object region and a first adjustment coefficient to obtain a first heatmap; wherein the first adjustment coefficient is a value greater than 0 and less than 1; and restoring the eliminated region in the first incomplete image to the background texture based on the first heatmap.

[0038] In some possible implementations, processing the heatmap of the region where the object to be eliminated is located based on the mask of the foreground object region and the first adjustment coefficient to obtain a first heatmap includes: obtaining heatmaps of each region of the first image from the self-attention layer of the first image elimination model; averaging the multiple heatmaps of the region where the object to be eliminated is located (hereinafter referred to as the region to be eliminated) among the heatmaps of each region of the first image to obtain a second heatmap; and multiplying the mask of the foreground object region, the first adjustment coefficient, and the second heatmap to obtain the first heatmap.

[0039] It should be noted that when the value of a pixel in the heatmap decreases, the reference strength of that pixel decreases when the area to be eliminated is restored to the background texture using the image elimination model.

[0040] Experimental tests revealed that when there are foreground objects in the first image that have similar features to the object to be eliminated, these similar foreground objects can affect image restoration to some extent. If the reference intensity of similar foreground objects is too high during image restoration, it is very likely that such foreground objects with similar features will be generated in the already eliminated area.

[0041] The above-mentioned scheme of this application can be used to suppress self-attention layers based on panoramic segmentation. By processing the heatmap of the region to be eliminated by algorithm, the reference intensity of the heatmap of the region to be eliminated to other foreground objects is reduced. That is, when the diffusion model generates the background image of the eliminated region, it will reduce the reference to other foreground object regions, thereby reducing the phenomenon of random generation by the diffusion model.

[0042] In some possible implementations, the first image removal model is an image removal model based on stable diffusion SD.

[0043] In the image removal and restoration processes of various scenarios, this application employs specific image removal models for image processing. It should be noted that this application not only proposes different image removal strategies for different scenarios and improves image processing strategies to enhance image removal and restoration effects—for example, for objects to be removed with different area proportions, an image processing strategy corresponding to the area proportion can be selected for removal and restoration, resulting in different image processing effects—this application also improves the image removal model used, further enhancing the image removal and restoration effects.

[0044] Secondly, this application provides an image removal and restoration method, the method comprising: in response to a user's first operation on a first image, determining an object to be removed in the first image; obtaining a first mask image based on the first image, wherein the mask in the first mask image is the mask of the object to be removed; inputting the first image and the first mask image into a first image removal model, performing image processing through the first image removal model, and outputting a second image; wherein the second image does not include the object to be removed and the area after the object to be removed is restored to a background texture;

[0045] The image processing using the first image elimination model includes: identifying a shadow region around the object to be eliminated in the first image; performing dilation processing on the mask in the first mask image, wherein the dilated mask includes the shadow region; processing the first image based on the dilated mask to obtain a first incomplete image, wherein the object to be eliminated and the shadow region are eliminated in the first incomplete image; and performing model inference using the first image elimination model to restore the eliminated region in the first incomplete image to the background texture.

[0046] In some possible implementations, the dilation process on the mask in the first mask image includes: performing panoramic segmentation on the first image to obtain a foreground object mask and a background mask; performing a first dilation process on the initial mask in the first mask image to obtain a first mask; finding the intersection of the first mask and the background mask to obtain a second mask; finding the difference between the second mask and the initial mask to obtain a ring-shaped third mask; performing pixel clustering on the third mask to divide the region where the third mask is located into multiple pixel blocks; comparing the average brightness of each pixel block in the multiple pixel blocks with the average brightness of the first image to obtain a processed third mask, wherein the average brightness of the pixel blocks contained in the processed third mask is less than the average brightness of the first image; and summing the processed third mask and the initial mask to obtain a dilated mask.

[0047] Thirdly, this application provides an image removal and restoration method, the method comprising: in response to a user's first operation on a first image, determining an object to be removed in the first image; obtaining a first mask image based on the first image, wherein the mask in the first mask image is the mask of the object to be removed; inputting the first image and the first mask image into a first image removal model, performing image processing through the first image removal model, and outputting a second image; wherein the second image does not include the object to be removed and the area after the object to be removed is restored to a background texture;

[0048] The image processing using the first image elimination model includes: processing the first image based on the first mask image to obtain a first incomplete image, in which the object to be eliminated is eliminated; identifying a foreground object in the first image that has similar features to the object to be eliminated; processing the heatmap of the region where the object to be eliminated is located based on the mask of the foreground object region and a first adjustment coefficient to obtain a first heatmap; wherein the first adjustment coefficient is a value greater than 0 and less than 1; and restoring the eliminated region in the first incomplete image to the background texture based on the first heatmap.

[0049] In some possible implementations, processing the heatmap of the region where the object to be eliminated is located based on the mask of the foreground object region and the first adjustment coefficient to obtain a first heatmap includes: obtaining heatmaps of each region of the first image from the self-attention layer of the first image elimination model; averaging the multiple heatmaps of the region where the object to be eliminated is located among the heatmaps of each region of the first image to obtain a second heatmap; and multiplying the mask of the foreground object region, the first adjustment coefficient, and the second heatmap to obtain the first heatmap.

[0050] Fourthly, this application provides an image removal and restoration apparatus, which includes units for performing the method described in the first aspect above. This apparatus can correspond to performing the method described in the first aspect above; for a detailed description of the units within the apparatus, please refer to the description in the first aspect above, and for the sake of brevity, will not be repeated here.

[0051] The method described in the first aspect above can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a processing module or unit, a display module or unit, etc.

[0052] Fifthly, this application provides an electronic device, the electronic device including a processor, a processor and a computer program or instructions stored in a memory, the processor being used to execute the computer program or instructions, causing the method in the first aspect to be performed.

[0053] In a sixth aspect, this application provides a computer-readable storage medium having a computer program (also referred to as instructions or code) stored thereon for implementing the method of the first aspect. For example, when the computer program is executed by a computer, it enables the computer to perform the method of the first aspect.

[0054] In a seventh aspect, this application provides a chip including a processor. The processor is used to read and execute a computer program stored in a memory to perform the methods in the first aspect and any possible implementation thereof. Optionally, the chip further includes a memory connected to the processor via a circuit or wire.

[0055] Eighthly, this application provides a chip system including a processor. The processor is used to read and execute a computer program stored in a memory to perform the methods in the first aspect and any possible implementation thereof. Optionally, the chip system further includes a memory connected to the processor via a circuit or wire.

[0056] Ninthly, this application provides a computer program product comprising a computer program (also referred to as instructions or code), which, when executed by an electronic device, causes the electronic device to implement the method of the first aspect.

[0057] It is understood that the beneficial effects of aspects two through nine above can be found in the relevant descriptions in aspect one above, and will not be repeated here. Attached Figure Description

[0058] Figure 1 A schematic diagram of the interface for generating deformed human images after image removal and restoration in related technologies;

[0059] Figure 2A This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0060] Figure 2B A schematic diagram of the software architecture of an electronic device provided in an embodiment of this application;

[0061] Figure 3 A schematic diagram of a user interface scenario for the image removal and restoration method provided in the embodiments of this application;

[0062] Figures 4A to 4C These are schematic diagrams of user interface scenarios for the image removal and restoration methods provided in the embodiments of this application;

[0063] Figure 5 A schematic flowchart illustrating the image removal and restoration method provided in this application embodiment;

[0064] Figure 6 This is a schematic diagram illustrating an application scenario of the image removal and restoration method provided in the embodiments of this application;

[0065] Figure 7 This is a schematic diagram of image cropping processing in the image removal and restoration method provided in the embodiments of this application;

[0066] Figure 8 This is a schematic diagram illustrating the cropping, scaling, and mask dilation processes in the image removal and restoration method provided in the embodiments of this application.

[0067] Figure 9 This is another schematic flowchart of the image removal and restoration method provided in the embodiments of this application;

[0068] Figure 10 This application scenario illustrates the application scenario of the image removal and restoration method provided in the embodiments of this application. Figure 1 ;

[0069] Figure 11 This is a flowchart illustrating the process of determining whether an image removal and restoration method has a complex background, as provided in the embodiments of this application.

[0070] Figure 12 Schematic diagram 2 illustrating the application scenario of the image removal and restoration method provided in this application embodiment;

[0071] Figure 13A This diagram illustrates the image removal and restoration method provided in this application, which involves judging the background of a large proportion of objects and implementing different image processing strategies accordingly.

[0072] Figure 13B This illustration shows a non-human-background image removal scenario for the image removal and restoration method provided in the embodiments of this application. Figure 3 ;

[0073] Figure 13C Schematic diagram four illustrating the image removal scenario of a crowd background for the image removal and restoration method provided in this application embodiment;

[0074] Figure 13D An improved schematic diagram of an image removal and restoration method for a crowd background scenario provided in the embodiments of this application;

[0075] Figure 14 This is a flowchart illustrating the process of determining whether the image removal and restoration method provided in this application is a scene with a crowd background.

[0076] Figures 15A to 15D A schematic diagram illustrating the process of the foreground and background decomposition training and inference method based on panoramic segmentation provided in the embodiments of this application;

[0077] Figures 16A to 16C A schematic diagram illustrating the adaptive mask expansion process based on brightness contrast provided in an embodiment of this application;

[0078] Figures 17A to 17F A schematic diagram illustrating the process of attention layer suppression based on panoramic segmentation provided in this application embodiment;

[0079] Figure 18 This is another schematic diagram of the image removal and restoration method provided in the embodiments of this application;

[0080] Figure 19 This is a schematic diagram illustrating the acquisition of the second mask image in the image removal and restoration method provided in the embodiments of this application;

[0081] Figure 20 This is a flowchart illustrating the process of determining whether a selected or painted area meets the elimination conditions in the image removal and repair method provided in this application embodiment.

[0082] Figure 21A This is a schematic diagram illustrating a scenario in the image removal and restoration method provided in this application, where the selected or painted area is determined to meet the removal conditions.

[0083] Figure 21B This is a schematic diagram illustrating a scenario in the image removal and repair method provided in this application, where the selected or painted area does not meet the removal conditions.

[0084] Figure 21C This is a schematic diagram illustrating a scenario in the image removal and repair method provided in this application, where the selected or painted area is determined to meet the removal conditions.

[0085] Figure 22This is a flowchart illustrating the process of determining whether a selected or painted area meets the elimination conditions in the image removal and repair method provided in this application embodiment.

[0086] Figure 23A and Figure 23B A schematic diagram of the user interface for the image removal and restoration method provided in the embodiments of this application;

[0087] Figure 24 This is a schematic diagram of the structure of an image removal and repair device provided in an embodiment of this application. Detailed Implementation

[0088] In the embodiments of this application, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "multiple" means two or more.

[0089] To facilitate understanding of the embodiments of this application, the relevant concepts involved in the embodiments of this application will be briefly explained first.

[0090] 1. GAN-based image inpainting network model

[0091] Image inpainting network models based on GANs can be simply referred to as GAN image removal models. GAN image removal models are a technique for image inpainting using generative adversarial networks (GANs), which can repair and complete specified regions (masked regions) in an image. GAN image removal models can be deep learning network models trained through iterative adversarial training of a generator network and a discriminator network. The training samples consist of a large number of images (millions of images) and randomly generated removal regions (masked regions), with the training objective being to make the inpainted image as close as possible to restoring the original image.

[0092] 2. SD-based image inpainting network model

[0093] The image inpainting network model based on SD (Stable Diffusion) can be simply referred to as the SD image removal model. The SD image removal model is a method based on a stable diffusion model. This method has a high number of parameters (approximately 1GB) and a large computational cost. Because this method is trained on large-scale image-text data (5 billion image-text pairs), it has stronger image-text matching and background generation capabilities, thus enabling it to handle the removal of large areas and complex backgrounds. The SD image removal model can include an encoder network, a denoising UNet network, and a decoder network.

[0094] Currently, when users take photos outdoors with their mobile phones, passersby may appear in the photos. In such cases, users usually want to remove the images of passersby from the photos. Figure 1 As shown, in some cases, the area of ​​the passerby image to be removed in the photo is obscured by other people's image areas or is relatively close to each other. In this case, if the user uses the removal and repair function to remove the passerby image, a deformed human image may be generated at the removal location after the passerby image is removed. As a result, the removed and repaired photo cannot meet the user's needs and reduces the user experience.

[0095] To address the aforementioned problems, this application provides an image removal and restoration method and electronic device. In this application's embodiments, in response to a user's operation on a first image, the object to be removed in the first image is determined. The first image and a first mask image are input into an image removal model for processing. After processing, the object to be removed from the image is eliminated, and the removed area is restored to background texture. This application's solution improves the image removal model. The image removal model can obtain positive and negative text from the first image and perform model inference based on the positive and negative text to regenerate image features in the removed area. Specifically, the model inference tends to generate background pixel information corresponding to positive text in the removed area while suppressing the generation of foreground object pixel information corresponding to negative text. Therefore, through model inference, no other foreground object images such as portraits or objects are generated in the removed area, thereby improving the image removal and restoration effect.

[0096] The image removal and repair method and electronic device in the embodiments of this application will now be described with reference to the accompanying drawings.

[0097] Figure 2A A hardware system for an electronic device applicable to this application is shown.

[0098] Electronic device 100 can be a mobile phone, smart screen, tablet computer, wearable electronic device, in-vehicle electronic device, augmented reality (AR) device, virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), projector, etc. This application embodiment does not limit the specific type of electronic device 100.

[0099] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0100] It should be noted that, Figure 2A The structure shown does not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include... Figure 2A The components shown may include more or fewer components, or the electronic device 100 may include... Figure 1 The components shown may be a combination of certain components, or the electronic device 100 may include... Figure 2A Sub-components of some of the components shown. Figure 2A The components shown can be implemented in hardware, software, or a combination of both.

[0101] Processor 110 may include one or more processing units. For example, processor 110 may include at least one of the following processing units: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and neural network processing unit (NPU). These different processing units may be independent devices or integrated devices. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0102] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0103] For example, processor 110 can be used to execute the image elimination and restoration method of the embodiments of this application: in response to a user's first operation on a first image, determining the object to be eliminated in the first image; obtaining a first mask image based on the first image, wherein the mask in the first mask image is the mask of the object to be eliminated; inputting the first image and the first mask image into a first image elimination model, performing image processing through the first image elimination model, and outputting a second image; the second image does not include the object to be eliminated and the area after the object to be eliminated is restored to background texture; wherein, the image processing through the first image elimination model includes: processing the first image based on the first mask image to obtain a first incomplete image, in which the object to be eliminated is eliminated; obtaining positive text and negative text based on the first image, wherein the positive text is used to enhance the generation of background texture and the negative text is used to suppress the generation of foreground object texture; performing model inference based on the positive text and negative text, so that the eliminated area in the first incomplete image is restored to background texture.

[0104] Figure 2A The connection relationships between the modules shown are merely illustrative and do not constitute a limitation on the connection relationships between the modules of the electronic device 100. Optionally, the modules of the electronic device 100 may also adopt a combination of various connection methods described in the above embodiments.

[0105] The wireless communication function of electronic device 100 can be realized through devices such as antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor.

[0106] Electronic device 100 can implement display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0107] The display screen 194 can be used to display images or videos. Exemplarily, in an embodiment of this application, the display screen 194 can be used to display a first captured image (the image to be removed and repaired), and to display a second image (the image after removal and repair).

[0108] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0109] The ISP is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the camera to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can perform algorithmic optimization of image noise, brightness, and color. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0110] The camera 193 (also known as a lens) is used to capture still images or videos. It can be activated via application commands to enable photo-taking, such as capturing images of any scene. The camera may include components such as an imaging lens, filters, and an image sensor. Light emitted or reflected by an object enters the imaging lens, passes through the filter, and is finally focused onto the image sensor. The imaging lens is primarily used to focus and image the light emitted or reflected by all objects within the shooting field of view (also known as the scene to be captured, the target scene, or the scene image the user expects to capture). The filter is primarily used to filter out excess light waves (such as infrared light waves other than visible light). The image sensor can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The image sensor is primarily used to perform photoelectric conversion on the received light signal, converting it into an electrical signal, which is then transmitted to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into standard RGB, YUV, and other image signal formats.

[0111] In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0112] The camera 193 can be located in front of the electronic device 100 or on the back of the electronic device 100. The specific number and arrangement of the cameras can be set according to the requirements, and this application does not impose any restrictions.

[0113] A touch sensor 180K, also known as a touch device, can be disposed on a display screen 194. The touch sensor 180K and the display screen 194 together form a touchscreen, also known as a touch display. The touch sensor 180K is used to detect touch operations applied to or near it. The touch sensor 180K can transmit the detected touch operation to an application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K may also be disposed on the surface of the electronic device 100, and in a different location from the display screen 194. Exemplarily, in an embodiment of this application, the touch sensor 180K can be used to detect user-triggered image deletion and restoration operations.

[0114] The hardware system of electronic device 100 has been described in detail above. The software system of electronic device 100 will be introduced below.

[0115] Figure 2BThis is a schematic diagram of the software system of the electronic device 100 provided in this application embodiment. The layered architecture divides the system into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: applications 201, application framework 202, hardware abstraction layer (HAL) 203, kernel 204, and hardware layer 205.

[0116] Application layer 201 may include a series of application packages. In this embodiment, the application packages may include applications such as camera and gallery.

[0117] The application framework layer 202 provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer 202 includes some predefined functions. In this embodiment, the application framework layer may include a camera access interface, wherein the image processing interface may include image removal and restoration services. The image processing interface is used to provide an API and programming framework for the gallery application.

[0118] The hardware abstraction layer 203 is an interface layer located between the application framework layer 202 and the kernel layer 204, providing a virtual hardware platform for the operating system. In this embodiment, the hardware abstraction layer 203 may include a camera hardware abstraction layer and an image processing algorithm library.

[0119] The camera hardware abstraction layer (HAL) can provide virtual hardware for camera devices such as the front and rear cameras (referred to as camera HAL). The image processing algorithm library may include image removal and restoration algorithms; in other words, the image processing algorithm library contains runtime code and data that implement the image removal and restoration methods provided in the embodiments of this application.

[0120] Kernel layer 204 is the layer between hardware and software. It includes drivers for various hardware components. Kernel layer 204 may include camera device drivers, digital signal processor (DSP) drivers, and image processor (IP) drivers, among others. Specifically, the camera device driver drives the camera sensor to acquire images and the image signal processor to preprocess them. The DSP driver drives the DSP to process images. The IIP driver drives the graphics processor to process images.

[0121] Hardware layer 205 includes camera sensors, image signal processors, digital signal processors, and image processors, etc.

[0122] It should be noted that although the embodiments of this application are illustrated using the Android system as an example, the basic principles are equally applicable to electronic devices based on operating systems such as iOS or Windows.

[0123] The image removal and repair method provided in this application can be implemented by the aforementioned electronic device (such as a mobile phone), or by a functional module and / or entity within the electronic device capable of implementing the image removal and repair method. Furthermore, the solution in this application can be implemented through hardware and / or software, the specific implementation depending on actual usage requirements, and this application does not impose any limitations. The following description uses an electronic device as an example, along with the accompanying drawings, to exemplarily illustrate the image removal and repair method provided in this application.

[0124] To better understand the embodiments of this application, the "image removal and repair" function provided in the embodiments of this application will be briefly described below with reference to a mobile phone interface diagram:

[0125] In some embodiments, the electronic device adds an "AI Removal" option to the editing interface of a gallery application (or photo album application). This "AI Removal" option serves as an entry point to trigger the display of image removal and restoration functions. Exemplarily, the image removal and restoration functions may include features such as intelligent selection and manual smearing. It should be noted that the various functions included in the image removal and restoration functions and their names are illustrative examples and are not limited in scope by this application.

[0126] Intelligent selection refers to the process by which electronic devices, in response to a user's swiping action on an image, select targets based on the swiping trajectory.

[0127] Manual smearing refers to the electronic device responding to the user's smearing action on an image and selecting the target based on the smeared area.

[0128] Figure 3 This illustration shows a user interface diagram of the "AI elimination" option in the image elimination and restoration method provided in this application embodiment. Figure 3 As shown in (a), the electronic device displays an icon for the Gallery app on desktop 10. When the user clicks the Gallery app icon, as shown in (a), the user sees the following: Figure 3 As shown in (b), the electronic device updates from displaying the desktop 10 to displaying the all-photos interface 11. When the user clicks on a photo in the all-photos interface 11, as shown in (b), the electronic device updates from displaying the desktop 10 to displaying the all-photos interface 11. Figure 3 As shown in (c), the electronic device updates from displaying all photos interface 11 to displaying a single photo interface 12. The single photo interface includes editing options 13. Figure 3As shown in (d), when editing option 13 is selected, the electronic device updates from displaying a single photo interface 12 to displaying an editing interface 14. Editing interface 14 includes the photo selected by the user and the "AI Removal" option 15. When the "AI Removal" option 15 is selected in editing interface 14, the electronic device displays the functions corresponding to the "AI Removal" option 15, such as intelligent selection and manual smoothing.

[0129] It should be noted that electronic devices can identify the selected area and the object to be removed through functions such as intelligent selection or manual smearing.

[0130] In some embodiments, the object to be eliminated can be a human body, such as a passerby in a photograph.

[0131] In other embodiments, the object to be eliminated can be an item, such as a trash can or clutter in a photograph.

[0132] According to the solution of this application, electronic devices can respond to the user's selection operation on the image, remove the object to be removed in the user-specified area (the specified area) of the image, and repair the removed area to generate a more aesthetically pleasing image.

[0133] Figure 4A This illustration shows a user interface diagram illustrating the image removal and restoration method provided in this application, where image removal and restoration are performed. For example... Figure 4A As shown in (a), the electronic device displays the user-selected photo (i.e., the original image) 16 in the editing interface 14 and displays the functions corresponding to the "AI Removal" option 15 in the function option area, such as intelligent selection and manual smearing. Figure 4A As shown in (a) and (b), when the user selects the "Smart Selection" function option 17, the electronic device can display a trajectory cursor 18 for smart selection in the original image 16. The user can drag the trajectory cursor 18 to select the objects to be eliminated in the original image 16. The user drags the trajectory cursor 18 from the starting point 19 to the ending point 20 and releases it; the selection trajectory is as follows. Figure 4A As shown by the dashed lines in (b) and (c), the selected trajectory can be closed manually by the user or automatically extended and closed by the electronic device. Figure 4A As shown in (d), the selection trajectory is closed, and the area enclosed by this selection trajectory is the area within the dashed box 21. The object to be eliminated is a human body, and the background is flowers and plants. Figure 4A As shown in (e), the image elimination and repair method provided in this application embodiment is used to eliminate and repair the object to be eliminated. The human image in the dashed frame 21 in the original image 16 is eliminated, while the background such as flowers and plants in the dashed frame 21 is still retained. The image after elimination and repair is shown in 22.

[0134] In some embodiments, after the user drags the trajectory cursor 18 from the starting point 19 to the ending point 20 and releases it, the object to be eliminated can be displayed using a dynamic flashing display method.

[0135] Figure 4B This illustration shows another user interface diagram illustrating the image removal and restoration method provided in this application, where image removal and restoration are performed. For example... Figure 4B As shown in (a) to (d), in response to the user's selection operation 23 on the original image, the electronic device recognizes that the object to be eliminated is a human image and meets the elimination conditions. The object to be eliminated 24 is highlighted. The electronic device performs image elimination and repair processing on the object to be eliminated 24 in the background, and then updates the original image. The updated image 25 does not include the object to be eliminated 24, and the eliminated area is repaired as the background image.

[0136] Figure 4C This illustration shows another user interface diagram illustrating the image removal and restoration method provided in this application, where image removal and restoration are performed. For example... Figure 4C As shown in (a) to (f), in response to the user's operation on the manual smear option 26, the electronic device starts the manual smear function; in response to the user's operation on the smear tool 27, the electronic device enables the smear tool 27; in response to the user's smear operation on the object to be eliminated 28 (trash can) in the original image, the electronic device recognizes that the object to be eliminated is not a human image and meets the elimination conditions. The electronic device performs image elimination and repair processing on the smeared area of ​​the object to be eliminated 28 in the background, and then updates and displays the original image. The updated image 29 does not include the object to be eliminated 28, and the eliminated area is repaired to the background image.

[0137] The image removal and repair method provided in this application will be described in detail below with reference to specific embodiments and the accompanying drawings.

[0138] Figure 5 This is a schematic flowchart illustrating an image removal and restoration method provided in an embodiment of this application. The method can be... Figure 1 The electronic device shown executes the method, which includes steps S301 to S304.

[0139] S301. In response to the user's selection operation (circle or smudge operation), identify the object to be eliminated in the first image.

[0140] According to the solution of this application, an electronic device can respond to the user's selection operation in an image (or first image, original image or image to be repaired), remove the portrait or object image selected by the user through circle or smear operation in the image, and repair the removed area to generate a more beautiful image.

[0141] The area circled or drawn by the user is called the selected area, which can include people or objects selected by the user in the image by circling or drawing.

[0142] For example, a selection operation can be an action where a user selects a region in an image by sliding their finger or dragging a cursor; this selection operation can be called smart selection. Accordingly, the smart-selected region can be defined as the selected area. Objects within the selected area are the objects to be eliminated.

[0143] As another example, the selection operation can be a user clicking on a specific area of ​​an image, which can trigger the electronic device to automatically select the object to be eliminated at the clicked location. Accordingly, the automatically selected area can be defined as the selected region.

[0144] For example, the selection operation can be a user's smearing action on a certain area of ​​an image. Accordingly, the area covered by the smearing can be defined as the selected area.

[0145] For ease of explanation, this application uses the example of a smart selection operation for illustration.

[0146] The objects in the selected area can be human bodies or items, etc.

[0147] In some embodiments, in response to a user's intelligent selection operation, a selection trajectory can be marked in the first image, and the selected area can be determined based on the selection trajectory. For example, the area enclosed by the selection trajectory can be determined as the selected area. Optionally, the selection trajectory can be a line trajectory, or it can be a closed rectangle, ellipse, or irregular shape. For ease of explanation, this application embodiment uses a line trajectory as an example for illustrative purposes.

[0148] In some embodiments, when the selected trajectory is closed, the area enclosed by the selected trajectory is defined as the selected area.

[0149] In other embodiments, if the distance between the start and end points of the selected trajectory meets a preset distance condition when the selected trajectory is not closed, the electronic device can automatically connect the start and end points to form a closed area of ​​the selected trajectory, and then use the closed area as the selected area.

[0150] In some embodiments, if the distance between the start and end points of the selected trajectory does not meet a preset distance condition (e.g., the distance between the start and end points is too large) when the selected trajectory is not closed, the electronic device determines that the selected trajectory does not meet the condition and displays a prompt message. For example, the prompt message may be: This selection operation is invalid; please select again.

[0151] S302. Obtain a first mask image based on the first image, the first mask image including a first mask region corresponding to the object to be eliminated.

[0152] The first mask image can be referred to as a mask image or a binary image. For example, the first mask image includes a mask region where pixels are marked as 1, and other regions where pixels are marked as 0.

[0153] Specifically, pixels in the first mask region are marked as 1 and displayed as white. Pixels in other regions of the first mask image, excluding the first mask region, are marked as 0 and displayed as black.

[0154] In some embodiments, the electronic device can determine a selected area based on a selection trajectory and determine the object to be eliminated based on the selected area, and then generate a first mask image. The shape of the mask area in the first mask image is equivalent to the shape of the object to be eliminated.

[0155] In other embodiments, the electronic device can obtain a first mask image by inputting an image marked with a circled trajectory into a mask generation model. The masked region in the first mask image corresponds to the selected region enclosed by the circled trajectory. The shape of the first masked region is similar to the shape of the region enclosed by the circled trajectory.

[0156] S303. Calculate the first area ratio (denoted as P) of the first mask region in the first mask image.

[0157] The area ratio of the first mask region in the first mask image is equal to the area ratio of the object to be removed in the first image.

[0158] In some embodiments, the first area ratio P can be obtained by acquiring the number of pixels in the first mask image and the number of pixels in the first mask region, and by calculating the ratio of the number of pixels in the first mask region to the number of pixels in the first mask image.

[0159] The number of pixels in the first mask image is the sum of the number of all 1 pixels and the number of all 0 pixels.

[0160] The number of pixels in the first mask region is equal to the total number of pixels 1 in the first mask image.

[0161] It can be understood that the area ratio of the first mask region in the first mask image is equivalent to the area ratio of the object to be removed in the first image.

[0162] Where 0 < P ≤ 100%.

[0163] It should be noted that when P is small (e.g., P < 2%), meaning the proportion of objects to be eliminated is small, elimination and repair are easier; when P is large (e.g., 2% ≤ P < 25%), meaning the proportion of objects to be eliminated is large, elimination and repair are more complex and require higher standards.

[0164] S304. The image processing flow corresponding to the first area ratio P is used to eliminate and repair the objects to be eliminated in the first image.

[0165] For example, when P < X1, the first image processing flow is used to eliminate and repair the object to be eliminated.

[0166] When X1≤P<X2, the first image processing flow is used to eliminate and repair the object to be eliminated.

[0167] When X2≤P<X3, the third image processing flow is used to eliminate and repair the object to be eliminated.

[0168] When P≥X3, no elimination or repair processing is performed on the object to be eliminated.

[0169] Where 0 ≤ X1 < X2 < X3.

[0170] It should be noted that in actual implementation, the number of thresholds can be adjusted (increased or decreased) according to actual usage needs. For example, four thresholds X1, X2, X3, and X4 can be set, where 0 ≤ X1 < X2 < X3 < X4, making image processing more refined and greatly improving the removal and restoration effects. For ease of explanation, X1, X2, and X3 are used as examples in the following embodiments.

[0171] It should be noted that in actual implementation, the values ​​of X1, X2, and X3 can be set according to actual usage requirements, and this application does not limit them. For ease of explanation, the following embodiments use X1 as 2%, X2 as 10%, and X3 as 25% as examples for illustrative purposes.

[0172] With the solution proposed in this application, for objects to be eliminated with different area proportions, an image processing strategy corresponding to the area proportion can be selected to eliminate and repair the objects, resulting in different image processing effects.

[0173] It should be noted that the embodiments of this application do not limit the image removal models used by the three image processing strategies. The image removal models used by the three image processing strategies may be the same or different.

[0174] In some embodiments, the first image processing strategy, the second image processing strategy, and the third image processing strategy may employ the same or different image elimination models under different circumstances; the image processing procedures may be the same or different, for example, some perform image cropping processing while others do not; the computational load of image processing increases sequentially, and the image processing effect is enhanced sequentially.

[0175] In some embodiments, for objects to be eliminated that have a very small area ratio, a first image processing strategy is adopted to perform image cropping processing, and elimination and repair processing is performed based on the first mask image, the first image, and the GAN image elimination model.

[0176] In other embodiments, for objects to be eliminated that have a slightly larger area, a first image processing strategy is adopted to perform image cropping and determine whether the background of the object to be eliminated is a complex background. If it is a simple background (e.g., sky or solid color background), then elimination and repair processing is performed based on the first mask image, the first image, and the GAN image elimination model. If it is a complex background (e.g., crowd background), then elimination and repair processing is performed based on the first mask image, the first image, and the SD image elimination model.

[0177] In some embodiments, for objects to be eliminated that occupy a large area, a second image processing strategy is adopted. Instead of image cropping, it determines whether the elimination scenario has a crowd background. If it does not, elimination and repair processing is performed based on the first mask image and an image processing network model based on SD or GAN. If it does have a crowd background, elimination and repair processing is performed based on the first mask image, the second mask image, and the SD image elimination model. The first mask image includes the mask region corresponding to the object to be eliminated in the first image, and the second mask image includes the mask region corresponding to each portrait in the first image.

[0178] It should be noted that, compared to GAN image removal models, SD image removal models have higher computational requirements but superior image processing performance. SD image removal models can handle large-area and complex background removal problems.

[0179] Figure 6 The diagram illustrates three scenarios where different image processing procedures are used to eliminate and repair objects with different area proportions.

[0180] Scene 1:

[0181] like Figure 6As shown in (a), the first image includes the object to be eliminated, 31, indicated by the dashed line. The object to be eliminated is a male, and it corresponds to the first mask region 32 in the first mask image. The area ratio P of the first mask region in the first mask image is calculated, where P < 2%. Since the area ratio of the object to be eliminated is very small, the first image processing strategy can be adopted: based on the first mask image and the first image, the GAN image elimination model is used to eliminate and repair the object 31. This approach has a small computational load and the image processing effect meets basic requirements such as small color difference.

[0182] exist Figure 6 In the image shown in (a) after removal and restoration, the image to be removed (the male image) selected by the circle or smear operation in the first image is removed. It should be noted that for image removal scenarios where the object to be removed accounts for a small proportion and removal and restoration are relatively easy to achieve, the GAN image removal model can basically meet the removal and restoration requirements and save computational resources.

[0183] Scene Two:

[0184] like Figure 6 As shown in (b), the first image includes the object to be removed, 33, indicated by the dashed line. The object to be removed 33 is a trash can, and it corresponds to the first mask region 34 in the first mask image. The area percentage P is calculated, where 2% ≤ P < 10%. Since the area percentage of the object to be removed is slightly larger, the first image processing flow can be used.

[0185] Case 1: If the background of the object to be eliminated 33 is a simple background (such as sky or solid color background), then the elimination and repair process is performed according to the first mask image and the first image and GAN image elimination model. The amount of computation is small and the basic requirements such as small color difference are met.

[0186] Case 2: If the background of the object to be eliminated 33 is a complex background (e.g., a crowd background), then the elimination and repair process is performed according to the first mask image and the first image and SD image elimination model. The elimination effect is good and meets the basic requirements such as small color difference.

[0187] exist Figure 6 In the image shown in (b) after removal and restoration, the image to be removed (the trash can image) selected by the circle or smear operation in the first image is eliminated. By adopting the second image processing strategy, the removal and restoration requirements can be met, while saving computational resources.

[0188] Third Scene:

[0189] like Figure 6As shown in (c), the first image includes the object to be removed, 35, indicated by the dashed line. The object to be removed 35 is a woman, and it corresponds to the first mask region 36 in the first mask image. The area percentage P is calculated; 10% ≤ P < 25%. Since the area percentage of the object to be removed is large, a third image processing strategy with high computational complexity and good image processing effect can be used to remove and repair the object 35.

[0190] Case 1: If the background of the object to be eliminated 35 is a non-human scene, then the elimination and repair process is performed according to the first mask image and the first image and SD image elimination model. The elimination effect is good and meets the basic requirements such as small color difference.

[0191] Scenario 2: If the background of the object to be removed 35 is a crowd scene, then the removal and repair processing is performed based on the first image, the first mask image, the second mask image, and the SD image removal model. The removal effect is good and meets basic requirements such as small color difference. The second mask image contains the masked areas of all human figures in the first image.

[0192] exist Figure 6 In the image after removal and restoration shown in (c), the image to be removed (the woman's image) selected by circling or smearing operations in the first image is removed. It should be noted that for image removal scenarios where the object to be removed occupies a large proportion and the requirements for removal and restoration are high, performing removal and restoration processing based on the first mask image, the first image, the second mask image, and the SD image removal model can well meet the removal and restoration requirements and achieve good removal and restoration results.

[0193] It should be noted that the embodiments of this application do not limit the image removal models used in the above three scenarios. For example, Figure 6 In (a), either a GAN-based image removal model or an SD-based image removal model can be used instead; similarly, in Figure 6 In (b) and (c), either an SD-based image removal model or a GAN-based image removal model can be used instead. The specific image removal model used can be determined based on the actual usage requirements.

[0194] The solution of this application allows for the selection of different area proportions of objects to be eliminated in the original image by the user. The image processing flow corresponding to the area proportion can be chosen to eliminate and repair the objects, resulting in different image processing effects. The image elimination and repair method provided by this application can eliminate people or objects in selected or painted areas, meeting various image elimination needs in terms of computational load, image processing performance, and image processing effect.

[0195] The image removal and repair method provided in this application is applicable not only to image removal scenarios with simple backgrounds, but also to image removal scenarios with complex backgrounds and image removal scenarios with crowds in the background.

[0196] The method provided in this application allows for the selection of an appropriate image processing procedure to eliminate and repair the object to be removed, based on the area ratio of the selected or painted area in the original image. This results in more refined image processing and improved removal and repair effects.

[0197] Image cropping:

[0198] In some embodiments, in a scenario where the object to be eliminated is selected by circling or smearing in the first image for elimination and repair processing, if the area ratio P meets certain conditions, the first image and the first mask image can be cropped first, and then the elimination and repair processing can be performed based on the cropped first image and the first mask image. This can optimize the effect of elimination and repair processing.

[0199] For example, when the area percentage P is less than X2 (e.g., 10%), the first image and the first mask image can be cropped separately. The cropped mask image contains the first mask region, and the cropped first image contains the object to be eliminated. Then, based on the cropped mask image and the cropped first image, the object to be eliminated selected by circling or smearing in the first image can be eliminated and repaired. The following describes the possible implementation methods for determining the cropping region and specific cropping provided by the embodiments of this application.

[0200] 1) Calculate the number of pixels in the first mask region (denoted as S).

[0201] 2) Multiply S by the coefficient corresponding to P to calculate the number of pixels in the cropped region (denoted as T).

[0202] In some embodiments, when P < 2%, the coefficient corresponding to P is N6, and the number of pixels in the cropped region T = S * N6 is calculated. Optionally, N6 can be 17. Multiplying S by 17 can be understood as cropping the image by 6%.

[0203] In some embodiments, when 2% ≤ P < 10%, the coefficient corresponding to P is N7, and the number of pixels in the cropped region T = S * N7 is calculated. Optionally, N7 can be 10. Multiplying S by 10 can be understood as cropping the image by 10%.

[0204] 3) Calculate the width and height (H1) of the cropped area (denoted as W1) based on the number of pixels T in the cropped area. For example, you can find the square root of T and determine the width W1 and height H1 of the cropped area. For instance, if T is 360000, then the width W1 and height H1 of the cropped area are both 600.

[0205] 4) such as Figure 7 As shown in (a), the width (denoted as W2) and height (denoted as H2) of the minimum bounding rectangle 17 of the first mask region are calculated.

[0206] 5) Adjust the width W1 and height H1 of the clipping region based on the width L2 and height H2 of the minimum bounding rectangle 17.

[0207] For example, if the width W1 of the clipping region is less than the width W2 of the minimum bounding rectangle 17, then W1 = W2 + N8.

[0208] For example, if the height H1 of the clipping region is less than the height H2 of the minimum bounding rectangle 17, then H1 = H2 + N8.

[0209] For example, if the width W1 or height H1 of the cropping region is less than N9, then the width W1 or height H1 of the cropping region is set to N9. Here, N9*N9 is the input image size defined by the inpainting network.

[0210] Optionally, N8 can be 6, and N9 can be 768.

[0211] 6) such as Figure 7 As shown in (a), the clipping region 42 is determined with the midpoint of the minimum bounding rectangle 41 of the first mask region as the center, according to the adjusted width L1 and height H1.

[0212] Here, it is assumed that the size of both the first image and the first mask image is a first dimension. For example, the first dimension is 1600*1200.

[0213] like Figure 7 As shown in (a) and (b) above, a first mask image of a second size is cropped from a first mask image, and a first image of a second size is cropped from a first image, following steps 1) to 6) above. For example, the second size is 1200*800.

[0214] Mask region dilation processing:

[0215] In some embodiments, in a scenario where the object to be eliminated is selected by circling or smearing in the first image for elimination and repair processing, the first mask region in the first mask image can be dilated first, and then the elimination and repair processing can be performed based on the dilated mask image, which can optimize the effect of elimination and repair processing.

[0216] In this embodiment of the application, the first mask image can be subjected to mask region dilation processing to increase the mask region, thereby increasing the scope of elimination and repair, so that the object to be eliminated can be eliminated and repaired more comprehensively, which can improve the elimination and repair effect.

[0217] In some embodiments, a mask dilation ratio N10 and an initial dilation kernel size N11 can be set. The dilation kernel size is increased by 1 each time, and the first mask region is cyclically dilated to obtain the dilated first mask image (denoted as dilatedMask) until the area (number of pixels) of the dilated first mask image is greater than or equal to R, where R = S * N10. S represents the area (number of pixels) of the first mask region.

[0218] Optionally, the expansion ratio N10 can be 1.2, and the initial expansion kernel size N11 can be 3.

[0219] Figure 8 This diagram illustrates cropping, scaling, and dilation of the first mask image. (See attached diagram.) Figure 8 As shown, the first mask image has a first size, the cropped first mask image has a second size, the dilated first mask image has a second size, and the dilated first mask area is larger than the undilated first mask area.

[0220] It is understandable that when N10 is greater than 1, the outline of the first mask region after expansion is larger than the outline of the first mask region before expansion. In this way, by increasing the mask region, the scope of elimination and repair is increased, thereby improving the elimination and repair effect.

[0221] Image removal and restoration process

[0222] The proposed solution allows for the selection of appropriate image processing methods for image removal and background restoration based on the area ratio of the object to be removed, resulting in more refined image processing and improved removal and restoration effects.

[0223] It should be noted that electronic devices can use various possible methods to implement the above-mentioned S304 image processing flow corresponding to the first area ratio P to eliminate and repair the objects to be eliminated in the first image (original image) by circling or smearing.

[0224] In this application embodiment, to facilitate the explanation of a specific scenario, the concept of a crowd background is introduced. This crowd background indicates different image features from the image background. The crowd background refers to the human portrait features relative to the object to be eliminated; a large number of human portraits exist around the object to be eliminated, forming a crowd background. The image background refers to image features in the image other than human portraits (such as trees, flowers, roads, bicycles, etc.).

[0225] It should also be noted that in the following examples, the process of image removal and background restoration involved in the solution of this application is represented by spots in order to better illustrate the effect of image removal and restoration.

[0226] The following details the possible implementation methods for image removal and background restoration by selecting an appropriate image processing method based on the area ratio of the object to be removed.

[0227] For example, combined Figure 5 ,like Figure 9 As shown, S304 may include the following S1 to S25.

[0228] S1. Determine whether P is less than 2%.

[0229] If P is determined to be less than 2%, proceed with steps S2-S8, which involves performing image removal and repair using the first image processing flow.

[0230] If P is determined to be no less than 2%, proceed to S9, which further determines whether P is less than 10%.

[0231] When P is less than 2%, the first image processing strategy is adopted.

[0232] S2. If P is determined to be less than 2%, calculate the area to be clipped based on the proportion of the first mask (6%).

[0233] For example, calculating the cropping region with a 6% mask ratio involves multiplying the number of pixels in the first mask region by 17 to obtain the number of pixels T. Then, the square root of the number of pixels T is taken to obtain the width W1 and height H1. Next, based on the width W2 and height H2 of the minimum bounding rectangle of the first mask region, the width W1 and height H1 are adjusted (increased) to obtain the width W3 and height H3. The width W3 and height H3 are then used as the width and height of the cropping region. The specific calculation process for the cropping region can be found in the detailed description above regarding the calculation of the cropping region, and will not be repeated here.

[0234] S3. Crop and scale the first image and the first mask image to obtain the cropped and scaled first image and the first mask image.

[0235] First, the first image and the first mask image are cropped according to the calculated cropping area. The cropped first image includes the object to be removed, and the cropped first mask image includes the first mask area. For details on the image cropping process, please refer to [link to documentation]. Figure 7 As shown.

[0236] Then, both the cropped first image and the first mask image are scaled to N9*N9 (e.g., 768*768) to meet the requirements of the elimination and repair network for the input image size.

[0237] The process of cropping and scaling the first mask image can be found in [reference needed]. Figure 8 It should be noted that the process of cropping and scaling the first image is similar to the process of cropping and scaling the first mask image.

[0238] The dimensions of the first image and the first mask image can be a first size, and the dimensions of the cropped area can be a second size. The second size is smaller than the first size.

[0239] S4. Dilate the mask region of the scaled first mask image.

[0240] In this application embodiment, return to reference Figure 8 As shown, the mask region of the first mask image is expanded to increase the mask region, thereby increasing the range of elimination and repair, so that the object to be eliminated is eliminated and repaired more comprehensively, and the elimination and repair effect is improved.

[0241] S5. Input the cropped and scaled first image and the cropped, scaled and dilated first mask image into the GAN image elimination model for model inference to obtain the seventh image.

[0242] The size of the seventh image is specified by the GAN image elimination model. For example, the size of the seventh image is 768*768.

[0243] In some embodiments, the GAN image removal model can be used to repair the large mask LAMA model.

[0244] In some embodiments, the GAN image removal model can be an aggregated contextual-transformation (AOT) model.

[0245] In some embodiments, the GAN image removal model can be a mask-interactive generative adversarial network (MI-GAN) model.

[0246] S6. After scaling the seventh image to the second size, fill it into the cropped area of ​​the first image to obtain the third image of the first size.

[0247] In this embodiment of the application, the seventh image is scaled (e.g., enlarged) from the third size to the second size, i.e., size restoration.

[0248] In some embodiments, an image super-resolution method may be used to scale the second image to be eliminated. Super-resolution processing refers to the process of recovering a high-resolution image from a low-resolution image.

[0249] It is understandable that by scaling the seventh image to the same size as the cropped area (the second size), the scaled seventh image can be seamlessly filled into the cropped area of ​​the first image.

[0250] S7. After scaling the cropped, scaled, and dilated first mask image to the second size, fill it into the cropped area of ​​the first mask image to obtain the dilated and filled first mask image.

[0251] This application does not limit the execution order of S6 and S7. For example, S6 can be executed first and then S7; or S7 can be executed first and then S6; or S6 and S7 can be executed simultaneously.

[0252] S8. Based on the first image, the third image, and the first mask image after dilation and filling, obtain the second image.

[0253] In this embodiment of the application, the second image is obtained by the following image matrix calculation formula.

[0254] Pi = Po × (1 – Pm) + Pe × Pm.

[0255] Where Po represents the first image, Pe represents the third image, Pm represents the first mask image after dilation and filling, and Pi represents the second image.

[0256] Where (1–Pm) represents inverting the pixel values ​​of the first mask image.

[0257] Figure 10 The diagram illustrates the process of image removal and restoration using the GAN image removal model when P is less than 2%.

[0258] like Figure 10 As shown, in response to a user's selection operation (e.g., circling or smearing) on ​​the first image (original image), the electronic device determines the object to be removed and calculates the area percentage P of the object to be removed in the first image, which is less than 2%.

[0259] A first mask image is obtained based on the first image. The mask region in the first mask image is the mask region of the object to be eliminated. The size of both the first image and the first mask image is the first size.

[0260] Then, based on the mask region in the first mask image, a cropped region is calculated with a mask ratio of 6%. The cropped region includes the mask region and is larger than the mask region, and the size of the cropped region is the second size. The first image and the first mask image are cropped based on the cropped region to obtain a first image of the second size and a first mask image of the second size, respectively.

[0261] Then, the first image of the second size and the first mask image of the second size are scaled to obtain the first image of the third size and the first mask image of the third size.

[0262] Then, the mask region of the first mask image with the third size is dilated to obtain the first mask image with the third size dilated.

[0263] Then, the first image at the third size and the first masked image at the third size are input into the GAN image elimination model to obtain the seventh image at the third size. The seventh image at the third size is then scaled to obtain the seventh image at the second size.

[0264] Then, the seventh image of the second size is filled into the cropped area of ​​the first image to obtain the third image of the first size.

[0265] On the other hand, the first mask image, after being expanded to a third size, is filled into the cropped area of ​​the first mask image of the first size to obtain the first mask image after being expanded and filled to the first size. Then, the pixel values ​​of the first mask image after being expanded and filled are inverted to obtain the fifth image. Then, the fifth image is multiplied by the first image to obtain the sixth image.

[0266] The third image and the first masked image after dilation and filling are ANDed (i.e. multiplied) to obtain the fourth image.

[0267] Then, the fourth and sixth images are fused together to obtain the second image.

[0268] like Figure 10 As shown, in the second image, the image to be eliminated (portrait) selected by circling or smearing is eliminated, and the background image of the eliminated area has been repaired.

[0269] With the proposed solution, for objects with a small area to be eliminated, the requirements for elimination and repair are relatively low. An image elimination model with low computational load can be used, and the image processing effect only needs to meet basic requirements such as small color difference.

[0270] The above steps S1-S8 illustrate the image removal and repair process when the area percentage P of the object to be removed is very small (e.g., P < 2%). The following steps will describe the image removal and repair process when the area percentage P of the object to be removed is slightly larger (e.g., 2% ≤ P < 10% or P < 25%).

[0271] S9. If P is determined to be no less than 2%, determine whether P is less than 10%.

[0272] If P is determined to be less than 10%, proceed with S10.

[0273] If P is determined to be no less than 10%, proceed to S13, which is to further determine whether P is less than 25%.

[0274] The second image processing strategy is adopted when 2% ≤ P < 10%.

[0275] S10. If P is determined to be less than 10%, calculate the area to be clipped based on the proportion of the second mask (10%).

[0276] The process of calculating the area to be clipped using a 10% mask ratio can refer to the process described above for calculating the area to be clipped using a 6% mask ratio.

[0277] S11. Crop the first image and the first mask image to obtain the cropped first image and the first mask image.

[0278] Both the first image and the first mask image are of the first size, and after cropping, they are of the second size.

[0279] The implementation process of S11 is similar to the implementation process of cropping the first image and the first mask image in S3 above, and will not be repeated here.

[0280] S12. Based on the cropped first image and the first mask image, determine whether the background of the object to be eliminated is a complex background.

[0281] Figure 11 This diagram illustrates how to determine whether the background of the object to be removed is a complex background. Figure 11 As shown, the first mask region is dilated, and then the difference is calculated to obtain the outer annular region. The feature point density in the outer annular region can be used to determine whether the background of the object to be removed is a complex background.

[0282] For example, the first mask region is expanded by an expansion ratio N12 to obtain dialtedMask1. Furthermore, the first mask region is expanded by an expansion ratio N13 to obtain dialtedMask2. Optionally, N12 is 15 and N13 is 75.

[0283] Then, determine the outer ring region of the first mask region: AroundMask = dialtedMask2 - dialtedMask1.

[0284] Then, the number of feature points FeaPtNum and the number of pixels PAM in the AroundMask annular region of the first image are calculated.

[0285] Then, the feature point density FeaPtDensity in the AroundMask annular region of the first image is calculated:

[0286] FeaPtDensity=FeaPtNum / PAM.

[0287] If FeaPtDensity > N14, then the background of the object to be eliminated is determined to be a complex background.

[0288] If FeaPtDensity≤N14, then the background of the object to be eliminated is determined to be a simple background.

[0289] Optionally, N14 is 1%.

[0290] Optionally, the number of feature points FeaPtNum can be calculated using the FAST (features from accelerated segment test) feature point extraction method, the ORB (oriented FAST and rotated BRIEF) feature point extraction method, or the SURF (speeded uprobust features) feature point extraction method.

[0291] refer to Figure 11 If it is determined that the background of the object to be removed is not a complex background, then proceed to steps S4-S8, which involves using a GAN image removal model that meets the actual usage requirements to perform image removal and restoration processing. If it is determined that the background of the object to be removed is a complex background in the image removal scenario, then the SD image removal model, which has stronger image processing performance, can be used.

[0292] For example, Figure 12 The flowchart illustrates the process of image removal and restoration for non-complex backgrounds when 2% ≤ P < 10%.

[0293] like Figure 12 As shown, in response to the user's selection operation on the first image (original image), the electronic device determines the selected area and calculates the area percentage P of the object to be eliminated in the first image, where 2% ≤ P < 10%.

[0294] A first mask image is obtained based on the first image. The mask region in the first mask image is the mask region of the object to be eliminated. The size of both the first image and the first mask image is the first size.

[0295] Then, based on the mask region in the first mask image, a cropped region is calculated with a mask ratio of 10%. The cropped region includes the mask region and is larger than the mask region, and the size of the cropped region is the second size. The first image and the first mask image are cropped based on the cropped region to obtain a first image of the second size and a first mask image of the second size, respectively.

[0296] Unlike Figure 10 The point is, in Figure 12 In the process, after image cropping, the first cropped image and the first mask image are used to determine whether the background of the object to be removed is a complex background.

[0297] If the background of the object to be eliminated is determined to be a complex background, the cropped first image and the first mask image are scaled, and then the mask region of the scaled first mask image is dilated. Then, the cropped and scaled first image and the cropped, scaled and dilated first mask image are input into the SD image elimination model to obtain the seventh image.

[0298] Alternatively, if it is determined that the background of the object to be eliminated is not a complex background, the cropped first image and the first mask image are scaled, and then the mask region of the scaled first mask image is dilated. Then, the cropped and scaled first image and the cropped, scaled and dilated first mask image are input into the GAN image elimination model to obtain the seventh image.

[0299] Then the seventh image is filled into the cropped area of ​​the first image to obtain the third image of the first size.

[0300] On the other hand, the first mask image, after being expanded to a third size, is filled into the cropped area of ​​the first image of the first size to obtain the first mask image after being expanded and filled to the first size. Then, the pixel values ​​of the first mask image after being expanded and filled are inverted to obtain the fifth image. Then, the fifth image is multiplied by the first image to obtain the sixth image.

[0301] The third image and the first masked image after dilation and filling are ANDed (i.e. multiplied) to obtain the fourth image.

[0302] Then, the fourth and sixth images are fused together to obtain the second image.

[0303] like Figure 12 As shown, in the second image, the portraits selected by circling or smearing are eliminated, and the background image of the eliminated area has been restored.

[0304] The solution proposed in this application allows for the use of a GAN image removal model that meets practical application requirements for image removal scenarios with non-complex backgrounds. For image removal scenarios with complex backgrounds, a SD image removal model with stronger image processing performance can be used. The image removal and restoration method provided in this application can remove images of people or objects from the selected area, and can meet various image removal needs in terms of computational load, image processing performance, and image processing effect.

[0305] Optionally, if the background of the object to be removed is determined to be a complex background, then S14 can be executed to determine whether it is a scene with a crowd background. It should be noted that for complex backgrounds, it is possible to further identify whether it is a crowd background. If it is not a crowd background, an image removal model that meets practical needs can be used. If it is a crowd background, to avoid generating distorted human images after image removal, a powerful image removal model (such as the SD image removal model) can be used based on the first image and the second mask image to avoid generating distorted human images. The specific implementation process will be explained below.

[0306] It should also be noted that, in the case of 2% ≤ P < 10%, since image cropping was performed in the preprocessing of image elimination and restoration, the cropped image filling process, as in S5 and S6, needs to be completed in the postprocessing of image elimination and restoration.

[0307] When P < 25%, the third image processing strategy is adopted.

[0308] The above describes the image removal and repair process when the area percentage P of the object to be removed is very small (e.g., P < 2%), and when the area percentage P of the object to be removed is slightly large (e.g., 2% ≤ P < 10%). The following describes the image removal and repair process when the area percentage P of the object to be removed is even larger (e.g., P < 25%).

[0309] S13. If P is not less than 10%, determine whether P is less than 25%.

[0310] If P is determined to be less than 25%, proceed to S14, which determines whether the current elimination scene is a scene with a crowd background. If P is determined to be not less than 25%, proceed to S25.

[0311] S14. If P is less than 25%, determine whether the current elimination scene is an elimination scene with a crowd background.

[0312] The scenario for eliminating crowd backgrounds refers to a situation where the object to be eliminated is a portrait or object, and the background of the object to be eliminated is a crowd background (with people overlapping or obscuring each other). The specific judgment method will be explained below.

[0313] If it is determined that the current elimination scene is not an elimination scene with a crowd background, continue to execute S15-S19, or if it is determined that the current elimination scene is an elimination scene with a crowd background, continue to execute S20-S24.

[0314] The following section first explains the elimination and repair process for elimination scenarios where the current elimination scene is not a crowd background.

[0315] S15. In cases where the current elimination scene is not a scene with a crowd background, scale the first image and the first mask image to the required size of the SD image elimination model.

[0316] S16. Perform mask region dilation processing on the scaled first mask image to obtain the dilated first mask image.

[0317] S17. Input the scaled first image and the scaled and dilated first mask image into the SD image elimination model for model inference to obtain the third image.

[0318] S18. Restore the size of the dilated first mask image to the first size to obtain the dilated first mask image.

[0319] S19. Based on the first image, the third image, and the first masked image after dilation, the second image is obtained.

[0320] In this embodiment of the application, the second image is obtained by the following image matrix calculation formula.

[0321] Pi = Po × (1 – Pm) + Pe × Pm.

[0322] Where Po represents the first image, Pe represents the third image, Pm represents the first masked image after dilation, and Pi represents the second image.

[0323] Where (1–Pm) represents inverting the pixel values ​​of the first mask image after dilation.

[0324] Figure 13A This diagram illustrates the image removal and restoration method provided in this application, which involves judging the background of a large proportion of objects and implementing different image processing strategies.

[0325] If the background image of the object to be removed is not a crowd background image, then the first image and the first mask image (containing the mask region corresponding to the object to be removed) are input into the second image removal model, and then the image output by the model is post-processed to obtain the second image. The specific implementation process can be found below. Figure 13B .

[0326] If the background image of the object to be removed is a crowd background image, then the first image and the second mask image (including the masked area of ​​each portrait in the first image) are input into the first image removal model, and then the image output by the model is post-processed to obtain the second image. The specific implementation process can be found below. Figure 13C .

[0327] Figure 13B The diagram illustrates the process of image removal and restoration for scenes where the current removal scene is not a crowd background, provided that 10% ≤ P < 25%.

[0328] like Figure 13B As shown, in response to the user's selection operation on the first image (original image), the electronic device determines the selected area and calculates the area percentage P of the object to be eliminated in the first image, where 10% ≤ P < 25%.

[0329] A first mask image is obtained based on the first image. The mask region in the first mask image is the mask region of the object to be eliminated. The size of both the first image and the first mask image is the first size.

[0330] Unlike Figure 10 and Figure 12 The point is, in Figure 13B In this process, neither image cropping nor image filling is performed.

[0331] Unlike Figure 10 and Figure 12 The point is that, in Figure 13B In the process, it is necessary to determine whether the current scene is a scene with a crowd background, that is, to determine whether the object to be eliminated is a human figure and whether the background of the object to be eliminated is a crowd background.

[0332] If it is determined that the current scene is not a scene where a crowd background is eliminated, the first image and the first mask image are scaled, and then the mask region of the scaled first mask image is dilated. Then, the scaled first image and the scaled and dilated first mask image are input into the SD image elimination model to obtain the third image.

[0333] The third image and the dilated first mask image are ANDed (i.e. multiplied) to obtain the fourth image.

[0334] Invert the pixel values ​​of the first masked image after dilation to obtain the fifth image. Then multiply the fifth image by the first image to obtain the sixth image.

[0335] Then, the fourth and sixth images are fused together to obtain the second image.

[0336] like Figure 13B As shown, the first object image (such as the first portrait) in the second image is eliminated, and the background image of the eliminated area has been repaired.

[0337] The proposed solution addresses the challenges of eliminating and repairing objects with large areas requiring high performance. Therefore, employing a superior image elimination model yields better elimination and repair results.

[0338] The above explains the elimination and repair process for elimination scenarios where the current elimination scene is not a crowd background. The following explains the elimination and repair process for elimination scenarios where the current elimination scene is a crowd background.

[0339] In this embodiment, the removal is performed on a first object image (which may be a human body or an object) with a crowd background. This image removal scenario has higher requirements because removing the first object image with a crowd background may generate deformed human figures in the area where the first object image was removed. To avoid generating deformed human figures, this application adopts the following scheme to perform image removal and restoration processing.

[0340] S20. In the scenario of eliminating the background of the crowd, obtain a multi-person portrait instance mask image (i.e., the second mask image) based on the first image.

[0341] S21. Scale the first image and the multi-person portrait instance mask image to the required size (third size) of the SD image elimination model.

[0342] For example, the SD image removal model requires a size of 768*768.

[0343] S22. Input the scaled first image and the multi-person portrait instance mask image into the SD image elimination model for model inference to obtain the third image.

[0344] It should be noted that the size of the third image is the third dimension, for example, the size of the third image can be 768*768.

[0345] S23. Dilate the mask region of the first mask image to obtain the dilated first mask image.

[0346] This application does not limit the execution order of S22 and S23. For example, S22 can be executed first and then S23; or S23 can be executed first and then S22; or S22 and S23 can be executed simultaneously.

[0347] S24. Based on the first image, the third image, and the first masked image after dilation, obtain the second image.

[0348] In this embodiment of the application, the second image is obtained by the following image matrix calculation formula.

[0349] Pi = Po × (1 – Pm) + Pe × Pm.

[0350] Where Po represents the first image, Pe represents the third image, Pm represents the first masked image after dilation, and Pi represents the second image.

[0351] Where (1–Pm) represents inverting the pixel values ​​of the first mask image after dilation.

[0352] Figure 13C The diagram illustrates the process of image removal and restoration in a scene where the current removal scene is a crowd background, provided that 10% ≤ P < 25%.

[0353] like Figure 13C As shown, in response to the user's selection operation on the first image (original image), the electronic device determines the selected area or the smeared area and determines the object to be eliminated, and calculates the area percentage P of the object to be eliminated in the first image, where 10% ≤ P < 25%.

[0354] It should be noted that the judgment logic of this application can determine that: the first image contains multiple portraits and the portraits in the selected area are occluded (i.e. closely adjacent) to other portraits. This elimination scenario is a crowd background elimination scenario, that is, to eliminate and repair a certain portrait in the crowd, which has relatively high requirements for image elimination and repair.

[0355] A first mask image is obtained based on the first image. The mask region in the first mask image is the mask region of the object to be eliminated. The size of both the first image and the first mask image is the first size.

[0356] Figure 13C Unlike Figure 13B The point is, in Figure 13C In this process, a multi-portal instance mask image (i.e., a second mask image) is obtained based on the first image. This multi-portal instance mask image includes the mask region for each portrait in the first image.

[0357] Figure 13C Unlike Figure 13B The point is that, in Figure 13C In the process, the first image and the second mask image are scaled, and then the scaled first image and the scaled second mask image are input into the SD image elimination model to obtain the third image.

[0358] exist Figure 13C In the process, the mask region of the first mask image is dilated to obtain the dilated first mask image.

[0359] The third image and the dilated first mask image are ANDed (i.e. multiplied) to obtain the fourth image.

[0360] Invert the pixel values ​​of the first masked image after dilation to obtain the fifth image. Then multiply the fifth image by the first image to obtain the sixth image.

[0361] Then, the fourth and sixth images are fused together to obtain the second image.

[0362] like Figure 13C As shown, the human figures in the selected area of ​​the second image are removed, and the background image of the removed area is restored.

[0363] like Figure 13D As shown in the embodiments of this application, the image removal and restoration in multi-person scenes has been improved. After performing removal and restoration processing on the image area specified by the user in the photo, new portraits are avoided. Therefore, the solution of this application solves the problem of regenerating deformed portraits after removing a portrait in a multi-person scene, thus improving the image removal and restoration experience.

[0364] It should be noted that, in this embodiment of the application, the example of eliminating a certain human figure in the background of a crowd to avoid the generation of deformed human figures is used for illustrative purposes. In actual implementation, this embodiment of the application is also applicable to the scenario of eliminating a certain object (such as a trash can) in the background of a crowd. In other words, a certain object in the background of a crowd can be eliminated, and the generation of deformed human figures can also be avoided.

[0365] S25. If P is determined to be greater than or equal to 25%, prompt that the selection operation is invalid.

[0366] If the selected area is too large, to avoid poor image removal and repair results, a message can be displayed indicating that the selection operation is invalid and suggesting that the operation be repeated. This will ensure better image removal and repair results and improve the user experience.

[0367] The method provided in this application selects an appropriate image processing procedure to eliminate and repair the object to be eliminated based on the area ratio of the object to be eliminated in the original image. The image processing is more refined, and the elimination and repair effect is improved.

[0368] It should be noted that the thresholds of 2%, 10%, and 25% mentioned above are illustrative examples. In actual implementation, the thresholds can be adjusted according to actual usage needs. For example, the thresholds can be adjusted to 3%, 12%, 50%, or 1%, 15%, and 75%.

[0369] In practice, the number of thresholds can be increased or decreased according to actual usage needs.

[0370] For example, a threshold can be set, such as 25%. When the area of ​​the object to be removed in the first image is less than the threshold of 25%, one image processing strategy can be used for image removal and restoration. When the area of ​​the object to be removed in the first image is greater than or equal to the threshold of 25%, another image processing strategy can be used for image removal and restoration.

[0371] As another example, two thresholds can be set, such as 10% and 25%. This reduces the computational load while ensuring the effectiveness of image removal and modification.

[0372] For example, four thresholds can be set, such as 2%, 10%, 15%, and 25%, which can make image processing more refined and improve the effect of image removal and restoration.

[0373] The image removal and repair method provided in this application can be applied to image removal scenarios with simple backgrounds or complex backgrounds.

[0374] In this application, the elimination and restoration method based on the GAN image elimination model and the elimination and restoration method based on the SD image elimination model are effectively combined, which can improve the effect and performance of elimination and restoration.

[0375] Image removal scenario to determine if it is a crowd background

[0376] The above embodiments illustrate the scenario of eliminating crowd backgrounds. The following details the possible implementation methods of image elimination scenarios for determining whether an image has a crowd background, as provided in this application.

[0377] Figure 14 An exemplary flowchart illustrating an image removal scenario for determining whether it is a crowd background, as provided in an embodiment of this application, is shown.

[0378] S401, the first image is input into a human image instance segmentation network or a panoramic segmentation network model to obtain a second mask image. This second mask image includes the masked regions of the Y human images. The second mask image can also be called a human image instance segmentation mask image.

[0379] S402, determine whether Y is greater than the threshold N15.

[0380] Optionally, N15 is 3.

[0381] If the second mask image is empty (Y=0), meaning there is no mask region or no region marked as 1, then it indicates that there is no human figure in the first image. Therefore, it can be determined that this scene does not belong to the scene where crowd backgrounds are eliminated.

[0382] If the second mask image is not empty, that is, if there is a mask region or a region marked as 1, then it means that there is at least one human figure in the first image.

[0383] In some embodiments, if the second mask image is not empty (i.e., there are human figures), and the number of human figure instances Y is less than N15, then it can be determined that the scene does not belong to the scene where the crowd background is eliminated.

[0384] In some embodiments, if the second mask image is not empty (i.e., there is a human image) and the number of human image instances is greater than or equal to N15, then S403 continues to be executed.

[0385] S403. Iterate through and calculate the number of pixels in the intersection area (denoted as Q1) and the number of pixels in the union area (denoted as Q2) between the first mask area (i.e., the mask area of ​​the object to be removed) and the i-th portrait mask area in the second mask image, as well as the intersection-union ratio (denoted as IoU). i starts from 1 and can take values ​​up to Y.

[0386] Where IoU = Q1 / Q2.

[0387] S404. Determine whether Q1 / Q2 is greater than the threshold N16.

[0388] Optionally, N16 = 0.8.

[0389] S405. If Q1 / Q2≥N16, then the first mask region is determined to be a single person region, and the traversal calculation is stopped and the mask image of the i-th person instance is recorded.

[0390] If no human face instance mask meets the conditions after traversing all the data, it is determined that the first mask area does not contain human faces, and therefore it can be determined that the scene does not belong to the scene where the crowd background is eliminated.

[0391] S406. Iterate through and calculate the minimum distance L between the i-th portrait mask region and the j-th portrait mask region in the second mask image. j starts from 1 and can take values ​​up to Y.

[0392] After recording the mask image of a single person's portrait instance, it is possible to determine whether the scene is a scene where the background of a crowd is to be eliminated based on the close adjacency of the recorded single person's portrait mask with the other portrait masks.

[0393] S407. Determine whether L is less than D, where D = W × N17.

[0394] Optionally, N17 can be 0.5.

[0395] Where W represents the width of the minimum bounding rectangle of the single-person portrait mask.

[0396] Specifically, it determines whether the minimum bounding rectangle of the recorded i-th person's image mask intersects with the minimum bounding rectangle of any other person's image mask, or whether the minimum distance between them is less than D.

[0397] S408. If the minimum bounding rectangle of the recorded i-th person's mask intersects with the minimum bounding rectangle of any other person's mask, or the minimum distance between them is less than W×N17, then it is determined that the i-th person's mask and the other person's mask are in a close adjacency state, and the traversal calculation stops. In this case, it can be determined that the scene belongs to the scene of eliminating crowd background.

[0398] S409. If no human face instance mask meets the conditions after traversal, it is determined that the scene is not tightly connected and therefore does not belong to the elimination scene with a crowd background (i.e., the elimination scene without a crowd background).

[0399] In this embodiment of the application, in the scenario of eliminating and repairing objects selected by circling or smearing in the first image, it is determined whether the current scene is an elimination scenario with a crowd background based on the first mask image and the mask image of multiple portrait instances (i.e., the second mask image). If it is determined that the current scene is an elimination scenario with a crowd background, an elimination and repair method suitable for crowd backgrounds is adopted to perform image elimination and repair, so as to avoid generating deformed portraits. This can optimize the effect of elimination and repair processing.

[0400] It should be noted that various possible methods can be used to implement the "image removal scenario of determining whether there is a crowd background," and this application does not limit this. In some embodiments, a large image-text model can also be used to determine whether there is a crowd background. For example, an image-text question-and-answer model can be used to ask whether the object to be removed is in a crowd, and the image removal scenario of determining whether there is a crowd background can be determined based on the output or answer of the image-text question-and-answer model.

[0401] The above describes the image removal strategies provided by the embodiments of this application based on various scenarios (such as complex backgrounds or crowd backgrounds), which can avoid the situation where other images are regenerated after image removal. Specifically, in the image removal and restoration process of various scenarios, the embodiments of this application employ specific image removal models for image processing. It should be noted that the embodiments of this application not only propose different image removal strategies for different scenarios and improve the image processing strategies to enhance the image removal and restoration effect, but also improve the image removal model used, further enhancing the image removal and restoration effect. The improvements and effects of the image removal model in the embodiments of this application are described in detail below.

[0402] The image removal and repair method provided in this application embodiment can be applied to the following scenarios.

[0403] Scenario 1: A user is traveling during a holiday and taking photos at a scenic spot. The scenic spot is crowded, and some passersby appear in the photos. The user wants to remove the passersby from the photos to generate a more aesthetically pleasing image.

[0404] Scenario 2: When users take photos, some unsightly items (such as trash cans, clutter, etc.) appear in the photos. Users want to remove these items to generate more aesthetically pleasing photos.

[0405] It should be noted that when using existing open-source image removal models (such as the SD image removal model), there is usually a problem of regenerating other foreground objects after removing the object.

[0406] For ease of explanation, the SD image removal model is used as an example below. To solve the aforementioned problem of random image generation, this application improves the SD image removal model in the following three aspects:

[0407] First embodiment: By using foreground and background decomposition training and inference based on panoramic segmentation, the problem of regeneration of foreground objects can be suppressed.

[0408] The above approach allows for model inference based on positive and negative text. The model inference tends to generate background pixel information corresponding to positive text in the eliminated area, while suppressing the generation of foreground object pixel information corresponding to negative text. Therefore, through model inference, no other foreground object images such as portraits or objects will be generated in the eliminated area, thereby improving the image elimination and restoration effect.

[0409] Second embodiment: For the model inference stage of the first embodiment, an adaptive mask expansion based on brightness contrast is adopted, which can further suppress the problem of regeneration of foreground objects.

[0410] Experimental tests revealed that when there is a shadow area around the object to be removed in the first image, the shadow area will affect the image restoration to a certain extent. If the mask of the object to be removed does not include the shadow area, other foreground objects such as portraits or objects may be generated in the removed area.

[0411] The above-described scheme of this application performs dilation processing on the mask of the object to be eliminated, so that the mask after dilation includes the shadow area. This can eliminate both the object to be eliminated and the shadow area. Then, when performing image restoration on the eliminated area, the shadow area can be avoided from affecting the image restoration, thereby avoiding the generation of other foreground object images such as portraits or objects in the eliminated area. Therefore, the effect of image elimination and restoration can be improved.

[0412] Third embodiment: For the model inference stage of the first embodiment, self-attention layer suppression based on panoramic segmentation is adopted, which can further suppress the problem of foreground object regeneration.

[0413] Experimental tests revealed that when there are foreground objects in the first image that have similar features to the object to be eliminated, these similar foreground objects can affect image restoration to some extent. If the reference intensity of similar foreground objects is too high during image restoration, it is very likely that such foreground objects with similar features will be generated in the already eliminated area.

[0414] The above-mentioned scheme of this application can be used to suppress self-attention layers based on panoramic segmentation. By processing the heatmap of the region to be eliminated by algorithm, the reference intensity of the heatmap of the region to be eliminated to other foreground objects is reduced. That is, when the diffusion model generates the background image of the eliminated region, it will reduce the reference to other foreground object regions, thereby reducing the phenomenon of random generation by the diffusion model.

[0415] The three embodiments described above will be explained below.

[0416] First Implementation Example: Foreground and Background Decomposition Training and Inference Based on Panoramic Segmentation

[0417] The structure of the diffusion model provided in this application will be described below with reference to the accompanying drawings. Figure 15A A schematic diagram of a diffusion model is shown. (Reference) Figure 15A As shown, the diffusion model provided in this application includes a pixel space module (or pixel space domain), a latent space module (or latent space domain), and a text input-based adjustment module.

[0418] The pixel space module includes an encoder ε and a decoder D. The incomplete image I_m and mask image M of the sample image I are input into the latent space module after being processed by the encoding unit. The latent space module performs image removal and restoration processing and generates image features. These image features are processed by the decoding unit and output to the pixel space module, with the sample image I serving as a monitoring signal.

[0419] The latent space module includes a diffusion processing module and a denoising U-net network.

[0420] The diffusion processing module performs diffusion processing on the input image. This diffusion processing includes forward diffusion and backward diffusion. Forward diffusion involves progressively corrupting the data structure by applying noise sampled from a normally distributed distribution until the image converges to pure noise. This process is implemented using a parameterized Markov chain, where the amount of noise at each time step is not constant but scaled by a time schedule to the mean and variance. Backdiffusion, on the other hand, is the process of learning how to generate a high-resolution image from Gaussian noise input. This involves training a neural network to predict the noise at each time step, enabling the model to generate data from the noise.

[0421] In the embodiments of this application, reference is made to Figure 15A As shown, x represents the image input to the latent space domain. After diffusion processing, x becomes x t x t This represents the implicit representation of the image at time step t.

[0422] The text-based adjustment module can input text (such as foreground and background text) into the hidden space module.

[0423] It should be noted that, unlike the traditional diffusion model, the improved diffusion model in this application adds a text-based adjustment module. By adding foreground and background text inputs for model training, the diffusion model can learn the foreground and background texts. During model inference, it can perform noise prediction based on the foreground and background texts. By strengthening the noise prediction of the background text and suppressing the noise prediction of the foreground text, it can favor the reference background pixels and suppress the reference foreground object pixels when regenerating the image in the eliminated region. This can suppress the problem of regenerating foreground objects.

[0424] The U-net network is a neural network architecture for image segmentation. It combines features from both the encoder and decoder, preserving spatial information through skip connections to achieve accurate segmentation of specific objects in an image. By incorporating attention-based convolutional blocks from a diffusion model into the U-net network, global image connections and multi-scale prediction and supervision can be achieved, further improving feature extraction capabilities and segmentation accuracy.

[0425] In the embodiments of this application, reference is made to Figure 15A As shown, the U-net network includes processing modules such as a self-attention layer (denoted as QKV). The input parameters of the U-net network include the image x. t Text input, etc. The U-net network can acquire data for image x. t The noise prediction value is predicted at each time step t, and the image features x at the next time step can be predicted. t-1 In the embodiments of this application, the U-net network can predict noise values ​​based on different text inputs. For example, the U-net network can predict foreground noise and background noise.

[0426] In the embodiments of this application, reference is made to Figure 15A As shown, for a diffusion model used to perform an image removal task, the training process may include: based on a sample image I and a mask region M of the object to be removed in the sample image I, setting the region of the object to be removed in the sample image I (referred to as the region to be removed) to 0, to obtain a fragmented image I_m; then, inputting the fragmented image I_m, the mask M, and the text text into the diffusion model, and using the complete sample image I as a supervision signal to train the diffusion model.

[0427] In this embodiment, the trained diffusion model employs classifier-free model inference to guide the model in generating background information and suppress the problem of foreground object regeneration.

[0428] Classifier-free guidance is a technique in machine learning that involves using different models or algorithms to guide the generation process of data or images without relying on a traditional classifier. In image generation, classifier-free guidance can alleviate the training cost of conditional generation by using the same model to represent both unconditional and conditional generation simultaneously, thus improving efficiency. For example, in the SD model, classifier-free guidance is implemented using gradient prediction models, one for unconditional generation and another for conditional generation. This approach does not require training an additional classifier; instead, it achieves this by training both targets simultaneously during training.

[0429] It should be noted that in this embodiment, the input image is decomposed into a foreground object region and a background region. When the sample mask M covers the foreground object region (i.e., the sample mask M is a foreground sample mask), a foreground text input (denoted as Obj) is added for training. When the sample mask M covers the background region (i.e., the sample mask M is a background sample mask), a background text input (denoted as Ctxt) is added for training.

[0430] This approach allows the model to generate background information during the inference phase using classifier-free guidance (CFG), thus suppressing the regeneration of foreground objects. Specifically, by employing classifier-free model inference, the model generates background information using foreground masks and foreground text input (as negative text), and background masks and background text input (as positive text), thereby suppressing the regeneration of foreground objects.

[0431] The implementation process of the model training phase and the model inference phase is explained in detail below.

[0432] Model training phase

[0433] refer to Figure 15B The model training phase includes the following steps 1 to 3:

[0434] Step 1: Panoramic Image Segmentation

[0435] Panoramic image segmentation allows for the differentiation of foreground and background in an image. A sample image I is input into the panoramic segmentation model to obtain foreground segmentation results and semantic segmentation results of the background regions in the image. The foreground segmentation results can be instance segmentation results for each foreground object (e.g., a person, an animal, etc.). For example, ... Figure 15B The panoramic segmentation result shown in the left image is as follows: the white part represents the foreground instance region, and the black part represents the background region.

[0436] Step 2: Foreground and background mask generation based on semantic understanding

[0437] In some embodiments, the panoptic segmentation result is a dataset uniformly distributed in the interval [0,1]. Sampling is performed from the dataset uniformly distributed in the interval [0,1], with the sampling probability denoted as P.

[0438] refer to Figure 15B When P>0.5, a foreground instance region is randomly selected, and this foreground instance region is set as the sample mask M (i.e., the foreground sample mask). The corresponding specific text input Obj (i.e., the foreground text) is generated, and the learnable text Obj is used as the text input for this training.

[0439] refer to Figure 15B When P≤0.5, a background region is randomly selected as the sample mask M (i.e., the background sample mask), and the corresponding specific text input CTXT (i.e., the background text) is generated. The learnable text CTXT is used as the text input for this training.

[0440] It should be noted that "randomly selecting a background region" includes the following situations: a background region can be selected from region A in the image other than the foreground instance region; or a background region can be selected from region B containing the foreground instance as a background sample mask. In some embodiments, the area of ​​the background sample mask in the selected region B is larger than the area of ​​the foreground instance.

[0441] In this embodiment, the dataset corresponding to the panoramic segmentation result is divided into two parts. One part contains a randomly selected foreground instance region, which is used as a foreground sample mask, and the learnable foreground text Obj is used as the text input for this training. The other part contains a randomly selected background region, which is used as a background sample mask, and the learnable background text CTXT is used as the text input for this training.

[0442] The foreground sample mask and the background sample mask are collectively referred to as foreground and background masks.

[0443] Step 3: Train the model using sample image I, sample mask M, and text.

[0444] Refer again Figure 15B Using sample image I, sample mask M (including foreground sample mask and background sample mask), and text (including foreground text Obj and background text CTXT), the following is performed: Figure 15A The diffusion model shown is trained.

[0445] For example, the model can be trained using a background sample mask and background text as the training set and sample images as the supervision signal; and the model can be trained using a foreground sample mask and foreground text as the training set and sample images as the supervision signal.

[0446] Through steps 1 to 3 above, the sample image I is first subjected to panoramic segmentation to distinguish the foreground instance region and the background region. Then, based on the semantic understanding of the panoramic segmentation result, a foreground sample mask and a background sample mask are generated, along with foreground text Obj and background text CTXT. The learnable foreground text Obj and learnable background text CTXT are then used as text inputs for model training. The model is trained using the sample image I, the foreground sample mask, the background sample mask, the foreground text Obj, and the background text CTXT. This yields the improved diffusion model provided in this embodiment. By using this improved diffusion model, image removal and restoration can be achieved, and the regeneration of foreground objects can be suppressed, greatly improving the image removal and restoration effect.

[0447] After model training is completed, the model inference phase (i.e., the model application phase) begins.

[0448] Model inference stage

[0449] In this embodiment of the application, the model inference stage may include the following steps 4 and 5.

[0450] Step 4: Image panoramic segmentation and acquisition of positive / negative text

[0451] Figure 15C A schematic diagram illustrating the implementation process of the model inference stage is shown. (Reference) Figure 15C The first image to be inferred is input into the panoramic segmentation model to obtain the foreground and background category information of the first image. The background category information and CTXT are used as positive text, and the foreground category information and Obj are used as negative text.

[0452] Positive text can be represented as: CTXT + background category information.

[0453] For example, positive text: CTXT, door, wall, etc.

[0454] Negative text can be represented as: Obj + foreground category information.

[0455] For example, negative text: Obj, person, human, etc.

[0456] Step 5: Classifier-free model inference

[0457] After obtaining the positive and negative text, model inference can be performed based on the positive and negative text in a classifier-free manner.

[0458] During model inference, based on the classifier-free approach, the model tends to generate information corresponding to positive text, i.e., background pixel information, while suppressing information corresponding to negative text, i.e., suppressing foreground object pixel information, thereby suppressing the problem of random generation of foreground objects.

[0459] The following describes the classifier-free model inference process provided in the embodiments of this application.

[0460] Step 5.1: Obtain the noise prediction value

[0461] Equation 1 below represents the input and output parameters of the diffusion model.

[0462] N = Unet(t, text, x) t (Equation 1)

[0463] Here, Unet() is the functional form of the diffusion model. t, text, and x trepresents the input parameters of the diffusion model Unet. N represents the output parameters of the diffusion model Unet.

[0464] Specifically, t represents the time step, text represents text input; x t This represents the implicit representation of the image at each time step t. N represents the noise prediction value of the diffusion model Unet at each time step t.

[0465] In the embodiments of this application, the diffusion model Unet() can predict noise values ​​based on different text inputs, such as obtaining foreground noise prediction values ​​or background noise prediction values.

[0466] On the one hand, for cases where the text input is positive text, replace "text" in Equation 1 above with the positive text "text". P Referring to Equation 2 below, the diffusion model Unet() can be based on the positive text text at each time step t. P Noise prediction is performed using (including CTXT and background category information) to obtain the background noise prediction value (denoted as Npos).

[0467] Npos = Unet(t, text) P x t (Equation 2)

[0468] On the other hand, for cases where the text input is negative text, replace "text" in Equation 1 above with the negative text "text". N Referring to Equation 3 below, the diffusion model Unet() can be based on negative text at each time step. N (Including Obj and foreground category information) to perform noise prediction, and obtain the foreground noise prediction value (denoted as Nneg).

[0469] Nneg = Unet(t, text) N x t (Equation 3)

[0470] In this embodiment of the application, given the predicted background noise value Npos and the predicted foreground noise value Nneg, the diffusion model Unet() can perform a classifier-free algorithm based on the predicted background noise value Nneg and the predicted foreground noise value Npos to obtain the predicted image noise value.

[0471] For example, the equation for the classifier-free algorithm can be found in Equation 4 below. The diffusion model Unet() can be calculated based on the background noise prediction value Nneg, the foreground noise prediction value Npos, and the hyperparameter α to obtain the image noise prediction value (denoted as Npred).

[0472] Npred=Nneg+α*(Npos-Nneg) (Equation 4)

[0473] Where Npred represents the image noise prediction value obtained by the diffusion model at each time step, Npos represents the background noise prediction value when the input text is positive text, and Nneg represents the foreground noise prediction value when the input text is negative text. α is a hyperparameter. For example, α can be set to 7.5.

[0474] In this embodiment, panoramic segmentation is performed on the image input to the diffusion model to obtain background and foreground category information. The background text (CTXT) and background category information are used as positive text, while the foreground text (Obj) and foreground category information are used as negative text. Then, classifier-free model inference is performed based on the positive and negative texts. During inference, the background noise prediction value of the diffusion model at each time step is obtained based on the positive text, and the foreground noise prediction value is obtained based on the negative text. Based on the background and foreground noise prediction values ​​at each time step, the image noise prediction value of the diffusion model at each time step can be calculated. This method enhances the background noise prediction value and suppresses the foreground noise prediction value, making the image inpainting process more inclined to reference background pixels and suppressing reference to foreground object pixels, thereby avoiding the problem of regenerating foreground objects in the removed areas.

[0475] The above describes the image noise prediction values ​​at each time step based on positive and negative text provided in the embodiments of this application. The process of repairing the image based on the noise prediction values ​​is described below.

[0476] Step 5.2: Obtain the RGB image based on the noise prediction value.

[0477] It should be noted that the image noise prediction value of the diffusion model at each time step follows a Gaussian distribution, denoted as N(0,1).

[0478] First, sample the initial noise x from the Gaussian distribution N(0,1). T Where T represents the initial sampling time step, and T can be set to 1000.

[0479] Then, iterate through the time steps t in Equation 5 from T to 1:

[0480] (1) When t = 1, let z = 0. When t > 1, sample random noise z from the Gaussian distribution N(0, 1). Figure 15D The results of multi-noise sampling for the diffusion model are shown.

[0481] (2) Predict the image feature x at the next time step using Equation 5 t-1 :

[0482]

[0483] α t =1-β t ,

[0484]

[0485] Where, β t Through β T The difference between β1 and β2 is obtained, β2 T =0.02, β1 = 1e-4.

[0486] It should be noted that, referring to Equation 1 above, Npred represents the image noise prediction value of the diffusion model at each time step, and N in Equation 5... t Let Npred be the value at step t.

[0487] Where, x t x represents the image features at the current time step t. t-1 The image features represent the next time step (t-1).

[0488] (3) By using Equation 5, iterating from T to 1 through t, the image feature x at each time step can be predicted. 999 x 998 , ..., x1, and finally obtain the image feature x0 at time step 0.

[0489] Image feature x0 is input into the decoding unit of the diffusion model. The decoding unit decodes the image feature x0 to obtain an RGB image.

[0490] The proposed solution involves inputting the first image and the mask of the object to be eliminated into the SD image elimination model. After model inference, an RGB image after image elimination and restoration processing can be output.

[0491] For example, suppose portrait 1 in the first image is selected as the object to be eliminated. The first image and the mask of portrait 1 are input into the SD image elimination model trained above. In the SD image elimination model, the first image is segmented panoramically to obtain positive and negative text. Then, the positive and negative text are used to complete model inference based on classifier-free guidance. The model inference can generate an RGB image in which portrait 1 is no longer displayed, that is, portrait 1 is eliminated, and the area where portrait 1 is eliminated is restored as the background image. No other portraits or objects are generated.

[0492] In the above Figure 12 , Figure 13B and Figure 13CIn the scenario shown, the SD image elimination model provided in the first embodiment of this application can be used for model inference, which can effectively avoid the problem of regenerating foreground objects in the eliminated area.

[0493] Second embodiment: Adaptive mask expansion based on brightness contrast

[0494] The second embodiment is an improvement on the model inference stage of the first embodiment. The second embodiment can employ adaptive mask expansion based on brightness contrast, which can further suppress the regeneration of foreground objects.

[0495] The second embodiment provides an improved solution based on the fact that, after multiple experimental tests, it was found that in some special cases, there are shadow areas near the object to be eliminated in the original image. If the initial mask of the input diffusion model does not cover the shadow areas near the object to be eliminated in the image, the diffusion model is still prone to generating random foregrounds.

[0496] For example, refer to Figure 16A As shown in (a) and (b), the object to be removed is a car, and there is a shadowed area near the object. The initial mask does not cover the shadowed area near the object (e.g., the car) in the image. (Reference) Figure 16A As shown in (c), when the initial mask is input into the diffusion model and sampled multiple times, other image features (e.g., other cars) are regenerated in the areas where the car was eliminated, resulting in the problem of random foreground generation. (See reference...) Figure 16A As shown in (d) and (e), by expanding the mask region to cover the shadow region, inputting the expanded mask into the diffusion model and sampling it multiple times, no other image features are generated where the car is eliminated in the multiple sampling results, and the phenomenon of random foreground generation is significantly improved.

[0497] Multiple tests revealed that residual information about objects (such as shadows) can cause serious foreground regeneration problems.

[0498] Based on this discovery, embodiments of this application propose an adaptive mask expansion implementation method based on brightness contrast to avoid the phenomenon of random foreground generation. Figure 16B A schematic diagram of the adaptive mask expansion based on brightness contrast is shown, which includes the following steps:

[0499] Step 1: Panoramic Image Segmentation

[0500] refer to Figure 16B As shown, panoramic segmentation is performed on the input image I to obtain the foreground object mask and the background mask (denoted as M_back). The dark area represents the background mask, and the colored area represents the foreground object mask.

[0501] In this embodiment of the application, a foreground object mask and a background mask are obtained through panoramic image segmentation, thereby determining the foreground object region and the background region in the image, which is used to limit the expansion of the mask.

[0502] Specifically, when inflating the initial mask M of the object to be eliminated, the inflating of the mask can be limited based on the foreground object mask and the background mask to prevent the inflated mask from covering the foreground object area. Finally, the area that may be a shadow is added to the final mask (denoted as M_final).

[0503] It should be noted that if the mask covers a foreground object region after dilation, the diffusion model will eliminate that foreground object when processing the image based on the dilated mask. Therefore, during mask dilation, the initial mask M is dilated, but the constraint must be met: the dilated mask cannot cover the foreground object region to avoid the problem of mistakenly eliminating other foreground objects.

[0504] Step 2: Inflate the initial mask

[0505] The initial mask refers to the mask of the object to be removed in the input image.

[0506] In this embodiment of the application, the size of the expansion kernel is set to b, and the initial mask M is expanded to obtain the expanded mask (denoted as M_hat). The expanded mask M_hat (the first mask) can cover other areas near the area to be eliminated.

[0507] It should be noted that, since shadows are usually closely connected to objects, the purpose of dilating the input mask is to include nearby possible shadow areas through mask dilution.

[0508] In this embodiment, the intersection of the dilated mask M_hat and the background mask M_back can be obtained to get the intersection mask (second mask). This intersection mask does not include other foreground objects, thus avoiding the false elimination of foreground objects.

[0509] The process of finding the intersection described above can be represented as: M h ′ at =M hat ∩M back .

[0510] Among them, M h ′ at M represents the intersection mask. hat M represents the expanded mask. back This represents the background mask.

[0511] Step 3: Pixel clustering based on superpixels

[0512] Herein, pixel clustering is a technology that divides pixels in an image into different groups or categories, and is generally used to identify pixels with similar brightness or color features.

[0513] In the embodiment of the present application, the initial mask M is dilated to obtain a dilated mask M_hat, then a difference operation is performed on the dilated mask M_hat and the initial mask M to obtain a mask M_res (the third mask). As shown in Figure 16B , the yellow area around the object to be eliminated is the mask M_res obtained after the difference operation.

[0514] In the embodiment of the present application, after the mask M_res is obtained, superpixel clustering is performed on the mask M_res, so as to partition the mask M_res to obtain a plurality of pixel blocks. For example, the number of clustering centers may be set to 10. As shown in Figure 16B the yellow area in the drawing, the dilated area is clustered into a plurality of pixel blocks after clustering.

[0515] Step 4: Select the required expansion region based on brightness contrast.

[0516] In the embodiment of the present application, after the mask M_res is divided into a plurality of pixel blocks, a brightness mean value (denoted as L_i) is calculated for each pixel block, and then a brightness mean value (denoted as L) of the entire image is calculated.

[0517] Referring to Figure 16B , the brightness mean L_i of each pixel block is compared with the brightness mean L of the entire image.

[0518] If L_i < L, it is determined that the current pixel block most probably belongs to a shadow region, and therefore the pixel block is retained in the mask M_res.

[0519] If L_i ≥ L, it is determined that the current pixel block does not belong to a shadow region, and therefore the pixel block is removed from the mask M_res to obtain a processed mask M_res.

[0520] Referring to Figure 16B , the final mask M_final includes the initial mask M and the processed mask M_res.

[0521] Through the above processing, the initial mask is dilated, and possible shadow regions nearby are wrapped through mask dilation, while ensuring that the dilated region does not cover the foreground object region, so as to avoid the problem of mistakenly deleting other foreground objects. Herein, the dilated region is subjected to block processing, and the brightness mean of each pixel block is calculated. If the brightness mean of a certain pixel block is greater than or equal to the brightness mean of the entire image, the pixel block is not a shadow region. In the final mask M_final, this pixel block is excluded.

[0522] The embodiments of this application employ adaptive mask expansion based on brightness contrast, so that the expanded mask includes possible shadow areas near the object to be eliminated, which can suppress the problem of foreground object regeneration caused by shadows.

[0523] The second embodiment will be described in conjunction with the first embodiment. Applying the final mask M_final from the second embodiment to the model training and inference in the first embodiment can suppress the problem of foreground object regeneration caused by shadows. Figure 16C The results of multi-noise sampling are shown when adaptive masking based on brightness contrast is employed. For example... Figure 16C As shown, no foreground object regeneration problem occurred in any of the images from the multi-noise sampling results.

[0524] It should be noted that the adaptive mask expansion scheme based on brightness contrast provided in the second embodiment of this application is not limited to the SD image elimination model trained in the first embodiment. In practical applications, the adaptive mask expansion scheme based on brightness contrast can also be applied to conventional SD image elimination models to achieve the purpose of suppressing the problem of foreground object regeneration.

[0525] In some embodiments, such as those described above Figure 12 , Figure 13B and Figure 13C In the scenario shown, the SD image elimination model provided in the first embodiment of this application can be used for model inference, and an adaptive mask expansion scheme based on brightness contrast can be adopted during the inference process, which can more effectively avoid the problem of regenerating foreground objects in the eliminated area.

[0526] Third embodiment: Self-attention layer suppression based on panoptic segmentation

[0527] The third embodiment is an improvement on the model inference stage of the first embodiment. The third embodiment can use self-attention layer suppression based on panoramic segmentation, which can further suppress the problem of foreground object regeneration.

[0528] The third embodiment provides an improved solution because, during the diffusion model inference process, it was discovered that the self-attention layer in the diffusion model might excessively reference foreground objects in the image when regenerating the image within the elimination region, leading to random generation problems. Multiple tests revealed that foreground feature information outside the region to be eliminated has an overly strong effect on the restoration of the region.

[0529] In some scenarios, when there are foreground objects in the image that are similar to the object to be eliminated, the foreground feature information other than the area to be eliminated has a particularly strong effect on the restoration of the area to be eliminated.

[0530] For example, if an image contains multiple portraits and one of them needs to be removed, the image will be regenerated by referencing the other portraits during the restoration phase after the removal of that portrait. Therefore, it is very likely that an image of a person will be generated in the area where the image was removed.

[0531] For example, if an image contains multiple vehicles, and one of the vehicles needs to be removed, the image will be regenerated by referencing the other vehicles during the restoration phase after the vehicle is removed. Therefore, it is very likely that an image of a vehicle will be generated in the area where the vehicle was removed.

[0532] Based on this, this application proposes a self-attention layer suppression based on panoramic segmentation. During the inference stage, the self-attention heatmap can be processed by an algorithm to suppress references to foreground objects and avoid random generation problems.

[0533] First, it should be noted that the third embodiment involves a self-attention heatmap, where each value represents the similarity between the current point and other points in the image. In the self-attention layer, the heatmap represents the degree to which information from other regions in the image is referenced during the generation of the current region. The higher the value of a point in the heatmap, the stronger the reference to that point's information is during the generation of the current region. This embodiment can process the self-attention heatmap using an algorithm during the inference stage to reduce the pixel values ​​of foreground object regions. This reduces the reference intensity to those foreground object regions during image regeneration, thereby suppressing excessive reference to foreground objects and avoiding haphazard generation.

[0534] Figure 17A A schematic diagram of a heatmap is shown. (For example...) Figure 17A As shown, the original image can be divided into N regions. N is an integer greater than 1; the larger N is, the better the image processing effect. Here, we take N=9 as an example for illustration. The original image includes 9 pixel blocks, and each pixel block has a heatmap, so the original image includes a total of 9 heatmaps. Each heatmap includes 9 values.

[0535] Each value in the heatmap represents the similarity between the current point and other points in the image; therefore, each value in the heatmap can be represented by a similarity value. The similarity value ranges from [0,1].

[0536] For ease of explanation, we use "sim" to represent the similarity calculation method, which typically employs the cosine function at the feature level. For example... Figure 17A The diagram schematically illustrates the heatmaps of pixel block 1, pixel block 2, and pixel block 3. For example, the heatmap of pixel block 1 will be used as an example. 1sim1 represents the similarity between pixel blocks 1 and 1sim2, ..., 1sim9 represents the similarity between pixel blocks 1 and 1sim9.

[0537] It should be noted that in practical applications, the similarity values ​​of [0,1] can be converted into grayscale values ​​of [0,255] to convert the similarity heatmap into a grayscale heatmap, which will facilitate further image processing.

[0538] like Figure 17B As shown in the embodiments of this application, a scheme for self-attention layer suppression based on panoramic segmentation is proposed, which may include the following steps:

[0539] Step 1: Panoramic Image Segmentation

[0540] Perform panoramic segmentation on the input image to obtain the foreground object region and the background region. (Reference) Figure 17C The input image is segmented into a panoramic view to obtain foreground and background regions. Colored regions are designated as foreground object regions, while other regions are designated as background regions. Then, the area within the foreground object region excluding the object to be removed is defined as the foreground object mask M. obj .

[0541] Step 2: Obtain the heatmap related to the region to be eliminated in the diffusion model.

[0542] In this embodiment, heatmaps from the attention layer are extracted from the conversion module in the diffusion model. Each heatmap represents the similarity between a point in the image and other points. For example, at a 32×32 scale, the conversion module contains 32×32=1024 heatmaps.

[0543] For example, refer to Figure 17D In (a), the input image is divided into 9 pixel blocks, thus yielding 9 heatmaps. (See reference) Figure 17D In step (b), only heatmaps belonging to the region to be eliminated are selected from the nine heatmaps. For example, heatmaps containing pixel blocks 4, 5, 7, and 8 belong to the region to be eliminated. The average similarity values ​​of these heatmaps are calculated to obtain a second heatmap, which is called a similarity-based heatmap. Here, A1 represents the average similarity of 4sim1, 5sim1, 7sim1, and 8sim1; A2 represents the average similarity of 4sim2, 5sim2, 7sim2, and 8sim2; and so on, with A9 representing the average similarity of 4sim9, 5sim9, 7sim9, and 8sim9.

[0544] refer to Figure 17D (c) in the diagram converts the heatmap in similarity format into a grayscale heatmap (Attn). mAs can be seen, the area within the red box in the grayscale heatmap has higher brightness, and this red box contains the foreground object (car). It should be noted that the higher the value of a point in the heatmap, the greater the model's reference to that point when generating the elimination region. This embodiment of the application reduces the value of the red box region to suppress the reference to pixels in that red box region when generating the elimination region.

[0545] It should be noted that the above example of averaging the similarity and then converting it to a grayscale image illustrates the heatmap processing process. In actual implementation, the following method can also be used: convert the heatmap corresponding to each pixel block into a grayscale image, and then average the grayscale images corresponding to multiple pixel blocks.

[0546] Step 3: Adjust the heatmap based on the foreground object mask and correction factor.

[0547] In this embodiment of the application, the following Equation 6 is used to process the heatmap corresponding to the region to be eliminated:

[0548] Attn′ m =Attn m *M obj *c(Equation 6)

[0549] Among them, Attn m This indicates that multiple heatmaps of each pixel block within the region to be eliminated are combined to form a heatmap (called the second heatmap), Attn′ m This represents the corrected heatmap (referred to as the first heatmap), M obj represents the mask for the foreground object regions other than the region to be eliminated, and c represents the correction coefficient.

[0550] Where c < 1, for example, c is 0.1, and the specific value is set according to the actual situation.

[0551] Figure 17E A schematic diagram of adjusting the heatmap is shown for reference. Figure 17E In (a), the foreground object mask M obj Correction factor c and heatmap Attn in grayscale form m Multiplying them together yields the corrected heatmap Attn′. m (Referred to as the first heatmap). Reference Figure 17E In (b), the brightness of the area within the green box increases; this green box corresponds to the background area in the input image. Figure 17D Compared to the red box area in (c) of the above, Figure 17E The brightness of the area in the red box in (b) is reduced.

[0552] It should be noted that the pixel values ​​of the red box area (foreground object) in the heatmap are reduced, thus decreasing the reference strength of the pixels in the red box area when the image is repaired by the image removal model; the pixel values ​​of the green box area (including background texture) in the heatmap are reduced, thus increasing the reference strength of the pixels in the green box area when the image is repaired by the image removal model.

[0553] By using the above correction method, the reference intensity of the heatmap of the region to be eliminated to other foreground objects can be reduced. That is, when the diffusion model generates a background image by eliminating the region, it will reduce the reference to other foreground object regions, thereby reducing the phenomenon of random generation by the diffusion model.

[0554] It should be noted that the above steps mainly corrected the self-attention layer within the transformation module of the U-net network in the diffusion model, resulting in the corrected heatmap Attn′. m For ease of explanation, the conversion module is based on the corrected heatmap Attn′. m Each suppression operation is denoted as TFB(). TFB1() represents the first suppression operation, and so on, TFB... M () indicates the Mth suppression operation.

[0555] Using Equation 7, the conversion module adjusts the heatmap Attn′ based on the corrected heatmap. m Perform a suppression operation.

[0556] TFB = (Attn) m ′, text, x t (equation 7)

[0557] Equation 8 is used to perform M suppression operations and input the suppression results into the U-net network.

[0558] Unet = TFB1(TFB2(...TFB) M (t, text, x) t Equation 8)

[0559] Then, following the algorithm provided in the first embodiment, based on Equations 1, 2, 3 and 4 above, the U-net network predicts the image noise prediction value at each time step. Then, based on Equation 6, the RGB image is obtained according to the noise prediction value. Thus, through model inference, the RGB image after image removal and restoration processing can be output.

[0560] The third embodiment will be described in conjunction with the first embodiment. After obtaining the image noise prediction value of the diffusion model at each time step according to the self-attention layer suppression method provided in the third embodiment, the model inference process in the first embodiment can be followed, and the image noise prediction value can be used for model inference to obtain the RGB image. Applying the image noise prediction value obtained in the third embodiment to the model inference in the first embodiment can suppress the problem of foreground object regeneration caused by shadows.

[0561] Figure 17F The diagram illustrates the multi-noise sampling result obtained by applying the image noise prediction value obtained after self-attention layer suppression in the third embodiment to the model inference in the first embodiment. For example... Figure 17F As shown, based on the corrected heatmap, the multi-noise sampling results were obtained. In each image of the multi-noise sampling results, there was no problem of foreground object regeneration.

[0562] It should be noted that the self-attention layer suppression scheme based on panoptic segmentation provided in the third embodiment of this application is not limited to the SD image removal model trained in the first embodiment. In practical applications, the self-attention layer suppression scheme based on panoptic segmentation can also be applied to conventional SD image removal models to achieve the purpose of suppressing the problem of foreground object regeneration.

[0563] In some embodiments, such as those described above Figure 12 , Figure 13B and Figure 13C In the scenario shown, the SD image elimination model provided in the first embodiment of this application can be used for model inference, and a self-attention layer suppression scheme based on panoramic segmentation can be used during the inference process, which can effectively avoid the problem of regenerating foreground objects in the eliminated area.

[0564] In other embodiments, the SD image elimination model provided in the first embodiment of this application can be used for model inference, and an adaptive mask expansion scheme based on brightness contrast and a self-attention layer suppression scheme based on panoramic segmentation can be used during the inference process, which can more effectively avoid the problem of regenerating foreground objects in the eliminated area.

[0565] Determine if the elimination condition is met.

[0566] In some embodiments, in scenarios where objects to be eliminated are selected by circling or smearing in the original image, in order to avoid the elimination of part of the human body image due to the circling or smearing area being located on the human body, resulting in a fragmented human body image and affecting the elimination effect, the solution of this application can first determine whether the circling or smearing area meets the elimination conditions. If the circling or smearing area meets the elimination conditions, then the elimination and repair process is performed. If the circling or smearing area does not meet the elimination conditions, then the elimination and repair process is not performed. This can optimize the effect of elimination and repair processing.

[0567] For example, combined Figure 5 ,like Figure 18 As shown, after S302 and before S303, the method also includes S305 and S306.

[0568] S305. Obtain a second mask image and a third mask image based on the first image. The second mask image includes mask regions of multiple human figures, and the third mask image includes mask regions corresponding to the selected regions.

[0569] The second mask image includes the mask region corresponding to each portrait in the first image.

[0570] In some embodiments, the first image can be input into a human instance segmentation network to obtain a second mask image. The second mask image can also be called a human instance segmentation mask image (HumanInstanceMask). The human instance segmentation network is a network model used to identify human figures or human bodies in the first image and generate a mask image for each human figure.

[0571] In this process, after inputting the first image into the human image instance segmentation network model, a mask image of each human image instance in the first image can be obtained. By combining the mask images of all human image instances in the first image, a second mask image is obtained.

[0572] In other embodiments, the first image can be input into a panoramic segmentation network model to obtain a human image instance segmentation mask image. The panoramic segmentation network model is a network model used to identify people or objects in the first image and then generate a mask image for each person or object.

[0573] In this process, after inputting the first image into the panoramic segmentation network model, a mask image of each person or object in the first image can be obtained. By combining the mask images of all people in the first image, a second mask image is obtained.

[0574] Figure 19 A schematic diagram illustrating the acquisition of a second mask image based on a first image is shown. For example... Figure 19As shown, the first image includes three portraits: portrait 1, portrait 2, and portrait 3; correspondingly, the second mask image generated based on the first image includes three portrait mask regions: mask 1, mask 2, and mask 3, which are the mask regions corresponding to the three portraits respectively.

[0575] It should be noted that if the second mask image is empty, that is, there is no mask area or no area marked as 1, then it means that there is no human image in the first image. If the second mask image is not empty, that is, there is a mask area or an area marked as 1, then it means that there is at least one human image in the first image.

[0576] S306. Based on the third mask image and the second mask image, determine whether the selected area or the painted area meets the elimination conditions.

[0577] It should be noted that the purpose of determining whether the selected or painted area meets the elimination criteria is to avoid the removal of parts of the human body image due to the selected or painted area being located on the human body, resulting in incomplete human body images and affecting the elimination effect.

[0578] Figure 20 The flowchart illustrates the process of determining whether a selected or smeared area meets the elimination criteria based on a third mask image and a second mask image. It is understood that further selection of elimination and repair methods is only permitted when the selected or smeared area is determined to meet the elimination criteria, allowing for image elimination and repair processing of the object to be eliminated. If the selected or smeared area does not meet the elimination criteria, an invalid selection operation can be indicated. This avoids the elimination of parts of the human body image due to the selected area being located within the human body, thus improving the elimination and repair effect for portraits.

[0579] In some embodiments, if the second mask image is empty, that is, there is no human image in the first image, then the object to be eliminated will not be a human body. In other words, the object to be eliminated may be an item such as a trash can or debris, which meets the elimination conditions. Therefore, elimination and repair processing can be performed on the object to be eliminated in the selected area of ​​the first image.

[0580] If the second mask image is not empty, meaning there is a human figure in the first image, then obtain the following parameters:

[0581] (1) Calculate the area S of the mask region of the third mask image (which corresponds to the selected region in the first image). The area of ​​the first mask region can be determined by counting the number of pixels in the first mask region. For example, refer to... Figure 10 As shown, the white area in the third mask image is the mask area. Calculate the number of pixels in this mask area to obtain the number of pixels S in the selected or smeared area.

[0582] (2) Calculate the area Si of each portrait mask region in the second mask image. For example, refer to... Figure 10 As shown, the second mask image includes three portrait mask regions: mask 1, mask 2, and mask 3. The number of pixels in mask 1 is denoted as S1, the number of pixels in mask 2 is denoted as S2, and the number of pixels in mask 3 is denoted as S3. Si includes S1, S2, and S3.

[0583] (3) Calculate the area S′ of the intersection region between the first mask region and each portrait mask region in the second mask image. The number of pixels in the intersection region between the first mask region and mask 1 is denoted as S1′, the number of pixels in the intersection region between the first mask region and mask 2 is denoted as S2′, and the number of pixels in the intersection region between the first mask region and mask 3 is denoted as S3′. Si′ includes S1′, S2′, and S3′.

[0584] Among them, the first mask region and mask 2 have an overlapping region (such as... Figure 21A (As shown by the diagonal lines in the diagram), the number of pixels in the intersection region S2′ is greater than zero. The first mask region has no intersection with either mask 1 or mask 3, and the number of pixels in the intersection region S2′=0 and S3′=0.

[0585] In the embodiments of this application, after calculating parameters such as S, Si (including S1, S1 and S3) and Si′ (including S1′, S2′ and S3′), it is determined whether the selected area or the smeared area meets the elimination condition in the following manner. i is taken as 1, 2, and 3 in sequence.

[0586] If Si′ / Si∈(N1,N2) and Si′ / S>N3, then the selected or painted area does not meet the elimination conditions, and the elimination process is not performed on the object to be eliminated.

[0587] Optionally, N1 can be 1%, N2 can be 50%, and N3 can be 80%. For ease of explanation, the following embodiments are illustrated with N1 as 1%, N2 as 50%, and N3 as 80%. That is, when Si′ / Si∈(1%, 50%) and Si′ / S>80%, it is determined that the selected area or the smeared area does not meet the elimination condition.

[0588] It should be noted that Si′ / S can be understood as the area ratio of the intersection region Si′ to the selected or smeared region S. If this area ratio is less than or equal to 80%, it means that part of the selected or smeared region falls within the portrait area, and the object to be removed could be a portrait or other objects. If this area ratio is greater than 80%, it means that most or all of the selected area falls within the portrait area, allowing for a more accurate identification of the object to be removed as a portrait.

[0589] Si′ / Si can be understood as the area ratio of the intersection region Si′ to the human image Si. If this area ratio is within the range of (1%, 50%), it means that the selected area or the painted area covers part of the human image. In this case, the object to be removed should not be removed to avoid accidental removal of parts of the human body, resulting in incomplete human image and affecting the image removal effect.

[0590] In other words, if Si′ / S>80% and Si′ / Si∈(1%,50%), then it can be determined that the object to be eliminated is a human body and the selected or smeared area is part of the human body. Therefore, it is considered that the elimination conditions are not met and the object to be eliminated will not be eliminated.

[0591] If neither Si′ nor Si belongs to (1%, 50%), then the elimination condition is met, and the elimination process can be performed on the object to be eliminated.

[0592] It should be noted that the calculation can be performed on each portrait mask area. If the elimination conditions are not met, the calculation can be stopped and an invalid selection operation will be displayed. If the conditions are met after traversing and calculating all portrait mask areas, the selected or painted area is determined to meet the elimination conditions, and the elimination process can be performed on the object to be eliminated. By performing traversal calculations, the amount of computation can be reduced.

[0593] The solution proposed in this application can determine whether the selected or painted area meets the elimination conditions in response to the user's selection operation on the first image. If the selected or painted area meets the elimination conditions, the object to be eliminated is eliminated. If the selected or painted area does not meet the elimination conditions, the object to be eliminated is not eliminated, so as to prevent accidental elimination, such as preventing accidental elimination of human body parts in the image.

[0594] The following illustrations, with reference to the accompanying drawings, illustrate scenarios where the selected or painted area meets the elimination criteria and scenarios where the selected or painted area does not meet the elimination criteria.

[0595] Scenes where the selected area (circled area or painted area) meets the elimination criteria.

[0596] The following is for reference. Figure 21AThis diagram illustrates how to determine if a selected or painted area meets the elimination criteria. For example... Figure 21A As shown, the selected area in the first image is indicated by the dashed line. The number of pixels S of the object to be eliminated is determined by the mask area in the third mask image; the number of pixels S1, S2, and S3 of each portrait is determined by the mask area of ​​each portrait in the second mask image; the number of pixels S1′, S2′, and S3′ of the intersection area of ​​the first mask area and each portrait mask area is calculated. S1′ = 0, S3′ = 0.

[0597] like Figure 21A As shown, starting with i = 1, the traversal begins. Since S1′ = 0, S1′ / S1 = 0, and S1′ / S = 0. The traversal continues. Then, i = 2, S2′ / S2 > 50%, and S2′ / S > 80%. The traversal continues. Then, i = 3, since S3′ = 0, S3′ / S3 = 0, and S3′ / S = 0. The traversal calculation ends. Si′ / Si does not belong to (1%, 50%), satisfying the elimination condition. Therefore, the elimination process can be performed on the object to be eliminated.

[0598] Scenes where the selected area (circled area or painted area) does not meet the elimination criteria.

[0599] The following is for reference. Figure 21B This diagram illustrates how to determine if a selected or painted area does not meet the elimination criteria. For example... Figure 21B As shown, the selected area in the first image is indicated by the dashed line. The number of pixels S of the object to be eliminated can be determined by the mask area in the third mask image. The number of pixels S1, S2, and S3 of each portrait can be determined by the mask area of ​​each portrait in the second mask image. The number of pixels S1′, S2′, and S3′ of the intersection area of ​​the first mask area and each portrait mask area can be calculated.

[0600] like Figure 21B As shown, the traversal calculation starts from i=1. Since the number of pixels in portrait 1, S1′=0, therefore S1′ / S1=0, S1′ / S=0. The traversal continues. Then i=2, S2′ / S2∈(1%, 50%) and S2′ / S>80%, it can be determined that the object to be eliminated is a human body and the selected area is part of the human body. Therefore, the elimination condition is not met, and the traversal calculation stops. In the case that the elimination condition is not met, the elimination process is not performed on the object to be eliminated. This can avoid the situation where part of the human body image is eliminated because the selected area is located in the human body.

[0601] Optionally, if a certain portrait mask region has an area ratio of less than N4 in all mask regions of the second mask image, meaning the portrait mask region is too small, then it can be excluded from the above traversal calculation, saving computational resources and speeding up the calculation. Optionally, N4 can be 1%. The following example uses N4 as 1% as an illustration.

[0602] For example, refer to the above Figure 21A If the electronic device determines that the area of ​​mask 3 in the second mask image is less than 1%, then the electronic device does not need to perform traversal calculations on mask 3.

[0603] Specifically, starting with i=1, the calculation iterates through the image. Since the number of pixels in portrait 1, S1′=0, therefore S1′ / S1=0 and S1′ / S=0, satisfying the elimination condition. Then, when i=2, S2′ / S2>50% and S2′ / S>80%, satisfying the elimination condition again. Mask 3's area is too small (less than 1%), so it is not calculated. The calculation ends. If all elimination conditions are met after the calculation, the object to be eliminated can be processed.

[0604] It should be noted that various possible methods can be used to "determine whether the selected area or the painted area meets the elimination conditions," and this application does not limit this. For example, embodiments of this application also provide the following implementation method to determine whether the selected area or the painted area meets the elimination conditions.

[0605] For example, after calculating the number of pixels S of the object to be eliminated, the number of pixels Si of each portrait (including S1, S2, and S3), and the number of pixels Si′ of the intersection region (including S1′, S2′, and S3′), the selected area or the painted area is judged to meet the elimination conditions in the following manner. i is taken as 1, 2, and 3 in sequence.

[0606] In some embodiments, if the ratio of the number of pixels Si′ in the intersection region to the number of pixels Si in the portrait region is within the range of (N1, N2) and the BBOX ratio is greater than N5, then the selected region or the smeared region is determined not to meet the elimination conditions. This situation usually occurs when the selected region or the smeared region contains a human body and would cause an elimination effect on the human body area, therefore, the elimination process is not performed on the object to be eliminated.

[0607] BBOX is the width of the bounding rectangle of the intersection area of ​​portrait i, divided by the width of the bounding rectangle of the mask of portrait i.

[0608] The intersection area of ​​portrait i refers to the intersection area between the mask area of ​​portrait i and the first mask area.

[0609] Let Y(i) denote the BBOX ratio. Y(i) = BBOX(i′).width / BBOX(i).width.

[0610] Here, BBOX represents the outer rectangle. BBOX(i).width is the width of the outer rectangle of the mask for each portrait. BBOX(i′).width represents the width of the outer rectangle of the intersection area of ​​the portraits.

[0611] Figure 21C A diagram illustrating the percentage of BBOX is shown, as follows: Figure 21C As shown, it can be determined that mask2 and the first mask region have an intersection area, and the proportion of the intersection area is Y(2)=BBOX(2′).width / BBOX(2).width.

[0612] Optionally, N5 can be 30%.

[0613] It's understandable that when the BBOX percentage is greater than 30%, it means that more than 30% of the selected or smeared area falls within the area containing the human image. If the number of pixels in the intersection area Si′ / the number of pixels in the human image area Si is within the range of (1%, 50%), then it can be determined that the object to be removed is a human body and the selected area is part of the human body. Therefore, it can be determined that the selected or smeared area does not meet the removal conditions. This situation usually occurs when the selected or smeared area contains a human body and will cause a removal effect on the human body area; therefore, removal processing is not performed on the object to be removed.

[0614] In other embodiments, if the elimination conditions are met after the traversal calculation is completed, this usually means that the selected area or the smeared area does not contain a human body (i.e., the object to be eliminated is not a human body), or the selected area or the smeared area contains a human body but will not have an elimination effect on the human body area. Therefore, the object to be eliminated can be eliminated.

[0615] Figure 22 A comparative diagram is shown to determine whether the selected area meets the elimination criteria.

[0616] like Figure 22 As shown in (a), the selected region 21 surrounds the human body image. Through the above judgment process, it can be determined that the selected region meets the elimination conditions, so elimination and repair processing is performed.

[0617] like Figure 22 As shown in (b) above, the dashed box 22 is the selected area in the first image. The selected area does not completely surround the human figure, but only covers a part of the human body. Through the above judgment process, it can be determined that the selected area does not meet the elimination condition. In this case, the electronic device can prompt that the selection operation is invalid.

[0618] In this embodiment of the application, if the selected area (e.g., the circled area or the painted area) does not meet the elimination conditions, the electronic device may display a prompt message, such as "This photo does not support elimination".

[0619] For example, such as Figure 23A As shown in (a) and (b) of the image, in the scenario of AI elimination through intelligent selection, in response to the user's selection operation in the first image, the electronic device displays the selection trajectory 51 in the first image. Through the solution of this application, the electronic device determines that the selected area corresponding to the selection trajectory 51 is a human body area, which does not meet the elimination conditions, as shown in (a) and (b). Figure 23A As shown in (c), the electronic device displays the message 52 "Prompt: This photo does not support deletion".

[0620] For example, such as Figure 23B As shown in (a) and (b) of the image, in a scenario where AI removal is performed by manual smearing, in response to the user's manual smearing operation in the first image, the electronic device displays smear mark 53 in the first image. Using the solution of this application, the electronic device determines that the area corresponding to smear mark 53 is a human body area and does not meet the removal conditions, as shown in (a) and (b). Figure 23B As shown in (c), the electronic device displays the message 54 "Prompt: This photo does not support deletion".

[0621] The proposed solution responds to a user's selection or smearing operation on the first image, and continues to perform image removal and repair if the selected or smeared area meets the removal conditions.

[0622] It should be noted that in the embodiments of this application, "greater than" can be replaced with "greater than or equal to", "less than or equal to" can be replaced with "less than", or "greater than or equal to" can be replaced with "greater than", and "less than" can be replaced with "less than or equal to".

[0623] The various embodiments described herein can be independent solutions or combinations thereof based on their inherent logic, and all such solutions fall within the protection scope of this application.

[0624] The foregoing mainly describes the solutions provided by the embodiments of this application from the perspective of method steps. It is understood that, in order to achieve the above functions, the electronic device implementing this method includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of protection of this application.

[0625] This application embodiment can divide an electronic device into functional modules based on the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware or as software functional modules. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, other feasible division methods may exist. The following description uses the division of functional modules according to each function as an example.

[0626] Figure 24 This is a schematic block diagram of an image removal and restoration apparatus 500 provided in an embodiment of this application. The apparatus 500 can be used to perform the actions performed by the electronic device in the above method embodiments. The apparatus 500 includes an image display unit 510 and an image processing unit 520.

[0627] The image display unit 520 is used to display the first image.

[0628] Image processing unit 520 is configured to respond to a user's first operation on a first image, determine the object to be eliminated in the first image; obtain a first mask image based on the first image, wherein the mask in the first mask image is the mask of the object to be eliminated; input the first image and the first mask image into a first image elimination model, perform image processing through the first image elimination model, and output a second image; the second image does not include the object to be eliminated and the area after the object to be eliminated is restored to the background texture;

[0629] The image processing using the first image elimination model includes: processing the first image based on the first mask image to obtain a first incomplete image, in which the object to be eliminated is eliminated; obtaining positive and negative text based on the first image, where the positive text is used to enhance the generation of background texture and the negative text is used to suppress the generation of foreground object texture; and performing model inference based on the positive and negative text to restore the eliminated area in the first incomplete image to the background texture.

[0630] The image removal and restoration apparatus provided in this application embodiment responds to a user's operation on a first image, determines the object to be removed in the first image, inputs the first image and a first mask image into an image removal model for processing, and after processing, the object to be removed in the image is removed, and the removed area is restored to background texture. This application's solution improves the image removal model used. The image removal model can obtain positive and negative text from the first image, and perform model inference based on the positive and negative text to regenerate image features in the removed area. Specifically, the model inference tends to generate background pixel information corresponding to positive text in the removed area, while suppressing the generation of foreground object pixel information corresponding to negative text. Therefore, after model inference, no other foreground object images such as portraits or objects are generated in the removed area, thereby improving the image removal and restoration effect.

[0631] The apparatus 500 according to the embodiments of this application can correspond to the execution of the method described in the embodiments of this application, and the above and other operations and / or functions of the units in the apparatus 500 are respectively for implementing the corresponding process of the method, which will not be described again here for the sake of brevity.

[0632] This application also provides a chip coupled to a memory, which is used to read and execute computer programs or instructions stored in the memory to perform the methods in the above embodiments.

[0633] This application also provides an electronic device including a chip for reading and executing computer programs or instructions stored in a memory, such that the methods in the various embodiments are performed.

[0634] This embodiment also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the image removal and repair method described in the above embodiment.

[0635] This embodiment also provides a computer program product. The computer-readable storage medium stores program code. When the computer program product is run on a computer, it causes the computer to perform the above-mentioned related steps to realize the image elimination and repair method in the above embodiment.

[0636] In this embodiment, the electronic device, computer-readable storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0637] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0638] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0639] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. In the description of embodiments in this application, unless otherwise stated, "multiple" means two or more; for example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0640] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image removal and restoration method, characterized in that, include: In response to a user’s first operation on a first image, determine the object to be eliminated in the first image; When a first foreground object with similar or the same category as the object to be eliminated is identified in the first image, panoramic segmentation processing is performed on the first image to obtain foreground category information and background category information of the first image; wherein, the foreground category information includes the object to be eliminated and the first foreground object, and the background category information includes other images in the first image other than the object to be eliminated and the first foreground object; Image noise prediction values ​​are obtained based on the positive and negative text of the first image; wherein, the positive text includes the background category information and background text of the first image, and the negative text includes the foreground category information and foreground text of the first image; Based on the image noise prediction value, perform first model inference without classifier guidance, and perform suppression operation based on the first heatmap. Apply the image noise prediction value obtained after suppression operation to the first model inference to obtain background texture; remove the object to be removed from the first image and display the background texture in the removed area to obtain a second image. The first heatmap represents the degree to which the region containing the object to be eliminated references the region containing the first foreground object during the regeneration process.

2. The method according to claim 1, characterized in that, The step of obtaining image noise prediction values ​​based on the positive and negative text of the first image, and performing first model inference based on the image noise prediction values, includes: Based on the positive text, the background noise prediction value of the first model at each time step is obtained, and based on the negative text, the foreground noise prediction value of the first model at each time step is obtained. Based on the background noise prediction value and foreground noise prediction value of the first model at each time step, the image noise prediction value of the first model at each time step is obtained. Based on the image noise prediction value, multiple noise sampling is performed to obtain a multiple noise sampling image, which is used to generate the background texture.

3. The method according to claim 1 or 2, characterized in that, The step of obtaining image noise prediction values ​​based on the positive and negative text of the first image includes: Based on the following equation, using a classifier-free approach, the predicted image noise value is calculated based on the positive and negative text: in, Npos This represents the predicted background noise value obtained based on the positive text prediction. Nneg This represents the foreground noise prediction value obtained based on the negative text prediction. α For hyperparameters, Npred This represents the image noise prediction value obtained by the first model at each time step.

4. The method according to any one of claims 1 to 3, characterized in that, Before obtaining the image noise prediction value based on the positive and negative text of the first image, the method further includes: The positive text is generated based on the background category information and the background text, and the negative text is generated based on the foreground category information and the foreground text. The positive text is used to enhance the generation of the background texture corresponding to the background category information, and the negative text is used to suppress the generation of the foreground object texture corresponding to the foreground category information. The background text and the foreground text are texts learned by the first model through training.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: A panoramic segmentation process is performed on the sample image to obtain a panoramic segmentation result, which includes a background region and multiple foreground instance regions. A sub-region of the background region is randomly selected as a background sample mask, and background text corresponding to the background sample mask is generated; the background sample mask and the background text are used as the training set, and the sample image is used as the supervision signal for model training. Randomly select one instance region from the plurality of foreground instance regions as a foreground sample mask, and generate foreground text corresponding to the foreground sample mask; use the foreground sample mask and the foreground text as a training set, and use the sample image as a supervision signal to train the model; The first model is obtained after model training.

6. The method according to any one of claims 1 to 5, characterized in that, After determining the object to be eliminated in the first image, the method further includes: The first image identifies a shadowed region around the object to be eliminated. The mask of the object to be eliminated is dilated, and the dilated mask includes the shadow area. The step of eliminating the object to be eliminated in the first image and displaying the background texture in the eliminated area includes: processing the first image based on the mask after the dilation process, so that the object to be eliminated and the shadow area are eliminated, and displaying the background texture in the eliminated area.

7. The method according to claim 6, characterized in that, The process of expanding the mask of the object to be eliminated includes: Perform panoramic segmentation on the first image to obtain the foreground object mask and the background mask; The mask of the object to be eliminated is subjected to a first dilation process to obtain a first mask; the intersection of the first mask and the background mask is obtained to obtain a second mask; the difference between the second mask and the mask of the object to be eliminated is obtained to obtain a ring-shaped third mask. The third mask is subjected to pixel clustering processing to divide the region where the third mask is located into multiple pixel blocks; The average brightness of each pixel block in the plurality of pixel blocks is compared with the average brightness of the first image to obtain a processed third mask. The average brightness of the pixel blocks contained in the processed third mask is less than the average brightness of the first image. The processed third mask and the mask of the object to be eliminated are summed to obtain the mask after dilation.

8. The method according to any one of claims 1 to 7, characterized in that, After determining the object to be eliminated in the first image, the method further includes: Based on the mask of the first foreground object and the first adjustment coefficient, the heat map of the area where the object to be eliminated is located is processed to obtain a first heat map; wherein, the first adjustment coefficient is a value greater than 0 and less than 1.

9. The method according to claim 8, characterized in that, The step of processing the heatmap of the region where the object to be eliminated is located based on the mask of the first foreground object and the first adjustment coefficient to obtain the first heatmap includes: Heatmaps of each region of the first image are obtained from the self-attention layer of the first model; The average of multiple heatmaps of the region where the object to be eliminated is located in the heatmaps of each region of the first image is calculated to obtain the second heatmap; The first heatmap is obtained by multiplying the mask of the foreground object region, the first adjustment coefficient, and the second heatmap.

10. The method according to any one of claims 1 to 9, characterized in that, The first model is an image removal model based on stable diffusion SD.

11. An electronic device, characterized in that, The electronic device includes: one or more processors, and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 10.

12. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 10.

Images