Image processing method, electronic device, storage medium, and program product

Abstract

The application provides an image processing method, an electronic device, a storage medium and a program product, and relates to the technical field of image processing. In response to a circle selection operation of a user in an image, the category of an object to be eliminated in the image can be automatically identified, the object to be eliminated is split into a face covering area and a non-face covering area according to the identification result, and then different elimination and repair methods are used to eliminate and repair the face covering area and the non-face covering area respectively. Through the scheme, the user only needs to perform one editing operation, and the dual effects of eliminating the face covering object and repairing the portrait can be achieved, and the elimination of the non-face covering object and the background repair can also be completed, so that the number of user operations is reduced, and the editing experience of the image is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] When users take photos with their mobile phones, passersby or other objects may appear in the pictures. In such cases, users often want to remove the people or objects from the photos. Electronic devices offer intelligent image removal functions that can achieve this. For example, if a user wants to remove certain people or objects from an image, they can select them, and the electronic device can remove the selected objects and fill in the removal area. Currently, electronic devices support removing entire objects or entire portraits from the background area, and the removed area is restored to its original texture.

[0003] However, in practice, electronic devices currently cannot remove or repair partial images of the human body. If a partial image of the human body is removed, it results in a distorted portrait, leading to a poor user experience. If a user selects a portion of a person's image in a photo, the electronic device will display a message indicating that removal is not supported or is impossible. Current technical solutions fail to meet users' needs for removing partial images of the human body, thus degrading the user experience. Summary of the Invention

[0004] This application provides an image processing method, an electronic device, a computer-readable storage medium, and a computer program product, which can meet the user's need to eliminate partial images of the human body and improve the image elimination and restoration experience.

[0005] In a first aspect, this application provides an image processing method, the method comprising: displaying a first image and activating an intelligent removal function; responding to a first operation by a user on the first image, displaying a selection trajectory, wherein the area enclosed by the selection trajectory is a selection region; identifying the category of the object to be removed in the selection region; based on the identification result, splitting the object to be removed into a face-occluded region and a non-face-occluded region; performing image processing on the face-occluded region using a first removal and repair algorithm to obtain a first local repair image; performing image processing on the non-face-occluded region using a second removal and repair algorithm to obtain a second local repair image; and generating a second image based on the first image, the first local repair image, and the second local repair image.

[0006] The image processing method provided in this application, in response to a user's selection operation in an image, can automatically identify the category of the object to be removed in the image. Based on the identification result, the object to be removed is divided into face-occluded areas and non-face-occluded areas. Then, different removal and repair methods are applied to the face-occluded areas and non-face-occluded areas respectively for removal and repair. With this solution, users only need one editing operation to achieve the dual effects of removing face occlusions and restoring the portrait, while also removing non-face occlusions and restoring the background, reducing the number of user operations and improving the image editing experience.

[0007] It should be noted that the removal and restoration process includes two stages. First, the user-selected object is removed from the image, and then the image is regenerated in the removed area (i.e., image restoration). Regarding the principles of image restoration, the first removal and restoration algorithm infers from the facial feature information of the reference image and then regenerates the facial texture in the removed area; the second removal and restoration algorithm infers from the image feature information surrounding the removed area and then regenerates the background texture in the removed area.

[0008] In the embodiments of this application, "eliminate" can also be replaced with "erase," "wipe," "remove," or other terms that are the same as or similar to "eliminate." The mask image can also be called a mask image, a masking image, or a masking mask.

[0009] In this embodiment, the first operation can be an operation whereby the user slides their finger around the image to be eliminated or drags the elimination cursor on the first image (referred to as a selection operation), thereby forming a selected area of ​​regular or irregular shape. The selected area can be a regular shape such as a rectangle or an ellipse, or it can be an irregular shape, depending on the actual usage.

[0010] With this solution, in actual use, users can directly select an area on a photo and remove facial occlusions and non-facial occlusions such as the entire background portrait / the entire background object with one click. In areas where facial occlusions are removed, facial textures are restored, and in areas where non-facial occlusions are removed, background textures are restored, thus improving the removal and repair effects.

[0011] This application's solution can identify the feature information of the object to be removed, including category, quantity, and / or location. Based on the feature information, the object to be removed is divided into face-occluded areas and non-face-occluded areas. Then, the task of removing face occlusions and the task of removing non-face occlusions are distributed to the "Most Beautiful Portrait" service and the "Elimination Pen" service provided in this application. The "Most Beautiful Portrait" service then completes the removal of face occlusions and portrait restoration based on a first elimination and repair algorithm, while the "Elimination Pen" service completes the removal of non-face occlusions and background restoration based on a second elimination and repair algorithm, resulting in the final edited image. This reduces the number of user operations and achieves the best user experience.

[0012] For example, the object to be eliminated can be a face covering such as a mask, sunglasses, or scarf.

[0013] It should be noted that the embodiments of this application are illustrated by example of eliminating facial obstructions. In actual implementation, the method provided by the embodiments of this application can also be applied to eliminate and repair human body obstructions other than the face. For example, the solution of this application can divide the object to be eliminated into human body obstruction areas and non-human body obstruction areas based on the feature information of the object to be eliminated.

[0014] For example, the object to be removed could be an obstruction on the human limbs (such as a watch or ring). When a user wants to remove a watch from their wrist, the electronic device can respond to the user's selection of the watch, remove the watch, and generate skin texture in the removed area.

[0015] For example, the object to be removed could be a label, stain, or other obstruction on clothing. When there is an obstruction on the upper part of a person's clothing and the user wants to remove it, the electronic device can respond to the user's selection of the obstruction, remove the obstruction, and repair and complete the area to be removed.

[0016] This application's solution uses the SAM image segmentation model to segment and predict the image to be processed (the original image), obtaining a segmented image. Based on the segmented image, the objects to be removed in the selected regions can be identified. Furthermore, this application's solution introduces a face detection model to detect faces in the image to be processed. The electronic device can determine two categories of objects to be removed based on the face detection results and the segmented image: face occlusions and non-face occlusions. This innovatively combines face removal and restoration tasks with non-face removal and restoration tasks, reducing the number of user operations and improving image processing results, thus enhancing the user experience in image editing.

[0017] In this application, the user-selected objects are segmented and classified, and the face detection results are combined to determine whether the objects to be removed contain face occlusions. Then, based on the judgment results, an appropriate removal and repair algorithm is determined.

[0018] In scenario one, if the object to be removed only contains facial occlusions and not non-facial occlusions, then the first removal and restoration algorithm is directly used to remove the facial occlusions and perform facial restoration. The solution in this application can remove facial occlusions and fill in the removed facial areas based on the portrait features in the reference image, thus improving the portrait restoration effect.

[0019] Scenario 2: If the objects to be removed include both facial occlusions and non-facial occlusions, then the first removal and repair algorithm is used to remove facial occlusions and repair the face, while the second removal and repair algorithm is used to remove and repair non-facial occlusions. With this approach, if the user selects multiple objects to be removed at once, the electronic device can remove all of them simultaneously, thus reducing the number of user operations and ensuring good image editing results.

[0020] Scenario 3: If the object to be eliminated does not contain face occlusions, then the non-face elimination and repair algorithm is directly used to eliminate and repair the non-face occlusions.

[0021] In some possible implementations, identifying the category of the object to be eliminated in the selected region includes: performing segmentation prediction on the object to be eliminated in the selected region according to an image segmentation model to obtain a segmented image; performing face detection processing on the first image according to a face detection model to obtain a face image; if the segmented image and the face image have an intersection, then it is determined that the object to be eliminated contains the face occlusion region; if the segmented image and the face image do not have an intersection, then it is determined that the object to be eliminated does not contain the face occlusion region.

[0022] This solution segments the object to be removed and, based on facial features, categorizes it into occluded and unoccluded areas. Different removal and repair methods are then applied to each area. Users simply select the target area; the electronic device automatically identifies and segments the object, then removes and repairs both occluded and unoccluded areas, thus fulfilling the user's need to remove facial obstructions.

[0023] In some possible implementations, the step of splitting the object to be eliminated into face-occluded regions and non-face-occluded regions based on the recognition results includes: determining the portion of the segmented image that intersects with the face image as the face-occluded region; and determining the portion of the segmented image that does not intersect with the face image as the non-face-occluded region.

[0024] This application's solution introduces a face detection model to detect faces in the image to be processed. The electronic device can then determine two categories of objects to be removed based on the face detection results and the segmented image: those with facial occlusions and those without. This innovative approach combines face removal and restoration tasks with non-face removal and restoration tasks, reducing the number of user operations and improving image processing results, thus enhancing the user experience in image editing.

[0025] In some possible implementations, the object to be eliminated is a first object. In this case, splitting the object to be eliminated into a face-occluded region and a non-face-occluded region includes: determining a first image region where the first object is located as the face-occluded region; and determining a second image region where the first object is located as the non-face-occluded region.

[0026] The method provided in this application embodiment can be applied to the following scenario: A user selects an object through a selection operation. The first part of the object is located in the face area and the second part is located in the non-face area. Then, a first elimination and repair algorithm can be used to eliminate and repair the first part of the object, and a second elimination and repair algorithm can be used to eliminate and repair the second part of the object. Then, the two repair results are stitched together, and then the stitched repaired image is pasted back into the original image to achieve the purpose of image elimination and repair.

[0027] For example, a user selects an object in the original image through a single selection action. For instance, the object to be removed might be a scarf, part of which is on the face and part of which is not. The user wants to remove the scarf. In response to the user's selection action, the electronic device recognizes the object to be removed as a scarf and splits the scarf image into the face-occluded area and the non-face-occluded area.

[0028] In some possible implementations, the object to be eliminated includes a first object and a second object. In this case, splitting the object to be eliminated into a face-occluded region and a non-face-occluded region includes: determining the image region where the first object is located as the face-occluded region; and determining the image region where the second object is located as the non-face-occluded region.

[0029] The method provided in this application embodiment can also be applied to the following scenarios: If a user selects multiple objects in a single selection operation, with the first object located in a face area and the second object located in a non-face area, then a first elimination and repair algorithm can be used to eliminate and repair the first object, and a second elimination and repair algorithm can be used to eliminate and repair the second object. Alternatively, if a user selects multiple objects in a single selection operation, with the first part of the first object located in a face area and the second part located in a non-face area, and the second object located in a non-face area, then a first elimination and repair algorithm can be used to eliminate and repair the first part of the first object, and a second elimination and repair algorithm can be used to eliminate and repair the second part of the first object and the second object, respectively.

[0030] For example, a user selects two objects in the original image with a single selection action. The objects to be removed might include a face mask and an object outside the body (such as a trash can). In response to the user's selection action, the electronic device recognizes that the objects to be removed consist of two objects and separates them into a face-occluded area and a non-face-occluded area.

[0031] In some possible implementations, the step of using a first elimination and restoration algorithm to process the face occlusion area to obtain a first locally restored image includes: acquiring a reference image, which is a frame preceding or following the first image, and the reference image contains facial features; obtaining reference facial feature information based on the reference image; inputting the reference facial feature information, the original image of the face occlusion area, and the mask image of the face occlusion area into a control network model to obtain a first feature image; and inputting the first feature image, the mask image of the face occlusion area, and the reference facial feature information into a first elimination and restoration model to obtain the first locally restored image, wherein the first locally restored image includes facial texture.

[0032] This application solution can utilize facial image information from a recent time period to repair and enhance facial features, thereby improving the image restoration effect.

[0033] In some possible implementations, the step of inputting the first feature image, the mask image of the face occlusion area, and the reference face feature information into a first elimination and restoration model to obtain the first local restoration image includes: inputting the first feature image, the mask image of the face occlusion area, and the reference face feature information into a first elimination and restoration model, performing model inference based on a head pose correction algorithm to obtain the first local restoration image; wherein, the head pose correction algorithm includes face orientation correction parameters and / or facial feature key point correction parameters.

[0034] The proposed solution uses a head pose correction algorithm for model inference, which can repair and enhance facial features, thereby improving the portrait restoration effect.

[0035] In some possible implementations, obtaining reference facial feature information based on the reference image includes: inputting the reference image into an image feature extraction network to generate first feature information; inputting the reference image into a facial feature extraction model to generate second feature information; and performing mapping processing based on the first feature information and the second feature information to obtain the reference facial feature information.

[0036] The proposed solution involves multiple processing steps on the reference image to obtain reference facial feature information, which can be better applied to portrait restoration and improve the restoration effect.

[0037] In some possible implementations, the first elimination and repair model is an elimination and repair model based on stable diffusion SD, which includes a denoising U-net network.

[0038] In some possible implementations, the second elimination and restoration algorithm is used to process the image of the non-face-occluded region to obtain a second local restoration image, including: obtaining positive text and negative text based on the original image of the non-face-occluded region, wherein the positive text is used to enhance the generation of background texture and the negative text is used to suppress the generation of foreground object texture; inputting the original image and mask image of the non-face-occluded region into the second elimination and restoration model, performing model inference based on the positive text and the negative text to obtain the second local restoration image, wherein the second local restoration image includes background texture.

[0039] The above-described scheme of this application, based on positive and negative text, tends to generate background pixel information corresponding to positive text in the eliminated area, while suppressing the generation of foreground object pixel information corresponding to negative text. Therefore, through model inference, no other foreground object images such as portraits or objects will be generated in the eliminated area, thereby improving the effect of image elimination and restoration.

[0040] In some possible implementations, the step of performing model inference based on the positive and negative text to obtain the second locally repaired image includes: calculating image noise prediction values ​​based on the positive and negative text using a classifier-free approach according to the following equation:

[0041] Npred = Nneg + α*(Npos - Nneg) ，

[0042] Wherein, Npos represents the background noise prediction value obtained based on the positive text prediction, Nneg represents the foreground noise prediction value obtained based on the negative text prediction, α is a hyperparameter, and Npred represents the image noise prediction value obtained by the second elimination and repair model at each time step;

[0043] Based on the image noise prediction value Npred, multiple noise sampling is performed to obtain a multiple noise sampled image; based on the multiple noise sampled image, the second local repair image is obtained.

[0044] The above approach allows for the use of a classifier-free method, based on positive and negative text, to guide the model in generating background information, suppressing the regeneration of foreground objects and thus improving the image removal and restoration effect.

[0045] In some possible implementations, obtaining positive and negative text from the original image of the non-face-occluded region includes: performing panoramic segmentation on the original image of the non-face-occluded region to obtain background category information and foreground category information; generating the positive text based on the background category information and background text, and generating the negative text based on the foreground category information and foreground text; wherein the positive text is used to enhance the generation of background texture corresponding to the background category information, and the negative text is used to suppress the generation of foreground object texture corresponding to the foreground category information; wherein the background text and the foreground text are texts learned by the second removal and repair model through training.

[0046] The above approach allows for model inference based on positive and negative text. The model inference tends to generate background pixel information corresponding to positive text in the eliminated area, while suppressing the generation of foreground object pixel information corresponding to negative text. Therefore, through model inference, no other foreground object images such as portraits or objects will be generated in the eliminated area, thereby improving the image elimination and restoration effect.

[0047] This application's solution trains the removal and restoration model based on sample images, background sample masks, and background text, and also trains it based on sample images, foreground sample masks, and foreground text, enabling the model to learn both background and foreground text. This way, when applying the removal and restoration model to image processing, it tends to generate background pixel information corresponding to positive text while suppressing the generation of foreground object pixel information corresponding to negative text. This prevents the generation of other foreground object images, such as portraits or objects, in the already removed areas, thereby improving the image removal and restoration effect.

[0048] It should be noted that in this embodiment, the input image is decomposed into foreground object regions and background regions. When the mask M covers the foreground object region (i.e., mask M is a foreground mask), foreground text input (denoted as Obj) is added for training. When the mask M covers the background region (i.e., mask M is a background mask), background text input (denoted as Ctxt) is added for training. In this way, the model can be guided to generate background information during the model inference stage through classifier-free guidance (CFG), thus suppressing the problem of foreground object regeneration.

[0049] In some possible implementations, the method further includes performing adaptive masking on the mask image before inputting the original image and mask image of the non-face-occluded region into the second removal and restoration model.

[0050] Experimental tests revealed that when there are shadow areas around the object to be removed, the shadow areas can affect image restoration to some extent. If the mask of the object to be removed does not include the shadow area, other foreground objects such as portraits or objects may be generated in the removed area.

[0051] The embodiments of this application employ adaptive mask expansion based on brightness contrast, so that the expanded mask includes possible shadow areas near the object to be eliminated, which can suppress the problem of foreground object regeneration caused by shadows, thus improving the effect of image elimination and restoration.

[0052] In some possible implementations, the second elimination and repair model includes an encoding network, a denoising U-net network, and a decoding network.

[0053] This application provides the following two user scenarios:

[0054] User Scenario 1: In response to user input, the electronic device enters the gallery or photo album interface. The gallery interface displays the first image. After the user clicks the edit option in the gallery interface, the electronic device displays options such as doodle and AI removal. When the AI ​​removal option is selected, in response to the user's selection or smearing action on a certain area of ​​the first image, the electronic device removes the object (e.g., a person or object) in the selected area and performs restoration processing on the removed area to ensure that the removed area is consistent with the background image, thus improving the image removal effect.

[0055] User Scenario 2: With the camera app enabled, in response to the user clicking the shutter button on the shooting interface, the electronic device captures a first image and displays a thumbnail of the first image on the shooting interface. In response to the user's interaction with the thumbnail, the electronic device navigates to the gallery interface, where the first image is displayed. After the user clicks the edit option in the gallery interface, the electronic device displays options such as drawing and AI removal. When the AI ​​removal option is selected, in response to the user's selection or drawing action on a specific area of ​​the first image, the electronic device removes objects (such as people or objects) from the selected area and performs retouching processing on the removed area to ensure that the removed area remains consistent with the background image, improving the image removal effect.

[0056] Secondly, this application provides an image processing apparatus, which includes units for performing the method described in the first aspect above. This apparatus is capable of performing the method described in the first aspect above, and a description of the units within this apparatus is provided in the description of the first aspect above; for brevity, it will not be repeated here.

[0057] The method described in the first aspect above can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a processing module or unit, a display module or unit, etc.

[0058] Thirdly, this application provides an electronic device, which includes a processor, a computer program or instructions stored in a memory, wherein the processor is used to execute the computer program or instructions to cause the method in the first aspect to be performed.

[0059] Fourthly, this application provides a computer-readable storage medium having a computer program (also referred to as instructions or code) stored thereon for implementing the method of the first aspect. For example, when the computer program is executed by a computer, it enables the computer to perform the method of the first aspect.

[0060] Fifthly, this application provides a chip including a processor. The processor is used to read and execute a computer program stored in a memory to perform the methods in the first aspect and any possible implementation thereof. Optionally, the chip further includes a memory connected to the processor via a circuit or wire.

[0061] Sixthly, this application provides a chip system including a processor. The processor is used to read and execute a computer program stored in a memory to perform the methods in the first aspect and any possible implementation thereof. Optionally, the chip system further includes a memory connected to the processor via a circuit or wire.

[0062] In a seventh aspect, this application provides a computer program product comprising a computer program (also referred to as instructions or code), which, when executed by an electronic device, causes the electronic device to implement the method in the first aspect.

[0063] It is understood that the beneficial effects of the second to seventh aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0064] Figure 1 This is a schematic diagram illustrating image removal and restoration scenarios in related technologies;

[0065] Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0066] Figure 3 A schematic diagram of the software architecture of an electronic device provided in an embodiment of this application;

[0067] Figure 4 This is a schematic diagram illustrating the application of the image processing method provided in this embodiment of the image processing method to a mobile phone user interface.

[0068] Figure 5 A schematic flowchart of the image processing method provided in the embodiments of this application;

[0069] Figure 6 This is a schematic diagram illustrating an application scenario of the image processing method provided in the embodiments of this application;

[0070] Figure 7A The flowchart illustrating the application of the image processing method provided in this application to eliminate facial occlusions and non-facial occlusions. Figure 1 ;

[0071] Figure 7B A schematic diagram of a user interface illustrating the application of the image processing method provided in this application to eliminate facial occlusions and non-facial occlusions;

[0072] Figure 7C The flowchart illustrating the application of the image processing method provided in this application to eliminate facial occlusions and non-facial occlusions. Figure 2 ;

[0073] Figure 8 A flowchart illustrating the first elimination and repair algorithm provided in an embodiment of this application;

[0074] Figure 9 A flowchart illustrating the second elimination and repair algorithm provided in this application embodiment;

[0075] Figure 10AA flowchart illustrating the application of the image processing method provided in this application to non-facial occlusions. Figure 1 ;

[0076] Figure 10B A schematic diagram of a user interface for applying the image processing method provided in this application to non-face occlusion objects;

[0077] Figure 10C A flowchart illustrating the application of the image processing method provided in this application to non-facial occlusions. Figure 2 ;

[0078] Figures 11A to 11D A schematic diagram illustrating the process of the foreground and background decomposition training and inference method based on panoramic segmentation provided in the embodiments of this application;

[0079] Figures 12A to 12C This is a schematic diagram illustrating the adaptive mask expansion process based on brightness contrast provided in an embodiment of this application.

[0080] Figure 13 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application. Detailed Implementation

[0081] In the embodiments of this application, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "multiple" means two or more.

[0082] When users take photos with their mobile phones, passersby or other objects may appear in the photos. In such cases, users often want to remove the people or objects from the photos. This can be achieved using the intelligent removal function provided by electronic devices. For example, if a user wants to remove certain people or objects from an image, they can select them. The electronic device can then remove the selected objects and fill in the missing areas based on the surrounding image, thus completing the removal and restoration.

[0083] For example, electronic devices can support the removal of entire background objects. Figure 1 As shown in (a), the user selects the entire object "trash can" in the photo and wants to eliminate the "trash can"; in response to the user's operation, the electronic device can perform elimination and repair processing on the object "trash can" selected by the user, and the eliminated area will be repaired to the background texture.

[0084] For example, electronic devices can support removing entire background portraits. Figure 1As shown in (b), the user selects the entire "passerby" in the photo and wants to remove the "passerby"; in response to the user's operation, the electronic device can perform removal and repair processing on the "passerby" selected by the user, and the removed area will be repaired as the background texture.

[0085] However, in practice, users often need to remove partial occlusions on the human body. Currently, electronic devices cannot remove or repair partial images of the human body because the current generation model only supports removing the entire portrait, restoring the removed area to background texture. Removing only a part of the human body results in a fragmented portrait, leading to a poor user experience. If a user selects a portion of the portrait in a photo, the electronic device will display a message indicating that removal is not supported or impossible. Figure 1 As shown in (c), when a user selects a part of a human body image and wants to remove the "mask" on the face, the electronic device cannot remove or repair the "mask" selected by the user in response to the user's selection operation, and will prompt that it does not support removal or cannot remove.

[0086] In some related technologies, to avoid creating incomplete human images due to partial removal of the human body, which could affect the removal effect, the relevant technical solutions first determine whether the removal conditions are met before deciding whether to perform removal and repair processing. If the selected area surrounds the entire human or object image, then the removal conditions are met, and removal and repair processing is performed in this case. If the selected area does not completely surround the human image and only covers a part of the human body, then the electronic device determines that the selected area does not meet the removal conditions. In this case, removal and repair processing is not performed to prevent accidental removal of partial human body images. In this situation, the electronic device will indicate that the selection operation is invalid.

[0087] Current technical solutions cannot meet users' needs for eliminating partial images of the human body (such as removing facial occlusions), thus reducing the user experience.

[0088] To address the aforementioned problems, this application provides an image processing method and electronic device. In this embodiment, in response to a user's selection operation in an image, the method automatically identifies the category of the object to be removed. Based on the identification result, the object is divided into face-occluded areas and non-face-occluded areas. Then, different removal and repair methods are applied to the face-occluded areas and non-face-occluded areas respectively for removal and repair. Through this solution, users only need one editing operation to achieve the dual effects of removing face occlusions and restoring the portrait, while also removing non-face occlusions and restoring the background, reducing the number of user operations and improving the image editing experience.

[0089] It should be noted that the removal and restoration process includes two stages. First, the user-selected object is removed from the image, and then the image is regenerated in the removed area (i.e., image restoration). Regarding the principles of image restoration, the first removal and restoration algorithm infers from the facial feature information of the reference image and then regenerates the facial texture in the removed area; the second removal and restoration algorithm infers from the image feature information surrounding the removed area and then regenerates the background texture in the removed area.

[0090] The following describes the improvement scheme provided by the embodiments of this application.

[0091] In this embodiment, the objects to be removed from the image are divided into two categories: face occluders and non-face occluders. Face occluders can refer to objects located on a person's face in the image, such as masks or sunglasses. Non-face occluders can refer to objects located outside the person's face in the image, such as passersby, animals / plants, or various objects. For example, refer to... Figure 1 In (a), "passerby", "trash can" and "bird" are non-face coverings, while "mask" is a face covering.

[0092] The test results based on image removal and restoration revealed a significant difference in restoration effectiveness between removing facial occlusions and removing non-facial occlusions. Applying the same removal and restoration algorithm to both scenarios often fails to achieve true portrait fidelity. In terms of image processing performance, the restoration of non-facial occlusions such as "passersby," "trash cans," and "birds" was good, while the restoration of the portrait at the "mask" location was poor.

[0093] In other words, the "smart removal" function currently provided by electronic devices works well when applied to the removal and restoration of overall portraits or objects. However, when users want to remove facial occlusions, the current "smart removal" function is limited by the image generation capabilities and often fails to achieve satisfactory results.

[0094] Analysis reveals that facial occlusions and non-facial occlusions are two distinct types of objects to be removed and restored. Comparatively, portrait restoration is more complex and demanding. This application proposes first identifying the category of the object to be removed, and then employing different removal and restoration algorithms for different categories to improve the image restoration effect.

[0095] This application's solution uses the SAM image segmentation model to segment and predict the image to be processed (the original image), obtaining a segmented image. Based on the segmented image, the objects to be removed in the selected regions can be identified. Furthermore, this application's solution introduces a face detection model to detect faces in the image to be processed. The electronic device can determine two categories of objects to be removed based on the face detection results and the segmented image: face occlusions and non-face occlusions. This innovatively combines face removal and restoration tasks with non-face removal and restoration tasks, reducing the number of user operations and improving image processing results, thus enhancing the user experience in image editing.

[0096] In this application, the user-selected objects are segmented and classified, and the face detection results are combined to determine whether the objects to be removed contain face occlusions. Then, based on the judgment results, an appropriate removal and repair algorithm is determined.

[0097] In scenario one, if the object to be eliminated only contains face occlusions and not non-face occlusions, then the first elimination and repair algorithm is directly used to eliminate face occlusions and repair the face.

[0098] Scenario 2: If the objects to be removed include facial occlusions and non-facial occlusions, then a first elimination and restoration algorithm is used to remove facial occlusions and perform facial restoration, while a second elimination and restoration algorithm is used to remove and restore non-facial occlusions. For ease of explanation, the first elimination and restoration algorithm is also referred to as the facial elimination and restoration algorithm, and the second elimination and restoration algorithm is also referred to as the non-facial elimination and restoration algorithm. Specifically, the facial elimination and restoration algorithm combines a reference image containing facial features to complete portrait restoration. In particular, the solution of this application can remove facial occlusions and fill in the removed facial areas based on the facial features in the reference image, thus improving the portrait restoration effect.

[0099] With this solution, if a user selects multiple objects to be eliminated at once, the electronic device can eliminate all of them at once, thus reducing the number of operations required by the user and ensuring good image editing results.

[0100] Scenario 3: If the object to be removed does not contain facial occlusions, then a non-face removal and repair algorithm is directly used to remove and repair the non-face occlusions. This non-face removal and repair algorithm can effectively remove and repair non-face occlusions. This application employs the following processing strategy: Based on the original image of the non-face occlusion area, positive and negative text are obtained. The positive text is used to enhance the generated background texture, and the negative text is used to suppress the generated foreground object texture. The original image of the non-face occlusion area and the mask image are input into the second removal and repair model. Based on the positive and negative text, model inference is performed to obtain a repaired image containing background texture, thereby improving the image removal and repair effect.

[0101] It should be noted that the embodiments of this application are illustrated by taking the removal of facial occlusions as an example. In actual implementation, the method provided by the embodiments of this application can also be applied to the removal and repair of human body occlusions other than the face. For example, when there is an occlusion on the upper part of the portrait and the user wants to remove the occlusion, the electronic device can respond to the user's selection operation of the occlusion, remove the occlusion and repair and complete the area that has been removed.

[0102] It should also be noted that the embodiments of this application are illustrated by way of example of removing facial occlusions. In actual implementation, the method provided by the embodiments of this application can also be applied to the following scenarios: the first part of the occlusion is located in the face area and the second part is located in the non-face area of ​​the human body. In this case, the first elimination and repair algorithm can be used to eliminate and repair the first part of the occlusion, and the second elimination and repair algorithm can be used to eliminate and repair the second part of the occlusion. Then, the two repair results are stitched together, and the stitched repaired image is pasted back into the original image to achieve the purpose of image elimination and repair.

[0103] The image processing method and electronic device in the embodiments of this application will now be described with reference to the accompanying drawings.

[0104] Figure 2 A hardware system for an electronic device applicable to this application is shown.

[0105] Electronic device 100 can be a mobile phone, smart screen, tablet computer, wearable electronic device, in-vehicle electronic device, augmented reality (AR) device, virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), projector, etc. This application embodiment does not limit the specific type of electronic device 100.

[0106] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0107] It should be noted that, Figure 2 The structure shown does not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include... Figure 2 The components shown may include more or fewer components, or the electronic device 100 may include... Figure 2 The components shown may be a combination of certain components, or the electronic device 100 may include... Figure 2 Sub-components of some of the components shown. Figure 2 The components shown can be implemented in hardware, software, or a combination of both.

[0108] Processor 110 may include one or more processing units. For example, processor 110 may include at least one of the following processing units: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and neural network processing unit (NPU). These different processing units may be independent devices or integrated devices. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0109] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0110] For example, the processor 110 can be used to execute the image processing method of this application embodiment: in response to a user's selection operation in an image, it can automatically identify the category of the object to be removed in the image, and according to the identification result, split the object to be removed into face-occluded areas and non-face-occluded areas, and then use different removal and repair methods to remove and repair the face-occluded areas and non-face-occluded areas respectively. Through this solution, the user only needs to perform one editing operation to achieve the dual effects of removing face occlusions and restoring the portrait, while also completing the removal of non-face occlusions and background restoration, reducing the number of user operations and improving the image editing experience.

[0111] Figure 2 The connection relationships between the modules shown are merely illustrative and do not constitute a limitation on the connection relationships between the modules of the electronic device 100. Optionally, the modules of the electronic device 100 may also adopt a combination of various connection methods described in the above embodiments.

[0112] The wireless communication function of electronic device 100 can be realized through devices such as antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor.

[0113] Electronic device 100 can implement display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0114] The display screen 194 can be used to display images or videos. Exemplarily, in an embodiment of this application, the display screen 194 can be used to display a first captured image (the image to be removed and repaired), and to display a second image (the image after removal and repair).

[0115] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0116] The ISP is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the camera to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can perform algorithmic optimization of image noise, brightness, and color. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0117] The camera 193 (also known as a lens) is used to capture still images or videos. It can be activated via application commands to enable photo-taking, such as capturing images of any scene. The camera may include components such as an imaging lens, filters, and an image sensor. Light emitted or reflected by an object enters the imaging lens, passes through the filter, and finally converges onto the image sensor. The imaging lens is primarily used to converge and image the light emitted or reflected by all objects within the shooting field of view (also known as the scene to be captured); the filter is primarily used to filter out excess light waves (such as infrared waves other than visible light); the image sensor can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The image sensor is primarily used to perform photoelectric conversion on the received light signal, converting it into an electrical signal, which is then transmitted to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into standard RGB, YUV, and other image signal formats.

[0118] In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0119] The camera 193 can be located in front of the electronic device 100 or on the back of the electronic device 100. The specific number and arrangement of the cameras can be set according to the requirements, and this application does not impose any restrictions.

[0120] A touch sensor 180K, also known as a touch device, can be disposed on a display screen 194. The touch sensor 180K and the display screen 194 together form a touchscreen, also known as a touch display. The touch sensor 180K is used to detect touch operations applied to or near it. The touch sensor 180K can transmit the detected touch operation to an application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K may also be disposed on the surface of the electronic device 100, and in a different location from the display screen 194. Exemplarily, in an embodiment of this application, the touch sensor 180K can be used to detect user-triggered image deletion and restoration operations.

[0121] The hardware system of electronic device 100 has been described in detail above. The software system of electronic device 100 will be introduced below.

[0122] Figure 3 This is a schematic diagram of the software system of the electronic device 100 provided in this application embodiment. The layered architecture divides the system into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: applications 201, application framework 202, hardware abstraction layer (HAL) 203, kernel 204, and hardware layer 205.

[0123] Application layer 201 may include a series of application packages. In this embodiment, the application packages may include applications such as camera and gallery.

[0124] The application framework layer 202 provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer 202 includes some predefined functions. In this embodiment, the application framework layer may include a camera access interface, wherein the image processing interface may include image removal and restoration services. The image processing interface is used to provide an API and programming framework for the gallery application.

[0125] The hardware abstraction layer 203 is an interface layer located between the application framework layer 202 and the kernel layer 204, providing a virtual hardware platform for the operating system. In this embodiment, the hardware abstraction layer 203 may include a camera hardware abstraction layer and an image processing algorithm library.

[0126] The camera hardware abstraction layer (HAL) can provide virtual hardware for camera devices such as the front and rear cameras (referred to as camera HAL). The image processing algorithm library may include a first elimination and repair algorithm and a second elimination and repair algorithm; in other words, the image processing algorithm library contains runtime code and data that implement the image processing methods provided in the embodiments of this application. When a portion of the object to be eliminated (the selected object) is located in the face region and a portion is located outside the face region, the first elimination and repair algorithm can be used to eliminate and repair the face region, and the second elimination and repair algorithm can be used to eliminate and repair the region outside the face.

[0127] Kernel layer 204 is the layer between hardware and software. It includes drivers for various hardware components. Kernel layer 204 may include camera device drivers, digital signal processor (DSP) drivers, and image processor (IP) drivers, among others. Specifically, the camera device driver drives the camera sensor to acquire images and the image signal processor to preprocess them. The DSP driver drives the DSP to process images. The IIP driver drives the graphics processor to process images.

[0128] Hardware layer 205 includes camera sensors, image signal processors, digital signal processors, and image processors, etc.

[0129] It should be noted that although the embodiments of this application are described using the Android system as an example, the basic principles are also applicable to electronic devices based on operating systems such as iOS or Windows.

[0130] The image processing method provided in this application can be executed by the aforementioned electronic device (such as a mobile phone), or by a functional module and / or functional entity within the electronic device capable of implementing the image processing method. Furthermore, the solution in this application can be implemented through hardware and / or software, and the specific implementation can be determined according to actual usage requirements; this application does not impose any limitations. The following description uses an electronic device as an example, along with the accompanying drawings, to exemplarily illustrate the image processing method provided in this application.

[0131] To better understand the embodiments of this application, the "image removal and repair" function provided in the embodiments of this application will be briefly described below with reference to a mobile phone interface diagram:

[0132] In some embodiments, the electronic device adds an "AI Removal" option to the editing interface of a gallery application (or photo album application). This "AI Removal" option serves as an entry point to trigger the display of image removal and restoration functions. Exemplarily, the image removal and restoration functions may include features such as intelligent selection and manual smearing. It should be noted that the various functions included in the image removal and restoration functions and their names are illustrative examples and are not limited in scope by this application.

[0133] Intelligent selection refers to the process by which electronic devices, in response to a user's swiping action on an image, select targets based on the swiping trajectory.

[0134] Manual smearing refers to the electronic device responding to the user's smearing action on an image and selecting the target based on the smeared area.

[0135] Figure 4 This illustration shows a user interface diagram of the "AI elimination" option in the image processing method provided in this application embodiment. Figure 4 As shown in (a), the electronic device displays an icon for the Gallery app on desktop 10. When the user clicks the Gallery app icon, as shown in (a), the user sees the following: Figure 4 As shown in (b), the electronic device updates from displaying the desktop 10 to displaying the all-photos interface 11. When the user clicks on a photo (such as the first image) in the all-photos interface 11, as shown in (b), the electronic device updates from displaying the desktop 10 to displaying the all-photos interface 11. Figure 4 As shown in (c), the electronic device updates from displaying all photos interface 11 to displaying a single photo interface 12. The single photo interface includes editing options 13. Figure 4 As shown in (d), when editing option 13 is selected, the electronic device updates from displaying a single photo interface 12 to displaying an editing interface 14. Editing interface 14 includes the photo selected by the user and the "AI Removal" option 15. When the "AI Removal" option 15 is selected in editing interface 14, the electronic device displays the functions corresponding to the "AI Removal" option 15, such as smart selection and manual smoothing, thereby activating the smart removal function.

[0136] It should be noted that users can use functions such as smart selection or manual brushing to select objects in the image, and the electronic device can remove the selected objects and repair the removed parts.

[0137] In some embodiments, after the user drags the trajectory cursor from the starting point to the ending point and releases it, the object to be eliminated can be displayed using a dynamic blinking display method.

[0138] Taking the selection operation as an example, in response to the user's selection operation on the image, the electronic device can determine the area selected by the user and identify the people or objects in the selected area, thereby determining the object to be eliminated; then the object to be eliminated is eliminated and repaired to generate a more aesthetically pleasing image.

[0139] In some embodiments, the object to be removed can be the entire portrait or an object, such as a passerby in the background area of ​​the photo; or a trash can or debris in the background area of ​​the photo; or an object to be removed can be a face obstruction in the portrait area, such as sunglasses or a mask.

[0140] The image processing method provided in this application will be described in detail below with reference to specific embodiments and the accompanying drawings.

[0141] Figure 5 This is a schematic flowchart illustrating an image processing method provided in an embodiment of this application. The method can be... Figure 2 The electronic device shown performs this action.

[0142] S301. Display the first image and activate the intelligent elimination function.

[0143] The first image is the original image. It is assumed that the original image includes facial features and that the face is obscured by objects such as sunglasses or masks; the original image may also include some debris or passersby.

[0144] In some cases, users' image processing needs are: to eliminate clutter or pedestrian features through a single intelligent removal operation, and whether the background features of the removed area can be restored.

[0145] In some cases, users' image processing needs are: to eliminate the features of obstructions such as sunglasses or masks through a single intelligent removal operation, and whether the facial features in the removed area can be restored.

[0146] In other cases, users' image processing needs are: to eliminate clutter or pedestrian features through a single intelligent removal operation, and whether the background features of the removed area can be restored; and to eliminate obstructions such as sunglasses or masks, and whether the facial features of the removed area can be restored.

[0147] S302, In response to the user's selection operation on the first image, determine the selected area.

[0148] For example, the selection operation can be an operation in which a user slides their finger or drags the eraser cursor in an image to select a certain area.

[0149] In some embodiments, in response to a user's selection operation, a selection trajectory can be marked in the first image, and a selection area can be determined based on the selection trajectory. For example, the area enclosed by the selection trajectory can be defined as the selection area. Optionally, the selection trajectory can be a line trajectory, or it can be a closed rectangle, ellipse, or irregular shape. For ease of explanation, this application embodiment uses a line trajectory as an example for illustrative purposes.

[0150] In some embodiments, when the selection trajectory is closed, the area enclosed by the selection trajectory is defined as the selection area.

[0151] In other embodiments, if the distance between the start and end points of the selected trajectory meets a preset distance condition when the selected trajectory is not closed, the electronic device can automatically connect the start and end points to form a closed area of ​​the selected trajectory, and then use the closed area as the selected area.

[0152] In some embodiments, if the distance between the start and end points of the selected trajectory does not meet a preset distance condition (e.g., the distance between the start and end points is too large) when the selected trajectory is not closed, the electronic device determines that the selected trajectory does not meet the condition and displays a prompt message. For example, the prompt message may be: This selection operation is invalid; please select again.

[0153] In this context, the people or objects within the selected area are the objects to be eliminated.

[0154] For example, the object to be removed includes the entire portrait in the background area.

[0155] For example, the objects to be eliminated include items such as trash cans in the background area.

[0156] For example, the objects to be eliminated include face masks or sunglasses.

[0157] It should be noted that users may select objects in various ways. For example, they may select one object at a time or multiple objects at once. In some cases, the objects to be removed include facial occlusions; in others, they include non-facial occlusions; and in still others, they include both facial occlusions and non-facial occlusions. Therefore, it is necessary to first perform scene recognition, and then apply a specific removal and restoration algorithm to the image processing based on the recognition results.

[0158] In this embodiment of the application, scene recognition is performed through the following S303 and S305, that is, by determining whether the selected area contains facial features and whether there are occluders on the face in the selected area, it is possible to identify whether the object to be eliminated includes facial occluders.

[0159] When the selected area does not contain facial features, or when the selected area contains facial features but there are no obstructions on the face in the selected area, it can be determined that the object to be removed does not include facial obstructions.

[0160] When the selected area contains facial features and there are obstructions on the face within the selected area, it can be determined that the object to be removed includes the facial obstructions.

[0161] The following example illustrates the removal and repair process for both face-obscuring and non-face-obscuring objects.

[0162] S303. Determine whether the selected area contains facial features.

[0163] S304. If the selected area does not contain facial features, a non-facial elimination and repair algorithm is used to eliminate and repair the objects to be eliminated in the selected area.

[0164] For example, the objects to be eliminated in the selected area are human figures and / or objects in the background area.

[0165] For example, such as Figure 6 As shown in (a), the user selected a background area in the image. The area enclosed by the dashed line in the image is the selected area, and the "bird" is the object to be removed, i.e., a non-face occlusion. The electronic device responds to the user's operation, removes the non-face occlusion "bird," and repairs the removed portion.

[0166] S305. If the selected area contains facial features, determine whether there are any obstructions on the face in the selected area.

[0167] S306. If the selected area contains facial features but there are no obstructions on the face in the selected area, then the elimination and repair process will not be performed.

[0168] S307. If the selected area contains facial features and there are occlusions on the face in the selected area, then the occlusions on the face in the selected area are identified as the first elimination target, and the non-face occlusions in the selected area are identified as the second elimination target.

[0169] S308. The first object to be removed is processed using a face removal and restoration algorithm to obtain a first local restoration image; and the second object to be removed is processed using a non-face removal and restoration algorithm to obtain a second local restoration image.

[0170] S309. Generate a second image based on the first image, the first local repair image, and the second local repair image.

[0171] For example, the first and second local repaired images are respectively pasted back into the eliminated areas of the first image to obtain the second image.

[0172] For example, such as Figure 6 As shown in (b), the user has selected a region of the face in the image. The area enclosed by the dashed line in the image is the selected region, and the "mask" is the object to be removed, i.e., the face obstruction. In response to the user's selection, the electronic device removes the face obstruction "mask" and repairs the removed portion of the face region.

[0173] like Figure 6As shown in (c), the user simultaneously selected both the human face area and the background area in the image. The area enclosed by the dashed line in the image is the selected area. The "mask" and "bird" are the objects to be removed. The "mask" is a face occlusion, and the "bird" is a non-face occlusion. In response to the user's selection operation, the electronic device removes the face occlusion "mask" and the non-face occlusion "bird," and repairs the removed portions of the background and human face areas.

[0174] With this solution, if a user selects multiple objects to be eliminated at once, the electronic device can eliminate all of them at once, thus reducing the number of operations required by the user and ensuring good image editing results.

[0175] In practical implementation, if the object to be removed includes facial occlusions but not non-facial occlusions, then the facial occlusion removal and restoration algorithm can be directly used to process the facial occlusions, remove the facial occlusions and generate a facial restoration image, and then paste the facial restoration image back into the removed area in the first image to obtain the second image.

[0176] Example 1: The objects to be eliminated include face obstructions and non-face obstructions.

[0177] In Example 1, the user selected two objects in one selection operation: one object was a face occlusion and the other object was a non-face occlusion.

[0178] Figure 7A For based on Figure 5 The flowchart shown illustrates the processing flow when the object to be removed includes facial obstructions. Figure 7A As shown, the method includes an image selection stage, a scene recognition stage, and a classification, elimination, and repair stage. These three stages are described below.

[0179] Phase 1: Image Selection Phase

[0180] refer to Figure 7A The electronic device displays a first image (the original image). The user selects a region on the first image, triggering the electronic device to perform image removal. The electronic device then acquires the first image with the selected area.

[0181] Phase Two: Scene Recognition Phase

[0182] refer to Figure 7A Electronic devices preprocess the original image to obtain a preprocessed image. For example, the long side of the original image is resized to 1024 pixels, and the short side of the first image is padded, finally resulting in a preprocessed image of size 1024*1024 pixels.

[0183] In this application embodiment, the preprocessed image is input into different image processing models through the following two branches.

[0184] Branch 1: Input the preprocessed image into a face detection model to perform face detection and obtain a face image. For example, by performing face detection on the preprocessed image using a face detection model, a masked face image can be obtained.

[0185] Branch 2: Input the preprocessed image into the SAM image segmentation model. The SAM image segmentation model then performs segmentation prediction on the image of the person or object in the selected area, resulting in a segmented image of the person or object within the selected area. In other words, through segmentation prediction, the object to be eliminated in the selected area can be determined.

[0186] SAM (Segment Anything Model) is a prompt-based image segmentation model. SAM aims to solve a wide range of image segmentation tasks through prompt engineering, enabling the segmentation of any image without any class annotations. The core architecture of SAM includes an image encoder, a prompt encoder, and a mask decoder. The image encoder uses a pre-trained visual transformer to encode the image into image vectors. The prompt encoder encodes prompt content (such as points, boxes, text, or masks) into prompt vectors. The mask decoder fuses the image vectors and prompt vectors and decodes them to obtain the final segmentation mask. Through this design, SAM generates high-quality object segmentation regions from input prompts such as points or boxes, achieving the segmentation of specified targets, thus enabling one-click removal, filling, or replacement of any object in an image.

[0187] Then, by finding the intersection of the face image and the segmentation image, it can be determined whether the object to be removed contains facial occlusions. If the face image and the segmentation image have an intersection, then the intersection part is the facial occlusion (such as a mask or sunglasses).

[0188] Phase Three: Categorization, Elimination, and Repair Phase

[0189] In this embodiment of the application, the segmented image is split into face occlusion image and non-face occlusion image. Specifically, the part of the segmented image that intersects with the face image is determined as the face occlusion image, and the part of the segmented image that does not intersect with the face image is determined as the non-face occlusion image.

[0190] Then, different removal and restoration algorithms were used to process the face-occluded and non-face-occluded images.

[0191] Branch 1: Based on the occluded face image, the preprocessed image, and the reference image containing facial features, a first elimination and restoration algorithm (or image face swapping algorithm) is used to eliminate and restore the occluded face image, resulting in a first locally restored image. This first elimination and restoration algorithm is implemented in conjunction with the portrait reference image, capable of generating facial textures and possessing excellent face restoration capabilities. The specific process of eliminating and restoring the occluded face image using the first elimination and restoration algorithm will be described in detail below.

[0192] Branch Two: Based on the non-face-occluded image and the preprocessed image, a second elimination and restoration algorithm is used to eliminate and restore the non-face-occluded image, resulting in a second locally restored image. This second elimination and restoration algorithm is used to remove pedestrians or objects from the background, generates background texture, and has good background restoration capabilities. The specific process of eliminating and restoring the non-face-occluded image using the second elimination and restoration algorithm will be described in detail below.

[0193] Then, the first and second locally repaired images are pasted back onto the first image to obtain the second image.

[0194] The proposed solution uses a SAM model to segment and predict the first image, then introduces a face detection model to determine whether the segmented prediction region is within the human face (i.e., the face). For segmented predictions not within the human face region, a second elimination and restoration algorithm is called to generate a local background region; for segmented predictions within the human face region, a second elimination and restoration algorithm with a reference image is called to generate a local facial region. The generated results from both parts are then pasted back onto the original-sized first image to achieve the overall image elimination and restoration effect.

[0195] Figure 7B This is a schematic diagram illustrating the application of the image processing method provided in this application to the interactive interface of an electronic device. For example... Figure 7B As shown in (a), the electronic device receives a selection operation from the user on the first image 11 and displays the selected area 12, as shown by the area enclosed by the dashed line in the figure. Figure 7B As shown in (b), the electronic device identifies two types of objects to be removed within the selected area 12: one type is face occlusions 13 (i.e., mask images), and the other type is non-face occlusions 14 (i.e., bird images). The electronic device uses different removal and restoration algorithms to process the face occlusions 13 and non-face occlusions 14 in the first image 11 respectively, to obtain the second image. Figure 7B As shown in (c), the electronic device displays a second image 15; when the second image 15 is compared with the first image 11, both the mask image and the bird image are eliminated, and the area where the mask image is located is replaced with a facial feature, while the area where the bird image is located is replaced with a background image.

[0196] Figure 7C In response to Figure 7B The interface diagram shown illustrates the detailed processing flow. (Reference) Figure 7C By performing face detection on the preprocessed image using a face detection model, a masked face image can be obtained. Then, the SAM image segmentation model is used to perform segmentation prediction on the preprocessed image, resulting in a masked segmentation image; for example, the masked segmentation image includes masks for birds and masks for face masks. In the masked segmentation image, the regions containing birds and masks are both represented as 1 (white areas); other regions are represented as 0 (black areas). Therefore, through segmentation prediction, the selected area containing birds and masks, awaiting removal, can be identified.

[0197] Then, the intersection of the masked face image and the masked segmented image is calculated. (Reference) Figure 7C The intersection results are as follows: the "mask" in the segmented image intersects with the face image, while the "bird" in the segmented image does not intersect with the face image.

[0198] Then, the parts of the segmented image that intersect with the face image (i.e., the "mask") are identified as face-occluded images, and the parts of the segmented image that do not intersect with the face image (i.e., the "bird") are identified as non-face-occluded images.

[0199] Then, different removal and restoration algorithms are used to process the face-occluded image and the non-face-occluded image to obtain the first local restoration image (face texture) and the second local restoration image (background texture).

[0200] Finally, the first and second partially repaired images are pasted back onto the first image to obtain the second image.

[0201] The first and second elimination and repair algorithms used in the above scenario are described in detail below.

[0202] The first removal and repair algorithm (used to remove facial obstructions)

[0203] Figure 8 This is a flowchart of the first elimination and restoration algorithm used in the method for removing facial occlusions provided in this application embodiment. Assume that the original image contains facial occlusions such as sunglasses or masks. The user's image processing requirement is: through intelligent elimination operations, can the facial occlusions such as sunglasses or masks be removed, and can the removed facial area be restored?

[0204] First, a face detection algorithm is used to detect faces in the original image. A square region containing facial features is then cropped from the original image to obtain the first input image. After scaling, the first input image is of a preset size (e.g., 512*512). Facial key point maps can be obtained from the first input image. Furthermore, the first input image can be processed by adding noise and masking to obtain the second input image. The second input image is a mask image, and the masked area in this mask image represents the face occlusion area. The second input image is also of a preset size (e.g., 512*512).

[0205] Simultaneously, a face detection algorithm is used to detect faces in a reference image containing facial features. A square region containing facial features is then cropped from the reference image to obtain a third input image (e.g., 224*224). This third input image is then fed into an image feature extraction network and a face feature extraction model to obtain image feature information (e.g., 257*1280) and face feature information (e.g., 1*512). These two feature information pieces are then converted into reference face feature information (e.g., 24*768) through a network mapping module.

[0206] The first input image, the second input image (facial landmark map), and reference facial feature information are input into the ControlNet model to obtain the first feature image. The ControlNet model is used for image generation and control, influencing the style and content of the generated image by controlling different image features (such as depth, normals, and edges).

[0207] Then, each image and feature information is input into the SD-based elimination and restoration model (SD elimination and restoration model for short) for image processing.

[0208] (1) Input the reference face feature information into the cross-attention layer of the U-net network in the SD removal and restoration model. U-Net is based on a convolutional neural network architecture and is capable of image feature recognition and image segmentation.

[0209] (2) Input the first feature image into the U-net network of the SD elimination and repair model.

[0210] (3) Input the second input image (i.e. the mask image of the face occlusion area) into the SD removal and repair model.

[0211] Image processing is performed using the SD elimination and restoration model to obtain a face restoration image (e.g., 512*512).

[0212] In this embodiment, the first feature image, the mask image of the face occlusion area, and the reference face feature information are input into the SD removal and restoration model. The model inference is performed based on the head pose correction algorithm to obtain the face restoration image.

[0213] The head posture correction algorithm includes facial orientation correction parameters and / or facial feature key point correction parameters.

[0214] In this embodiment of the application, when it is found that there is a difference between the face orientation in the reference image and the face orientation in the original image, the face orientation can be corrected by the face orientation correction parameter so that the generated face image is consistent with the entire face image in terms of orientation, thereby improving the image restoration effect.

[0215] In this embodiment of the application, when it is found that there are differences between the facial key points in the reference image and the facial key points in the original image, the facial key points can be corrected by the key point correction parameters so that the key points of the generated face image are coordinated, thereby improving the image restoration effect.

[0216] Then, face-swapping is performed on the restored face image and the original image; that is, the restored face image is pasted back onto the face-occluded area of ​​the original image to obtain the face-swapped image. This image does not contain any face occlusions.

[0217] The second elimination and repair algorithm (used to eliminate non-face occlusions)

[0218] Figure 9 This is a flowchart of the second removal and restoration algorithm used in the method for removing face occlusions provided in this application embodiment. An adaptive mask image is performed on the initial mask image to prevent random generation. An image encoder encodes both the expanded mask image and the original image to obtain image features. These image features are then input into the U-net network of the SD removal and restoration model. Simultaneously, panoramic segmentation is performed on the original image to obtain image semantic information, including positive and negative text. This image semantic information is then input into the cross-attention layer of the U-net model. The SD removal and restoration model generates latent space features based on the image features and image semantic information. An image decoder decodes these latent space features to obtain the restored image.

[0219] The specific implementation of the second elimination and repair algorithm will be described in detail below.

[0220] Example 2: The object to be eliminated is a face obstruction.

[0221] In scenario one, if a user selects an object in a single selection operation, and the object is entirely within the face-occluded area, then the first elimination and repair algorithm can be used to eliminate and repair the object. The repaired image can then be pasted back into the original image to achieve the purpose of image elimination and repair.

[0222] For example, a user selects "sunglasses" in the original image through a selection operation, with the entire "sunglasses" positioned on the face. In response to the user's selection operation, the electronic device recognizes the object to be removed as "sunglasses" and uses a first removal and repair algorithm to remove and repair the "sunglasses".

[0223] The proposed solution can eliminate facial obstructions and fill in the eliminated facial areas based on the portrait features in the reference image, thereby improving the portrait restoration effect.

[0224] Scenario 2: If a user selects an object in a single selection operation, with the first part of the object located in the face area and the second part located in the non-face area, then the first elimination and repair algorithm can be used to eliminate and repair the first part of the object, and the second elimination and repair algorithm can be used to eliminate and repair the second part of the object. Then, the two repaired parts are stitched together, and the stitched repaired image is pasted back into the original image to achieve the purpose of image elimination and repair.

[0225] For example, a user selects an object in the original image through a selection operation. For instance, the object to be removed is a scarf, part of which is on the face and another part is not (covering the background area). The user wants to remove the scarf. In response to the user's selection operation, the electronic device recognizes the object to be removed as a scarf and splits the scarf image into the face-occluded area and the non-face-occluded area. Then, a first removal and repair algorithm is used to remove and repair the portion of the scarf that is on the face, and a second removal and repair algorithm is used to remove and repair the portion of the scarf that is in the background area.

[0226] It should be noted that the above embodiment of this application takes as an example that the objects to be eliminated include both facial occlusions and non-facial occlusions, and different elimination and restoration algorithms are used to process the images of the facial occlusions and non-facial occlusions respectively, to illustrate the image processing method provided by this application embodiment. In actual implementation, by classifying and splitting the segmented image, facial occlusion images are obtained, but non-facial occlusions do not exist. In this case, the objects to be eliminated include facial occlusions but not non-facial occlusions, so facial elimination and restoration algorithms can be directly used to process the images of the facial occlusions, which will not be elaborated here.

[0227] Example 3: The object to be eliminated is a non-facial obstruction.

[0228] In Example 3, the user selects an object that is not a face obstruction by making a single selection operation.

[0229] Figure 10A For based on Figure 5 The flowchart shown illustrates the processing flow when the object to be removed does not include facial obstructions. Figure 10A As shown, the method includes an image selection stage, a scene recognition stage, and an elimination and repair stage, wherein the image selection stage and the scene recognition stage are the same as those described above. Figure 7A The implementation method is the same as in [previous version], so it will not be repeated here. The following mainly explains the differences. [and] Figure 7A The difference is: in Figure 10A In this model, the face image and the segmented image do not overlap, thus determining that the object to be removed does not contain face occlusions, but only non-face occlusions. The non-face occlusions and the preprocessed image are input into the second removal and restoration model. The second model removes and restores the non-face occlusions, obtaining a locally restored image (background texture). Then, the locally restored image is pasted back onto the first image to obtain the second image.

[0230] Figure 10B This is a schematic diagram illustrating the application of the image processing method provided in this application to the interactive interface of an electronic device. For example... Figure 10B As shown in (a), the electronic device receives a selection operation from the user on the first image 21 and displays the selected area 22, as shown by the area enclosed by the dashed line in the figure. Figure 10B As shown in (b), the electronic device identifies the object to be removed in the selected area 21 as a non-face occlusion 22 (i.e., the trash can image). The electronic device uses a second removal and repair algorithm to process the non-face occlusion 22, obtaining a second image. Figure 10B As shown in (c), the electronic device displays a second image 23. Compared with the first image 21, the trash can image in the selected area 21 is removed from the second image 23, and the area where the trash can image is located is replaced with the background image.

[0231] Figure 10C In response to Figure 10B The interface diagram provided illustrates the detailed processing flow. Since there is no overlap between the face image and the segmented image, it can be determined that the object to be removed only includes non-face-occluded images, excluding face occlusions. The non-face-occluded images are then removed and repaired using a second removal and restoration model.

[0232] The implementation of the second elimination and repair algorithm provided in the embodiments of this application is described below.

[0233] The second removal and restoration algorithm can remove images from non-face-occluded areas and generate background textures in the removed areas. This application embodiment improves the removal and restoration model, avoiding the generation of foreground object images such as portraits or objects in the removed areas, thus enhancing the image removal and restoration effect. The improvements and effects of this application embodiment on the removal and restoration model are described in detail below. The SD removal and restoration model is used as an example for illustrative explanation.

[0234] Among them, the SD-inpainting image inpainting network model can be simply referred to as the SD removal and inpainting model. The SD removal and inpainting model is a method based on a stable-diffusion model. This method has a high number of parameters and high computational cost. Because this method is trained on large-scale image and text data, it has stronger image and text matching and background generation capabilities. Therefore, the SD removal and inpainting model can handle the removal of large areas and complex backgrounds, and can repair and complete specified regions (masked regions) in an image. The SD removal and inpainting model can include an encoder network, a denoising U-net network, and a decoder network.

[0235] To address the aforementioned problem of arbitrary generation, this application improves the SD elimination and repair model in the following two aspects:

[0236] Foreground and background decomposition training and inference based on panoramic segmentation: This approach suppresses the regeneration of foreground objects. Through this method, model inference can be performed based on positive and negative text. The model inference tends to generate background pixel information corresponding to positive text in the eliminated areas, while suppressing the generation of foreground object pixel information corresponding to negative text. Therefore, after model inference, no other foreground object images such as portraits or objects will be generated in the eliminated areas, thus improving the image elimination and restoration effect.

[0237] Adaptive Mask Expansion: For the model inference stage of the above embodiments, adaptive mask expansion based on brightness contrast is adopted to further suppress the regeneration of foreground objects. Experimental tests revealed that when there are shadow areas around the object to be removed in the first image, these shadow areas can affect image restoration to some extent. If the mask of the object to be removed does not include the shadow areas, other foreground object images, such as portraits or objects, may be generated in the already removed areas.

[0238] The above-described scheme of this application performs dilation processing on the mask of the object to be eliminated, so that the mask after dilation includes the shadow area. This can eliminate both the object to be eliminated and the shadow area. Then, when performing image restoration on the eliminated area, the shadow area can be avoided from affecting the image restoration, thereby avoiding the generation of other foreground object images such as portraits or objects in the eliminated area. Therefore, the effect of image elimination and restoration can be improved.

[0239] Training and inference based on foreground and background decomposition for panoramic segmentation

[0240] The structure of the diffusion model provided in this application will be described below with reference to the accompanying drawings. Figure 11A A schematic diagram of a diffusion model is shown. (Reference) Figure 11AAs shown, the diffusion model provided in this application includes a pixel space module (or pixel space domain), a latent space module (or latent space domain), and a text input-based adjustment module.

[0241] The pixel space module includes an encoder ε and a decoder D. The incomplete image I_m and mask image M of the sample image I are input into the latent space module after being processed by the encoding unit. The latent space module performs image removal and restoration processing and generates image features. These image features are processed by the decoding unit and output to the pixel space module, with the sample image I serving as a monitoring signal.

[0242] The latent space module includes a diffusion processing module and a denoising U-net network.

[0243] The diffusion processing module performs diffusion processing on the input image. This diffusion processing includes forward diffusion and backward diffusion. Forward diffusion involves progressively corrupting the data structure by applying noise sampled from a normally distributed distribution until the image converges to pure noise. This process is implemented using a parameterized Markov chain, where the amount of noise at each time step is not constant but scaled by a time schedule to the mean and variance. Backdiffusion, on the other hand, is the process of learning how to generate a high-resolution image from Gaussian noise input. This involves training a neural network to predict the noise at each time step, enabling the model to generate data from the noise.

[0244] In the embodiments of this application, reference is made to Figure 11A As shown, x represents the image input to the latent space domain. After diffusion processing, x becomes x t x t This represents the implicit representation of the image at time step t.

[0245] The text-based adjustment module can input text (such as foreground and background text) into the hidden space module.

[0246] It should be noted that, unlike the traditional diffusion model, the improved diffusion model in this application adds a text-based adjustment module. By adding foreground and background text inputs for model training, the diffusion model can learn the foreground and background texts. During model inference, it can perform noise prediction based on the foreground and background texts. By strengthening the noise prediction of the background text and suppressing the noise prediction of the foreground text, it can favor the reference background pixels and suppress the reference foreground object pixels when regenerating the image in the eliminated region. This can suppress the problem of regenerating foreground objects.

[0247] The U-net network is a neural network architecture for image segmentation. It combines features from both the encoder and decoder, preserving spatial information through skip connections to achieve accurate segmentation of specific objects in an image. By incorporating attention-based convolutional blocks from a diffusion model into the U-net network, global image connections and multi-scale prediction and supervision can be achieved, further improving feature extraction capabilities and segmentation accuracy.

[0248] In the embodiments of this application, reference is made to Figure 11A As shown, the U-net network includes processing modules such as a self-attention layer (denoted as QKV). The input parameters of the U-net network include the image x. t Text input, etc. The U-net network can acquire data for image x. t The noise prediction value is predicted at each time step t, and the image features x at the next time step can be predicted. t-1 In the embodiments of this application, the U-net network can predict noise values ​​based on different text inputs. For example, the U-net network can predict foreground noise and background noise.

[0249] In the embodiments of this application, reference is made to Figure 11A As shown, for a diffusion model used to perform an image removal task, the training process may include: based on a sample image I and a mask region M of the object to be removed in the sample image I, setting the region of the object to be removed in the sample image I (referred to as the region to be removed) to 0, to obtain a fragmented image I_m; then, inputting the fragmented image I_m, the mask M, and the text text into the diffusion model, and using the complete sample image I as a supervision signal to train the diffusion model.

[0250] In this embodiment, the diffusion model trained is guided by a classifier-free model inference to generate background information and suppress the problem of regeneration of foreground objects.

[0251] Classifier-free guidance is a technique in machine learning that involves using different models or algorithms to guide the generation process of data or images without relying on a traditional classifier. In image generation, classifier-free guidance can alleviate the training cost of conditional generation by using the same model to represent both unconditional and conditional generation simultaneously, thus improving efficiency. For example, in the SD model, classifier-free guidance is implemented using gradient prediction models, one for unconditional generation and another for conditional generation. This approach does not require training an additional classifier; instead, it achieves this by training both targets simultaneously during training.

[0252] It should be noted that in this embodiment, the input image is decomposed into a foreground object region and a background region. When the sample mask M covers the foreground object region (i.e., the sample mask M is a foreground sample mask), a foreground text input (denoted as Obj) is added for training. When the sample mask M covers the background region (i.e., the sample mask M is a background sample mask), a background text input (denoted as Ctxt) is added for training.

[0253] This approach allows the model to generate background information during the inference phase using classifier-free guidance (CFG), thus suppressing the regeneration of foreground objects. Specifically, by employing classifier-free model inference, the model generates background information using foreground masks and foreground text input (as negative text), and background masks and background text input (as positive text), thereby suppressing the regeneration of foreground objects.

[0254] The implementation process of the model training phase and the model inference phase is explained in detail below.

[0255] refer to Figure 11B The model training phase includes the following steps 1 to 3:

[0256] Step 1: Panoramic Image Segmentation

[0257] Panoramic image segmentation allows for the differentiation of foreground and background in an image. A sample image I is input into the panoramic segmentation model to obtain foreground segmentation results and semantic segmentation results of the background regions in the image. The foreground segmentation results can be instance segmentation results for each foreground object (e.g., a person, an animal, etc.). For example, ... Figure 11B The panoramic segmentation result shown in the left image is as follows: the white part represents the foreground instance region, and the black part represents the background region.

[0258] Step 2: Foreground and background mask generation based on semantic understanding

[0259] In some embodiments, the panoptic segmentation result is a dataset uniformly distributed in the interval [0,1]. Sampling is performed from the dataset uniformly distributed in the interval [0,1], with the sampling probability denoted as P.

[0260] refer to Figure 11B When P>0.5, a foreground instance region is randomly selected, and this foreground instance region is set as the sample mask M (i.e., the foreground sample mask). The corresponding specific text input Obj (i.e., the foreground text) is generated, and the learnable text Obj is used as the text input for this training.

[0261] refer to Figure 11BWhen P≤0.5, a background region is randomly selected as the sample mask M (i.e., the background sample mask), and the corresponding specific text input CTXT (i.e., the background text) is generated. The learnable text CTXT is used as the text input for this training.

[0262] It should be noted that "randomly selecting a background region" includes the following situations: a background region can be selected from region A in the image other than the foreground instance region; or a background region can be selected from region B containing the foreground instance as a background sample mask. In some embodiments, the area of ​​the background sample mask in the selected region B is larger than the area of ​​the foreground instance.

[0263] In this embodiment, the dataset corresponding to the panoramic segmentation result is divided into two parts. One part contains a randomly selected foreground instance region, which is used as a foreground sample mask, and the learnable foreground text Obj is used as the text input for this training. The other part contains a randomly selected background region, which is used as a background sample mask, and the learnable background text CTXT is used as the text input for this training.

[0264] The foreground sample mask and the background sample mask are collectively referred to as foreground and background masks.

[0265] Step 3: Train the model using sample image I, sample mask M, and text.

[0266] Refer again Figure 11B Using sample image I, sample mask M (including foreground sample mask and background sample mask), and text (including foreground text Obj and background text CTXT), the following is performed: Figure 11A The diffusion model shown is trained.

[0267] For example, the model can be trained using a background sample mask and background text as the training set and sample images as the supervision signal; and the model can be trained using a foreground sample mask and foreground text as the training set and sample images as the supervision signal.

[0268] Through steps 1 to 3 above, the sample image I is first subjected to panoramic segmentation to distinguish the foreground instance region and the background region. Then, based on the semantic understanding of the panoramic segmentation result, a foreground sample mask and a background sample mask are generated, along with foreground text Obj and background text CTXT. The learnable foreground text Obj and learnable background text CTXT are then used as text inputs for model training. The model is trained using the sample image I, the foreground sample mask, the background sample mask, the foreground text Obj, and the background text CTXT. This yields the improved diffusion model provided in this embodiment. By using this improved diffusion model, image removal and restoration can be achieved, and the regeneration of foreground objects can be suppressed, greatly improving the image removal and restoration effect.

[0269] After model training is completed, the model inference phase (i.e., the model application phase) begins.

[0270] In this embodiment of the application, the model inference stage may include the following steps 4 and 5.

[0271] Step 4: Image panoramic segmentation and acquisition of positive / negative text

[0272] Figure 11C A schematic diagram illustrating the implementation process of the model inference stage is shown. (Reference) Figure 11C The first image to be inferred is input into the panoramic segmentation model to obtain the foreground and background category information of the first image. The background category information and CTXT are used as positive text, and the foreground category information and Obj are used as negative text.

[0273] Positive text can be represented as: CTXT + background category information.

[0274] For example, positive text: CTXT, door, wall, etc.

[0275] Negative text can be represented as: Obj + foreground category information.

[0276] For example, negative text: Obj, person, human, etc.

[0277] Step 5: Classifier-free model inference

[0278] After obtaining the positive and negative text, model inference can be performed based on the positive and negative text in a classifier-free manner.

[0279] During model inference, based on the classifier-free approach, the model tends to generate information corresponding to positive text, i.e., background pixel information, while suppressing information corresponding to negative text, i.e., suppressing foreground object pixel information, thereby suppressing the problem of random generation of foreground objects.

[0280] The following describes the classifier-free model inference process provided in the embodiments of this application.

[0281] Step 5.1: Obtain the noise prediction value

[0282] Equation 1 below represents the input and output parameters of the diffusion model.

[0283] N = Unet(t, text, x) t (Equation 1)

[0284] Here, Unet() is the functional form of the diffusion model. t, text, and x t represents the input parameters of the diffusion model Unet. N represents the output parameters of the diffusion model Unet.

[0285] Specifically, t represents the time step, text represents text input; x t This represents the implicit representation of the image at each time step t. N represents the noise prediction value of the diffusion model Unet at each time step t.

[0286] In the embodiments of this application, the diffusion model Unet() can predict noise values ​​based on different text inputs, such as obtaining foreground noise prediction values ​​or background noise prediction values.

[0287] On the one hand, for cases where the text input is positive text, replace "text" in Equation 1 above with the positive text "text". P Referring to Equation 2 below, the diffusion model Unet() can be based on the positive text text at each time step t. P Noise prediction is performed using (including CTXT and background category information) to obtain the background noise prediction value (denoted as Npos).

[0288] Npos = Unet(t, text) P x t (Equation 2)

[0289] On the other hand, for cases where the text input is negative text, replace "text" in Equation 1 above with the negative text "text". N Referring to Equation 3 below, the diffusion model Unet() can be based on negative text at each time step. N(Including Obj and foreground category information) to perform noise prediction, and obtain the foreground noise prediction value (denoted as Nneg).

[0290] Nneg = Unet(t, text) N x t (Equation 3)

[0291] In this embodiment of the application, given the predicted background noise value Npos and the predicted foreground noise value Nneg, the diffusion model Unet() can perform a classifier-free algorithm based on the predicted background noise value Nneg and the predicted foreground noise value Npos to obtain the predicted image noise value.

[0292] For example, the equation for the classifier-free algorithm can be found in Equation 4 below. The diffusion model Unet() can be calculated based on the background noise prediction value Nneg, the foreground noise prediction value Npos, and the hyperparameter α to obtain the image noise prediction value (denoted as Npred).

[0293] Npred=Nneg+α*(Npos-Nneg) (Equation 4)

[0294] Where Npred represents the image noise prediction value obtained by the diffusion model at each time step, Npos represents the background noise prediction value when the input text is positive text, and Nneg represents the foreground noise prediction value when the input text is negative text. α is a hyperparameter. For example, α can be set to 7.5.

[0295] In this embodiment, panoramic segmentation is performed on the image input to the diffusion model to obtain background and foreground category information. The background text (CTXT) and background category information are used as positive text, while the foreground text (Obj) and foreground category information are used as negative text. Then, classifier-free model inference is performed based on the positive and negative texts. During inference, the background noise prediction value of the diffusion model at each time step is obtained based on the positive text, and the foreground noise prediction value is obtained based on the negative text. Based on the background and foreground noise prediction values ​​at each time step, the image noise prediction value of the diffusion model at each time step can be calculated. This method enhances the background noise prediction value and suppresses the foreground noise prediction value, making the image inpainting process more inclined to reference background pixels and suppressing reference to foreground object pixels, thereby avoiding the problem of regenerating foreground objects in the removed areas.

[0296] The above describes the image noise prediction values ​​at each time step based on positive and negative text provided in the embodiments of this application. The process of repairing the image based on the noise prediction values ​​is described below.

[0297] Step 5.2: Obtain the RGB image based on the noise prediction value.

[0298] It should be noted that the image noise prediction value of the diffusion model at each time step follows a Gaussian distribution, denoted as N(0,1).

[0299] First, sample the initial noise x from the Gaussian distribution N(0,1). T Where T represents the initial sampling time step, and T can be set to 1000.

[0300] Then, iterate through the time steps t in Equation 5 from T to 1:

[0301] (1) When t = 1, let z = 0. When t > 1, sample random noise z from the Gaussian distribution N(0, 1). Figure 11D The results of multi-noise sampling for the diffusion model are shown.

[0302] (2) Predict the image feature x at the next time step using Equation 5 t-1 :

[0303]

[0304] α t =1-β t ,

[0305]

[0306] Where, β t Through β T The difference between β1 and β2 is obtained, β2 T =0.02, β1 = 1e-4.

[0307] It should be noted that, referring to Equation 1 above, Npred represents the image noise prediction value of the diffusion model at each time step, and N in Equation 5... t Let Npred be the value at step t.

[0308] Where, x t x represents the image features at the current time step t. t-1 The image features represent the next time step (t-1).

[0309] (3) By using Equation 5, iterating from T to 1 through t, the image feature x at each time step can be predicted. 999 x 998 , ..., x1, and finally obtain the image feature x0 at time step 0.

[0310] Image feature x0 is input into the decoding unit of the diffusion model. The decoding unit decodes the image feature x0 to obtain an RGB image.

[0311] The proposed solution involves inputting the first image and the mask of the object to be removed into the SD removal and repair model. After model inference, an RGB image after image removal and repair processing can be output.

[0312] For example, suppose portrait 1 in the first image is selected as the object to be removed. The first image and the mask of portrait 1 are input into the SD removal and restoration model trained above. In the SD removal and restoration model, the first image is segmented panoramically to obtain positive and negative text. Then, the positive and negative text are used to complete model inference based on classifier-free guidance. The model inference can generate an RGB image in which portrait 1 is no longer displayed, that is, portrait 1 is removed, and the area where portrait 1 is removed is restored as the background image. No other portraits or objects are generated.

[0313] In the scenario described above, the SD elimination and repair model provided in this application embodiment can be used for model inference, which can effectively avoid the problem of regenerating foreground objects in the eliminated area.

[0314] Adaptive mask expansion

[0315] This application employs an adaptive mask expansion based on brightness contrast, which can further suppress the problem of foreground object regeneration. The starting point of this improvement scheme is that, after multiple experimental tests, it was found that in some special cases, there are shadow areas near the object to be eliminated in the original image. If the initial mask input to the diffusion model does not cover the shadow areas near the object to be eliminated in the image, the diffusion model is still prone to generating random foreground objects.

[0316] For example, refer to Figure 12A As shown in (a) and (b), the object to be removed is a car, and there is a shadowed area near the object. The initial mask does not cover the shadowed area near the object (e.g., the car) in the image. (Reference) Figure 12A As shown in (c), when the initial mask is input into the diffusion model and sampled multiple times, other image features (e.g., other cars) are regenerated in the areas where the car was eliminated, resulting in the problem of random foreground generation. (See reference...) Figure 12A As shown in (d) and (e), by expanding the mask region to cover the shadow region, inputting the expanded mask into the diffusion model and sampling it multiple times, no other image features are generated where the car is eliminated in the multiple sampling results, and the phenomenon of random foreground generation is significantly improved.

[0317] Multiple tests revealed that residual information about objects (such as shadows) can cause serious foreground regeneration problems.

[0318] Based on this discovery, embodiments of this application propose an adaptive mask expansion implementation method based on brightness contrast to avoid the phenomenon of random foreground generation. Figure 12B A schematic diagram of the adaptive mask expansion based on brightness contrast is shown, which includes the following steps:

[0319] Step 1: Panoramic Image Segmentation

[0320] refer to Figure 12B As shown, panoramic segmentation is performed on the input image I to obtain the foreground object mask and the background mask (denoted as M_back). The dark area represents the background mask, and the colored area represents the foreground object mask.

[0321] In this embodiment of the application, a foreground object mask and a background mask are obtained through panoramic image segmentation, thereby determining the foreground object region and the background region in the image, which is used to limit the expansion of the mask.

[0322] Specifically, when inflating the initial mask M of the object to be eliminated, the inflating of the mask can be limited based on the foreground object mask and the background mask to prevent the inflated mask from covering the foreground object area. Finally, the area that may be a shadow is added to the final mask (denoted as M_final).

[0323] It should be noted that if the mask covers a foreground object region after dilation, the diffusion model will eliminate that foreground object when processing the image based on the dilated mask. Therefore, during mask dilation, the initial mask M is dilated, but the constraint must be met: the dilated mask cannot cover the foreground object region to avoid the problem of mistakenly eliminating other foreground objects.

[0324] Step 2: Inflate the initial mask

[0325] The initial mask refers to the mask of the object to be removed in the input image.

[0326] In this embodiment of the application, the size of the expansion kernel is set to b, and the initial mask M is expanded to obtain the expanded mask (denoted as M_hat). The expanded mask M_hat (the first mask) can cover other areas near the area to be eliminated.

[0327] It should be noted that since shadows are usually closely connected to objects, the purpose of dilating the input mask is to include nearby possible shadow areas through mask dilution.

[0328] In the embodiment of the present application, an intersection can be obtained from the dilated mask M_hat and the background mask M_back, to obtain an intersection mask (the second mask). The intersection mask does not include other foreground objects, which can avoid false elimination of foreground objects.

[0329] The above intersection process can be expressed as: M h ′ at = M hat ∩M back .

[0330] wherein, M h ′ at represents an intersection mask, M hat represents a dilated mask, and M back represents a background mask.

[0331] Step 3: Pixel clustering based on superpixels

[0332] Wherein, pixel clustering is a technology for dividing pixels in an image into different groups or categories, and is usually used to identify pixels with similar brightness or color features.

[0333] In the embodiment of the present application, after an initial mask M is dilated, a dilated mask M_hat is obtained, then the dilated mask M_hat is differentiated from the initial mask M to obtain a mask M_res (a third mask). As shown in Figure 12B , the yellow area around the object to be eliminated is the mask M_res obtained after differentiation.

[0334] In the embodiment of the present application, after the mask M_res is obtained, superpixel clustering is performed on the mask M_res, so as to divide the mask M_res into blocks to obtain a plurality of pixel blocks. For example, the number of clustering centers can be set to 10. As shown by the yellow area in Figure 12B , after clustering, the dilated area is clustered into a plurality of pixel blocks.

[0335] Step 4: Select the expansion region that meets the requirements based on brightness contrast.

[0336] In the embodiment of the present application, after the mask M_res is divided into a plurality of pixel blocks, a brightness average value is calculated separately for each pixel block (denoted as L_i), and then a brightness average value of the entire image is calculated (denoted as L).

[0337] Refer to Figure 12B , the brightness average value L_i of each pixel block is compared with the brightness average value L of the entire image.

[0338] If L_i<L, it is considered that the current pixel block is highly likely to belong to a shadow area, so the pixel block is retained in the mask M_res.

[0339] If L_i≥L, then the current pixel block is considered not to belong to the shadow region, so the pixel block is removed from the mask M_res to obtain the processed mask M_res.

[0340] refer to Figure 12B The final mask M_final includes the initial mask M and the processed mask M_res.

[0341] The above processing inflates the initial mask, encompassing potential shadow areas while ensuring the inflated area doesn't cover foreground objects, thus preventing accidental deletion of other foreground objects. Specifically, the inflated area is divided into blocks, and the average brightness of each block is calculated. If the average brightness of a pixel block is greater than or equal to the average brightness of the entire image, that pixel block is not a shadow area and is excluded from the final mask M_final.

[0342] The embodiments of this application employ adaptive mask expansion based on brightness contrast, so that the expanded mask includes possible shadow areas near the object to be eliminated, which can suppress the problem of foreground object regeneration caused by shadows.

[0343] Applying the final mask M_final to the above model training and inference can suppress the problem of foreground object regeneration caused by shadows. Figure 12C The results of multi-noise sampling are shown when adaptive masking based on brightness contrast is employed. For example... Figure 12C As shown, no foreground object regeneration problem occurred in any of the images from the multi-noise sampling results.

[0344] It should be noted that the adaptive mask expansion scheme based on brightness contrast provided in this application is not limited to the SD elimination and repair model trained in the above embodiments. In practical applications, the adaptive mask expansion scheme based on brightness contrast can also be applied to conventional SD elimination and repair models to achieve the purpose of suppressing the problem of foreground object regeneration.

[0345] In some embodiments, such as the scenario described above, the SD elimination and repair model provided in this application can be used for model inference, and an adaptive mask expansion scheme based on brightness contrast can be used during the inference process to more effectively avoid the problem of regenerating foreground objects in the eliminated area.

[0346] It should be noted that in the embodiments of this application, "greater than" can be replaced with "greater than or equal to", "less than or equal to" can be replaced with "less than", or "greater than or equal to" can be replaced with "greater than", and "less than" can be replaced with "less than or equal to".

[0347] The various embodiments described herein can be independent solutions or combinations based on internal logic, and all such solutions fall within the protection scope of this application.

[0348] The foregoing mainly describes the solutions provided by the embodiments of this application from the perspective of method steps. It is understood that, in order to achieve the above functions, the electronic device implementing this method includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of protection of this application.

[0349] This application embodiment can divide an electronic device into functional modules based on the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware or as software functional modules. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, other feasible division methods may exist. The following description uses the division of functional modules according to each function as an example.

[0350] Figure 13 This is a schematic block diagram of an image processing apparatus 400 provided in an embodiment of this application. The apparatus 400 can be used to perform the actions performed by the electronic device in the above method embodiments. The apparatus 400 includes an image display unit 410 and an image processing unit 420.

[0351] The image display unit 410 is used to display a first image and activate the intelligent elimination function; in response to the user's first operation on the first image, it displays a selection trajectory, wherein the area enclosed by the selection trajectory is the selection area.

[0352] Image processing unit 420 is configured to identify the category of the object to be eliminated in the selected area; based on the identification result, divide the object to be eliminated into a face-occluded area and a non-face-occluded area; perform image processing on the face-occluded area using a first elimination and repair algorithm to obtain a first local repair image; perform image processing on the non-face-occluded area using a second elimination and repair algorithm to obtain a second local repair image; and generate a second image based on the first image, the first local repair image, and the second local repair image.

[0353] The image display unit 410 is also used to display a second image.

[0354] The image processing apparatus provided in this application embodiment, in response to a user's selection operation in an image, can automatically identify the category of the object to be removed in the image. Based on the identification result, the object to be removed is divided into face-occluded areas and non-face-occluded areas. Then, different removal and repair methods are applied to the face-occluded areas and non-face-occluded areas respectively for removal and repair. With this solution, users only need one editing operation to achieve the dual effects of removing face occlusions and restoring the portrait, while also removing non-face occlusions and restoring the background, reducing the number of user operations and improving the image editing experience.

[0355] The apparatus 400 according to the embodiments of this application can be used to perform the methods described in the embodiments of this application, and for the sake of brevity, it will not be described again here.

[0356] This application also provides a chip coupled to a memory, which is used to read and execute computer programs or instructions stored in the memory to perform the methods in the above embodiments.

[0357] This application also provides an electronic device including a chip for reading and executing computer programs or instructions stored in a memory, causing the methods in the various embodiments to be performed.

[0358] This embodiment also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the image processing method described above.

[0359] This embodiment also provides a computer program product. The computer-readable storage medium stores program code. When the computer program product is run on a computer, it causes the computer to perform the above-mentioned related steps to implement the image processing method in the above embodiment.

[0360] In this embodiment, the electronic device, computer-readable storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0361] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0362] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0363] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. In the description of embodiments in this application, unless otherwise stated, "multiple" means two or more; for example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0364] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, include: Display the first image and activate the intelligent removal function; In response to the user's first operation on the first image, a selection trajectory is displayed, and the area enclosed by the selection trajectory is the selection area; Identify the category of the object to be eliminated in the selected area. The identification results include face occlusions and occlusions other than faces. Based on the recognition results, the object to be eliminated is divided into a face-occluded area and a non-face-occluded area. The face-occluded area includes the face-occluder, and the non-face-occluded area includes the occluders other than the face. The first elimination and repair algorithm is used to process the image of the face occlusion area to obtain a first local repair image; The second elimination and restoration algorithm is used to process the image of the non-face occlusion area to obtain the second local restoration image; A second image is generated based on the first image, the first partially repaired image, and the second partially repaired image.

2. The method according to claim 1, characterized in that, The process of identifying the category of the object to be eliminated in the selected area includes: The objects to be eliminated in the selected area are segmented and predicted according to the image segmentation model to obtain a segmented image; The face detection process is performed on the first image using a face detection model to obtain a face image; If the segmented image intersects with the face image, then it is determined that the object to be eliminated contains the face occlusion region; If the segmented image and the face image do not intersect, then it is determined that the object to be eliminated does not contain the face occlusion area.

3. The method according to claim 2, characterized in that, The step of dividing the object to be eliminated into a face-occluded region and a non-face-occluded region based on the recognition result includes: The portion of the segmented image that intersects with the face image is determined as the face occlusion region; The portion of the segmented image that does not intersect with the face image is identified as the non-face occlusion region.

4. The method according to claim 1, characterized in that, The object to be eliminated is the first object; The step of splitting the object to be eliminated into a face-occluded region and a non-face-occluded region includes: The first image region where the first object is located is defined as the face occlusion region; The second image region where the first object is located is determined as the non-face occlusion region.

5. The method according to claim 1, characterized in that, The objects to be eliminated include a first object and a second object; The step of splitting the object to be eliminated into a face-occluded region and a non-face-occluded region includes: The image region where the first object is located is defined as the face occlusion region; The image region where the second object is located is determined as a non-face-occluded region.

6. The method according to any one of claims 1 to 5, characterized in that, The first elimination and restoration algorithm is used to process the image of the occluded facial area to obtain a first local restoration image, including: Obtain a reference image, which is a frame preceding or following the first image, and the reference image contains facial features; Reference facial feature information is obtained based on the reference image; The reference facial feature information, the original image of the face occlusion area, and the mask image of the face occlusion area are input into the control network model to obtain the first feature image; The first feature image, the mask image of the face occlusion area, and the reference face feature information are input into the first elimination and repair model to obtain the first local repair image, which includes face texture.

7. The method according to claim 6, characterized in that, The step of inputting the first feature image, the mask image of the face occlusion region, and the reference face feature information into the first elimination and restoration model to obtain the first local restoration image includes: The first feature image, the mask image of the face occlusion area, and the reference face feature information are input into the first elimination and repair model. The model inference is performed based on the head pose correction algorithm to obtain the first local repair image. The head posture correction algorithm includes facial orientation correction parameters and / or facial feature key point correction parameters.

8. The method according to claim 6, characterized in that, The step of obtaining reference facial feature information based on the reference image includes: The reference image is input into an image feature extraction network to generate first feature information; The reference image is input into the face feature extraction model to generate second feature information; The reference face feature information is obtained by mapping the first feature information and the second feature information.

9. The method according to claim 6, characterized in that, The first elimination and repair model is an elimination and repair model based on stable diffusion SD, which includes a denoising U-net network.

10. The method according to any one of claims 1 to 5, characterized in that, The second elimination and restoration algorithm is used to process the image of the non-face-occluded area to obtain a second local restoration image, including: Based on the original image of the non-face occlusion area, positive text and negative text are obtained. The positive text is used to enhance the generation of background texture, and the negative text is used to suppress the generation of foreground object texture. The original image and mask image of the non-face-occluded area are input into the second elimination and repair model. Based on the positive text and the negative text, model inference is performed to obtain the second local repair image, which includes background texture.

11. The method according to claim 10, characterized in that, The step of performing model inference based on the positive and negative text to obtain the second local repair image includes: Based on the following equation, using a classifier-free approach, the predicted image noise value is calculated based on the positive and negative text: in, Npos This represents the predicted background noise value obtained based on the positive text prediction. Nneg This represents the foreground noise prediction value obtained based on the negative text prediction. α For hyperparameters, Npred This represents the image noise prediction value obtained by the second noise reduction and restoration model at each time step; Based on the image noise prediction value Npred Perform multi-noise sampling to obtain a multi-noise sampled image; The second local repair image is obtained based on the multi-noise sampled image.

12. The method according to claim 10, characterized in that, The step of obtaining positive and negative text from the original image of the non-face-occluded area includes: The original image of the non-face-occluded area is subjected to panoramic segmentation processing to obtain background category information and foreground category information; The positive text is generated based on the background category information and background text, and the negative text is generated based on the foreground category information and foreground text. The positive text is used to enhance the generation of the background texture corresponding to the background category information, and the negative text is used to suppress the generation of the foreground object texture corresponding to the foreground category information. The background text and the foreground text are texts learned by the second elimination and repair model through training.

13. The method according to claim 10, characterized in that, Before inputting the original image and mask image of the non-face-occluded region into the second removal and restoration model, the method further includes: Adaptive mask expansion is performed on the mask image.

14. The method according to claim 10, characterized in that, The second elimination and repair model includes an encoding network, a denoising U-net network, and a decoding network.

15. An electronic device, characterized in that, The electronic device includes: one or more processors, and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 14.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 14.

17. A computer program product, characterized in that, The computer program product includes computer program code that, when run on an electronic device, causes the electronic device to perform the method as described in any one of claims 1 to 14.

Images