Image elimination and restoration processing method, electronic equipment, chip system and medium

Abstract

The invention provides an image elimination and restoration processing method, electronic equipment, a chip system and a medium, and relates to the technical field of image processing. According to the scheme, in response to the operation of the user on the first image, image elimination and background restoration can be carried out on people or objects in the area selected by the user. In actual use, a user can directly select an area on a photo, a portrait image or an object image in the area can be eliminated by one key, a background image can be recovered in the eliminated area, and according to the area proportion of the object to be eliminated in the first image, an appropriate image processing strategy is selected to carry out image elimination and background restoration, so that the user experience is improved. Therefore, image processing is finer, various image elimination requirements can be met from various dimensions such as operand, image processing performance, image processing effect and the like, and the image elimination and restoration effect is improved.

Description

[0001] This application claims priority to the Chinese patent application No. 202410874283.6, filed on June 27, 2024, and entitled “Image processing method, electronic device, chip system and storage medium”, the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of image processing, in particular to an image elimination and repair processing method, an electronic device, a chip system and a storage medium. BACKGROUND

[0003] Scenic spot photography is an important scenario for users to take photos with mobile phones. Due to the large number of scenic spot tourists, strangers or some objects (such as garbage cans) are often accidentally photographed into the frame, affecting the aesthetic degree of the photos. Users want to eliminate the images of strangers or objects in the photos, so there is an urgent need for an image elimination and repair method that is convenient to operate and meets user needs. SUMMARY

[0004] The present application provides an image elimination and repair processing method, an electronic device, a chip system and a storage medium, which can select appropriate image processing methods for image elimination and background repair, making the image processing more precise, and meeting various image elimination needs from various dimensions such as computational complexity, image processing performance and image processing effect, thereby improving the elimination and repair effect.

[0005] In a first aspect, the present application provides an image elimination and repair processing method, which includes: in response to a first operation of a user, enabling an image elimination function; in response to a second operation of the user on a first image, highlighting a selected region in the first image; identifying a to-be-eliminated object in the selected region; determining a target image processing strategy according to a first area ratio of the to-be-eliminated object in the first image; performing elimination and repair processing on the to-be-eliminated object using the target image processing strategy to obtain a second image; the selected region of the second image does not include the to-be-eliminated object, and the region after the to-be-eliminated object is eliminated is repaired as a background image of the to-be-eliminated object; and displaying the second image.

[0006] The background image of the to-be-eliminated object is an image around the to-be-eliminated object.

[0007] The processing method for image elimination and repair provided by the embodiment of the present application can eliminate the image of a person or object in a selected region of a user and repair the background in response to the user's operation on the first image. In actual use, the user can select a region on the photo, and the image of the person or object in the region can be eliminated by one key, and the background image in the eliminated region is restored. The present application selects a suitable image processing strategy for image elimination and background repair according to the area ratio of the object to be eliminated in the first image, so that the image processing is more accurate, and the elimination and repair effect is improved.

[0008] In the present application, elimination can be replaced by erasing, wiping, removing or other words with the same or similar meaning.

[0009] In the present application, repair refers to the restoration of the background image after image elimination.

[0010] In the present application, the mask image can also be called mask image, mask image or mask image, and the mask region can also be called mask region or mask region.

[0011] In the present application, the selected region can be a regular shape such as rectangle or ellipse, or an irregular shape, which is determined according to actual use.

[0012] For example, the first operation is the operation of clicking the intelligent elimination control. The second operation is the operation of selecting or painting. For example, the second operation can be the operation of sliding the finger or dragging the elimination cursor around the image to be eliminated on the first image, thereby forming a regular or irregular selected region.

[0013] The present application provides the following three user scenarios suitable for the present application:

[0014] User scenario one: in response to the user operation, the electronic device enters the gallery interface or the album interface. The gallery interface displays the first image. After the user clicks the edit option in the gallery interface, the electronic device displays the doodling, AI elimination and other options. After the AI elimination option is selected, in response to the user's selection or painting operation on a region in the first image, the electronic device eliminates the object (such as a person or object) in the selected region, and repairs the eliminated region to ensure that the eliminated region is consistent with the background image, thereby improving the image elimination effect.

[0015] User scenario two: in a case where the camera application is in an enabled state, in response to a user operation of clicking a shooting button in a shooting interface, the electronic device shoots a first image and displays a thumbnail of the first image in the shooting interface. In response to a user operation on the thumbnail of the first image, the electronic device jumps to a gallery interface, and the first image is displayed in the gallery interface. After the user clicks an editing option in the gallery interface, the electronic device displays a scribble, AI removal, and the like. After the AI removal option is selected, in response to a user operation of circling or erasing on a certain region in the first image, the electronic device removes an object (for example, a person or an object) in the selected region and performs a repair process on the removed region to ensure that the removed region is consistent with a background image, thereby improving the image removal effect.

[0016] In some possible implementation manners, the removing the to-be-removed object and performing the repair process according to the target image processing strategy to obtain a second image includes: inputting the first image and a target mask image into a target image removal model to obtain a third image; and obtaining the second image according to the third image, a first mask image, and the first image.

[0017] The mask region in the first mask image is a mask region of the to-be-removed object. The target image removal model is determined according to the target image processing strategy. The target mask image is a mask image obtained based on the first image.

[0018] According to the scheme, the target image processing strategy is determined according to the area proportion of the to-be-removed object in the original image, and then a suitable image removal model is determined according to the target image processing strategy to implement image removal, so that better image removal and repair effects can be achieved.

[0019] In some possible implementation manners, the target image processing strategy is determined according to the first area proportion of the to-be-removed object in the first image, including the following possible scenarios:

[0020] Scenario one: when the first area proportion is less than a first threshold, a first image processing strategy is used as the target image processing strategy.

[0021] For example, the first threshold is 2%.

[0022] In actual implementation, for a to-be-removed object with a smaller area proportion, the removal and repair requirements are relatively low, and a smaller image removal model can be used, and the image processing effect meets the basic requirement of small color difference.

[0023] Scenario two: when the first area ratio is greater than or equal to the first threshold value and less than a second threshold value, it is identified whether the background image of the object to be eliminated is a complex background image; if it is not a complex background image, a second image processing strategy is used as the target image processing strategy; if it is a complex background image, a third image processing strategy is used as the target image processing strategy.

[0024] Exemplarily, the second threshold value is 10%.

[0025] In the image elimination scene with a slightly large area ratio and a non-complex background, a small image elimination model can be used, and the image processing effect meets the basic requirements such as small color difference. In the image elimination scene with a slightly large area ratio and a complex background, a better image elimination model is used to obtain a better elimination and repair effect.

[0026] Scenario three: when the first area ratio is greater than or equal to the second threshold value and less than a third threshold value, it is identified whether the background image of the object to be eliminated is a crowd background image; if it is not a crowd background image, a fourth image processing strategy is used as the target image processing strategy; if it is a crowd background image, a fifth image processing strategy is used as the target image processing strategy.

[0027] Exemplarily, the third threshold value is 25%.

[0028] Through the scheme of the present application, for the image elimination scene with a relatively large area ratio and a complex background, it can be identified whether it is a crowd background image elimination scene; if it is a crowd background, a deformed portrait may be generated after eliminating the image, and therefore a powerful image elimination model can be used to avoid generating a deformed portrait.

[0029] Scenario four: when the first area ratio is greater than or equal to the third threshold value, it is determined that the elimination and repair processing is not performed, and a first prompt information is displayed, the first prompt information is used to prompt the user that the current operation is invalid.

[0030] The first threshold value is less than the second threshold value, and the second threshold value is less than the third threshold value.

[0031] Through the above scheme, according to the area ratio of the object to be eliminated in the original image, a suitable image processing method can be selected for image elimination and background repair, so that the image processing is more precise, and the image elimination and repair effect is improved.

[0032] For example, in actual implementation, for a region with a smaller area ratio, the elimination and repair requirements are relatively low, and a GAN-based image elimination model with a smaller operation amount can be used, and the image processing effect can meet the basic requirements such as small color difference; for a region with a larger area, the elimination and repair requirements are higher, and therefore a SD-based image elimination model with better performance is used to obtain better elimination and repair effect.

[0033] Through the scheme, the portrait or object in the selected region of the user can be eliminated, and various image elimination requirements can be met in terms of operation amount, image processing performance, image processing effect, and the like.

[0034] In some possible implementation manners, the inputting the first image and the target mask image into a target image elimination model to obtain a third image includes the following possible implementation manners:

[0035] Implementation manner one: in the case of using the first image processing strategy, inputting the first image and a first mask image into a first image elimination model to obtain the third image;

[0036] Implementation manner two: in the case of using the second image processing strategy for a non-complex background image, inputting the first image and the first mask image into a second image elimination model to obtain a third image;

[0037] Implementation manner three: in the case of using the third image processing strategy for a complex background image, inputting the first image and the first mask image into a third image elimination model to obtain a third image;

[0038] Implementation manner four: in the case of using the fourth image processing strategy for a non-human crowd background image, inputting the first image and a first mask image into a fourth image elimination model to obtain a third image;

[0039] Implementation manner five: in the case of using the fifth image processing strategy for a human crowd background image, inputting the first image and a second mask image into a fifth image elimination model to obtain a third image;

[0040] The second mask image includes a mask region of each portrait in the first image.

[0041] The image elimination model is not limited in the present application.

[0042] For example, the first image elimination model is a generative adversarial network (GAN)-based image elimination model.

[0043] For example, the second image elimination model, the third image elimination model, and / or the fourth image elimination model are respectively a GAN-based image elimination model or a super-duper (SD)-based image elimination model.

[0044] Exemplarily, the fifth image elimination model is an image elimination model based on stable diffusion (SD).

[0045] In actual implementation, for a region with a smaller area ratio, the elimination and repair requirements are relatively low, and a GAN-based image elimination model with a smaller operation amount can be used, and the image processing effect can meet the basic requirement of small color difference; for a region with a larger area, the elimination and repair requirements are higher, and therefore a SD-based image elimination model with better performance is used to obtain better elimination and repair effect.

[0046] In some possible implementation manners, after the object to be eliminated in the selected region is identified, the method further includes: in a case where the object to be eliminated is identified as a human image and the selected region covers part of the image of the human body, determining whether the current elimination operation meets an elimination condition according to a second mask image and a third mask image, wherein a mask region of the third mask image is a mask region corresponding to the selected region; and in a case where it is determined that the current elimination operation meets the elimination condition, obtaining a first area ratio of the object to be eliminated in the first image.

[0047] Through the scheme of the present application, in response to the selection operation of the user on the first image, it can be judged whether the selected region or the smearing region meets the elimination condition, and in a case where the selected region or the smearing region meets the elimination condition, the object to be eliminated is eliminated, and in a case where the selected region or the smearing region does not meet the elimination condition, the object to be eliminated is not eliminated, so as to prevent false elimination, for example, to prevent false elimination of part of the image of the human body.

[0048] In some possible implementation manners, the determining according to the third image, the first mask image and the first image includes: performing AND operation on the third image and the first mask image to obtain a fourth image; performing AND operation on the first image and a fifth image to obtain a sixth image, the fifth image being an image obtained by taking inverse of pixel values of the first mask image; and performing image fusion on the fourth image and the sixth image to obtain the second image.

[0049] In some possible implementation manners, the performing AND operation on the third image and the first mask image includes: performing AND operation on the third image and the first mask image after inflation; and the fifth image is an image obtained by taking inverse of pixel values of the first mask image after inflation.

[0050] Through the scheme of the present application, through mask inflation, image fusion and other processing, better image repair effect can be achieved.

[0051] In some possible implementation manners, the inputting the first image and the target mask image into the target image elimination model to obtain a third image comprises: in a case where the first area ratio is less than or equal to a second threshold, performing image cropping processing on the first image and the target mask image, inputting the cropped first image and the cropped target mask image into the target image elimination model to obtain a seventh image; and filling the seventh image into a cropped region of the first image to obtain the third image.

[0052] According to the solution, for a region with a small area ratio, image cropping processing can be performed on the first image first, so that the object to be eliminated is in a central position or a suitable position or proportion in the cropped image, and the image elimination and repair effect is improved.

[0053] In some possible implementation manners, different image cropping ratios can be used for image cropping for objects to be eliminated with different area ratios. For example, for an object to be eliminated with a very small area ratio (for example, less than 2%), image cropping can be performed according to a first ratio (for example, 6%), so as to improve the image elimination and repair effect. For an object to be eliminated with a medium area ratio (for example, less than 10% and greater than or equal to 2%), image cropping can be performed according to a second ratio (for example, 10%), so as to improve the image elimination and repair effect.

[0054] In some possible implementation manners, the inputting the cropped first image and the cropped target mask image into the target image elimination model to obtain a seventh image comprises: performing second dilation processing on a mask region of the cropped target mask image; inputting the cropped first image and the cropped and dilated target mask image into the target image elimination model to obtain the seventh image.

[0055] According to the solution, through image cropping and mask dilation processing, better image elimination and repair effect can be achieved.

[0056] In some possible implementation manners, the determining whether the background image of the object to be removed is a complex background image comprises: performing first dilation processing on the mask region of the first mask image to obtain a first dilated image; performing second dilation processing on the mask region of the first mask image to obtain a second dilated image; the dilation ratios of the first dilation processing and the second dilation processing are different; performing difference calculation on the second dilated image and the first dilated image to obtain a first annular image; determining a feature point density of the first annular image according to a ratio of a number of feature points of the first annular image to a number of pixels of the first annular image; if the feature point density of the first annular image is greater than a fourth threshold value, it is determined that the background image of the object to be removed is a complex background image; if the feature point density of the first annular image is less than or equal to the fourth threshold value, it is determined that the background image of the object to be removed is not a complex background image.

[0057] By the scheme, whether the background of the object to be removed is a complex background can be determined, and then a suitable image removal model can be used for image removal and background repair, so that the image removal effect can be improved.

[0058] In some possible implementation manners, the determining whether the background image of the object to be removed is a crowd background image comprises: if a number of portrait instances in the second mask image is less than a fifth threshold value, it is determined that the background image of the object to be removed is not a crowd background image; if the number of portrait instances in the second mask image is greater than or equal to the fifth threshold value, a first portrait mask region in the second mask image is determined according to the first mask image and the second mask image; if the first portrait mask region intersects with at least one other portrait mask region in the second mask image or the distance is less than a sixth threshold value, it is determined that the background image of the object to be removed is a crowd background image; if the distance between the first portrait mask region and at least one portrait mask region in the second mask image is greater than or equal to the sixth threshold value, it is determined that the background image of the object to be removed is not a crowd background image.

[0059] For example, the fifth threshold value can be 3, and the sixth threshold value can be a product of the width of the minimum bounding rectangle of the determined one portrait mask region and a certain set value, for example, the set value can be 0.5.

[0060] For example, when the first portrait image and the second portrait image intersect, or the distance between the two portrait images is less than a certain distance value, the second portrait image can be eliminated when the first portrait image is eliminated, and a distorted portrait is generated in the elimination area. Through the scheme of the present application, the positional relationship of the object to be eliminated and other portraits can be determined, and it is determined whether it is a crowd background according to the positional relationship. Then, in the case of determining that it is a crowd background, the first image and the second mask image are used, and an image elimination model with powerful image processing performance is used, so that the purpose of avoiding the generation of a distorted portrait can be achieved.

[0061] Through the embodiments of the present application, the image elimination and repair in a multi-person scene is improved, the problem of eliminating a certain portrait and then generating a distorted portrait in a multi-person scene is solved, and the image elimination and repair experience is improved.

[0062] In some possible implementation manners, the method further includes: determining the first area ratio according to a ratio of a number of pixels of the mask region in the first mask image to a number of pixels of the first mask image.

[0063] In some possible implementation manners, the highlighting the selected region in the first image in response to the second operation of the user on the first image includes: in response to a circle selection operation of the user on the first image, determining a closed region formed by a movement track of the circle selection operation as the selected region; or, in response to a smearing operation of the user on the first image, determining a smearing region as the selected region. That is, the second operation is a circle selection operation or a smearing operation.

[0064] Through the scheme of the present application, in actual use, the user can directly select a region on the photo, that is, the portrait or object image in the region can be eliminated by one key, and the background image in the eliminated region is restored, thereby improving the elimination and repair effect.

[0065] In a second aspect, the present application provides a processing device for image elimination and repair. The device includes units for executing the method in the first aspect. The device can correspond to the method described in the first aspect, and the related description of the units in the device is referred to the description of the first aspect. For brevity, the description is not repeated here.

[0066] The method described in the first aspect can be implemented by hardware, or can be implemented by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, processing modules or units, display modules or units, etc.

[0067] In a third aspect, the present application provides an electronic device, which comprises a processor, and a computer program or instructions stored in the memory and readable by the processor, wherein the processor is configured to execute the computer program or instructions so as to perform the method in the first aspect.

[0068] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program (also referred to as instructions or codes) for implementing the method in the first aspect. For example, when the computer program is executed by a computer, the computer can perform the method in the first aspect.

[0069] In a fifth aspect, the present application provides a chip, which comprises a processor. The processor is configured to read and execute a computer program stored in a memory so as to perform the method in the first aspect and any possible implementation manner thereof. Optionally, the chip further comprises the memory, which is connected to the processor by a circuit or a wire.

[0070] In a sixth aspect, the present application provides a chip system, which comprises a processor. The processor is configured to read and execute a computer program stored in a memory so as to perform the method in the first aspect and any possible implementation manner thereof. Optionally, the chip system further comprises the memory, which is connected to the processor by a circuit or a wire.

[0071] In a seventh aspect, the present application provides a computer program product, which comprises a computer program (also referred to as instructions or codes). When the computer program is executed by an electronic device, the electronic device can implement the method in the first aspect.

[0072] It can be understood that the beneficial effects of the second aspect to the seventh aspect described above can be referred to the related description of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0073] Figure 1 A structural schematic diagram of an electronic device provided by an embodiment of the present application;

[0074] Figure 2 A software architecture schematic diagram of an electronic device provided by an embodiment of the present application;

[0075] Figure 3 A user interaction interface scene schematic diagram of the image elimination and repair processing method provided by an embodiment of the present application;

[0076] Figure 4A to Figure 4C A user interaction interface scene schematic diagram of the image elimination and repair processing method provided by an embodiment of the present application;

[0077] Figure 5 A flow schematic diagram of the image elimination and repair processing method provided by an embodiment of the present application;

[0078] Figure 6 An application scenario diagram of the image elimination and repair processing method provided by the embodiment of the present application is shown in FIG. 1;

[0079] Figure 7 An image cropping processing diagram in the image elimination and repair processing method provided by the embodiment of the present application is shown in FIG. 2;

[0080] Figure 8 A cropping, scaling and mask dilation processing diagram in the image elimination and repair processing method provided by the embodiment of the present application is shown in FIG. 3;

[0081] Figure 9 Another flow diagram of the image elimination and repair processing method provided by the embodiment of the present application is shown in FIG. 4;

[0082] Figure 10 An application scenario diagram of the image elimination and repair processing method provided by the embodiment of the present application is shown in FIG. 5 Figure 1 ;

[0083] Figure 11 A flow diagram of determining whether it is a complex background in the image elimination and repair processing method provided by the embodiment of the present application is shown in FIG. 6;

[0084] Figure 12 An application scenario diagram of the image elimination and repair processing method provided by the embodiment of the present application is shown in FIG. 7 Figure 2 ;

[0085] Figure 13 An application scenario diagram of the image elimination and repair processing method provided by the embodiment of the present application is shown in FIG. 8 Figure 3 ;

[0086] Figure 14A An application scenario diagram of the image elimination and repair processing method provided by the embodiment of the present application is shown in FIG. 9;

[0087] Figure 14B An interface diagram of the application scenario of the image elimination and repair processing method provided by the embodiment of the present application is shown in FIG. 10;

[0088] Figure 14C A flow diagram of determining whether it is a crowd background elimination scenario in the image elimination and repair processing method provided by the embodiment of the present application is shown in FIG. 11;

[0089] Figure 15 Another flow diagram of the image elimination and repair processing method provided by the embodiment of the present application is shown in FIG. 12;

[0090] Figure 16 An image cropping processing diagram in the image elimination and repair processing method provided by the embodiment of the present application is shown in FIG. 13;

[0091] Figure 17 A flowchart of a process of determining whether a circled region or a smearing region meets an erasing condition in the image erasing and repairing processing method provided in an embodiment of the present application;

[0092] Figure 18 A scenario diagram of a circled region or a smearing region meeting an erasing condition in the image erasing and repairing processing method provided in an embodiment of the present application;

[0093] Figure 19 A scenario diagram of a circled region or a smearing region not meeting an erasing condition in the image erasing and repairing processing method provided in an embodiment of the present application;

[0094] Figure 20 A scenario diagram of a circled region or a smearing region meeting an erasing condition in the image erasing and repairing processing method provided in an embodiment of the present application;

[0095] Figure 21 A flowchart of a process of determining whether a circled region or a smearing region meets an erasing condition in the image erasing and repairing processing method provided in an embodiment of the present application;

[0096] Figure 22A And Figure 22B A user interaction interface diagram of the image erasing and repairing processing method provided in an embodiment of the present application;

[0097] Figure 23 A structure diagram of an image erasing and repairing processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0098] In the embodiments of the present application, the following terms "first", "second", and the like are only used for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the embodiments, the meaning of "a plurality of" is two or more, unless otherwise specified.

[0099] In order to facilitate the understanding of the embodiments of the present application, first, the related concepts involved in the embodiments of the present application are briefly described.

[0100] 1. GAN-based image inpainting network model (GAN-inpainting)

[0101] The image inpainting network model based on GAN can be referred to as a GAN image inpainting model. The GAN image inpainting model is a technology of image inpainting using a generative adversarial network (GAN) method, which can repair and complete a specified region (mask region) in an image. The GAN image inpainting model can be a deep learning network model trained in an iterative adversarial training manner using a generator network and a discriminator network. The training samples are a large number of pictures (million-level pictures) and randomly generated inpainting regions (mask regions), and the training target is to make the generated picture after image inpainting as close to the original picture as possible.

[0102] 2. SD-based image inpainting network model (SD-inpainting)

[0103] The SD-based image inpainting network model can be referred to as an SD image inpainting model. The SD image inpainting model is a method based on a stable diffusion model. The method has a high model parameter quantity (about 1G) and a large calculation amount. Since the method model is trained by large-scale image-text data (50 billion image-text pairs), the SD image inpainting model has stronger image-text matching and background generation capabilities, and thus can handle large-area and complex background inpainting problems. The SD image inpainting model can include an encoder network, a denoising Unet network, and a decoder network.

[0104] Currently, when a user uses a mobile phone to take a photo outdoors, people or garbage cans and other objects may appear in the taken photo. In this case, the user usually wants to eliminate the people image or garbage can image appearing in the photo. Therefore, there is an urgent need for an image elimination and repair method that is convenient to operate and meets user needs.

[0105] To solve the above problems, the embodiments of the present application provide an image elimination and repair processing method and an electronic device. In the embodiments of the present application, in response to a user operation on a first image, the image of a person or object in a user-selected region can be eliminated and the background can be repaired. Through the solution of the present application, in actual use, the user can directly select a region on the photo, and the image of the person or object in the region can be eliminated by one key, and the background image in the eliminated region is restored. Furthermore, the present application selects a suitable image processing strategy for image elimination and background repair according to the area ratio of the object to be eliminated in the first image, so that the image processing is more precise, and the elimination and repair effect is improved.

[0106] For example, for a region with a small area ratio, the elimination and repair requirements are relatively low, and a GAN-based image elimination model with a small amount of calculation can be used, and the image processing effect can meet the basic requirement of small color difference; for a region with a large area, the elimination and repair requirements are high, and therefore a SD-based image elimination model with better performance is used to obtain better elimination and repair effect.

[0107] Through the scheme of the present application, the portrait or object in the region selected by the user can be eliminated, and various image elimination requirements can be met in terms of calculation amount, image processing performance, image processing effect, and the like.

[0108] The processing method of image elimination and repair and the electronic device in the embodiments of the present application will be described below with reference to the accompanying drawings.

[0109] Figure 1 A hardware system suitable for the electronic device of the present application is shown.

[0110] The electronic device 100 can be a mobile phone, a smart screen, a tablet computer, a wearable electronic device, a vehicle-mounted electronic device, an augmented reality (AR) device, a virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a projector, and the like, and the specific type of the electronic device 100 is not limited in the embodiments of the present application.

[0111] The electronic device 100 can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 can include a pressure sensor 180A, a gyro sensor 180B, a barometric sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0112] It should be noted that, Figure 1 The structures shown do not constitute specific limitations on the electronic device 100. In other embodiments of the present application, the electronic device 100 can include more or fewer components than those shown, or the electronic device 100 can include a combination of some of the components shown, or the electronic device 100 can include sub-components of some of the components shown. Figure 1 The structures shown do not constitute specific limitations on the electronic device 100. In other embodiments of the present application, the electronic device 100 can include more or fewer components than those shown, or the electronic device 100 can include a combination of some of the components shown, or the electronic device 100 can include sub-components of some of the components shown. Figure 1 The structures shown do not constitute specific limitations on the electronic device 100. In other embodiments of the present application, the electronic device 100 can include more or fewer components than those shown, or the electronic device 100 can include a combination of some of the components shown, or the electronic device 100 can include sub-components of some of the components shown. Figure 1 The structures shown do not constitute specific limitations on the electronic device 100. In other embodiments of the present application, the electronic device 100 can include more or fewer components than those shown, or the electronic device 100 can include a combination of some of the components shown, or the electronic device 100 can include sub-components of some of the components shown. Figure 1 The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0113] The processor 110 can include one or more processing units. For example, the processor 110 can include at least one of the following processing units: an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, a neural-network processing unit (NPU). Different processing units can be independent devices or integrated devices. The controller can generate operation control signals according to instruction operation codes and timing signals, and complete the control of fetching and executing instructions.

[0114] The processor 110 can also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can hold instructions or data that the processor 110 has just used or is using in a loop. If the processor 110 needs to use the instructions or data again, it can be called directly from the memory. This avoids repeated access and reduces the waiting time of the processor 110, thus improving the efficiency of the system.

[0115] Exemplarily, the processor 110 can be configured to execute the image elimination and repair processing method of the embodiments of the present application: in response to a first operation of a user, enabling an image elimination function; in response to a second operation of the user on a first image, highlighting a selected region in the first image; identifying an object to be eliminated in the selected region; determining a target image processing strategy according to a first area ratio of the object to be eliminated in the first image; eliminating and repairing the object to be eliminated according to the target image processing strategy to obtain a second image; the selected region of the second image does not include the object to be eliminated, and the region after the object to be eliminated is eliminated is repaired as a background image of the object to be eliminated; and displaying the second image.

[0116] Figure 1 The connection relationship between the modules shown is only illustrative and does not constitute a limitation on the connection relationship between the modules of the electronic device 100. Alternatively, the modules of the electronic device 100 can also use a combination of the above-mentioned various connection modes.

[0117] The wireless communication function of the electronic device 100 can be realized by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, etc.

[0118] The electronic device 100 can realize the display function through the GPU, the display screen 194, and the application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs that execute program instructions to generate or change display information.

[0119] The display screen 194 can be used to display images or videos. Exemplarily, in the embodiments of the present application, the display screen 194 can be used to display the first image (the image to be eliminated and repaired) obtained by shooting, and to display the second image (the image after elimination and repair).

[0120] The electronic device 100 can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, and the application processor, etc.

[0121] ISP is used to process the data feedback by the camera 193. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the camera, the optical signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and conversion into a visible image. The ISP can optimize the noise, brightness and color of the image through algorithm, and the ISP can also optimize the exposure and color temperature of the shooting scene and other parameters. In some embodiments, the ISP can be arranged in the camera 193.

[0122] The camera 193 (also referred to as a lens) is used to capture still images or videos. The camera can be triggered to start by application instructions to realize the function of taking photos, such as capturing images of any scene. The camera can include an imaging lens, a filter, an image sensor and other components. The light emitted or reflected by an object enters the imaging lens, passes through the filter, and finally converges on the image sensor. The imaging lens is mainly used to converge the light emitted or reflected by all objects in the shooting angle (which can also be referred to as the scene to be shot, the target scene, or the scene image that the user expects to shoot) into an image; the filter is mainly used to filter out the excess light waves (such as light waves other than visible light, such as infrared) in the light; the image sensor can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The image sensor is mainly used to convert the received optical signal into an electrical signal, and then transmit the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into a standard RGB, YUV, etc. format image signal.

[0123] In some embodiments, the electronic device 100 can include 1 or N cameras 193, N being a positive integer greater than 1.

[0124] The camera 193 can be located on the front of the electronic device 100, or on the back of the electronic device 100. The specific number and arrangement of the cameras can be set according to requirements, and the present application does not make any limitation.

[0125] Touch sensor 180K, also referred to as a touch device. Touch sensor 180K can be disposed on display screen 194, and touch sensor 180K and display screen 194 together form a touch screen, also referred to as a touch panel. Touch sensor 180K is configured to detect a touch operation acting on or near the touch sensor 180K. Touch sensor 180K can transmit the detected touch operation to the application processor to determine the touch event type. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K can also be disposed on the surface of electronic device 100 and disposed at a location different from display screen 194. Illustratively, in embodiments of the present application, touch sensor 180K can be configured to detect a user operation of triggering image removal and repair.

[0126] The hardware system of electronic device 100 is described in detail above, and the software system of electronic device 100 is described below.

[0127] Figure 2 FIG. 1 is a schematic diagram of the software system of electronic device 100 according to an embodiment of the present application. The layered architecture divides the system into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom, applications 201, application framework 202, hardware abstraction layer (HAL) 203, kernel 204, and hardware layer 205.

[0128] Applications layer 201 can include a series of application packages. In embodiments of the present application, the application packages can include camera, gallery, and other applications.

[0129] Application framework layer 202 provides application programming interfaces (APIs) and programming frameworks for the applications of the applications layer. Application framework layer 202 includes a number of pre-defined functions. In embodiments of the present application, the application framework layer can include a camera access interface, and the image processing interface can include an image removal and repair service. The image processing interface is configured to provide application programming interfaces and programming frameworks for the gallery application.

[0130] Hardware abstraction layer 203 is an interface layer between application framework layer 202 and kernel layer 204, and provides a virtual hardware platform for the operating system. In embodiments of the present application, hardware abstraction layer 203 can include a camera hardware abstraction layer and an image processing algorithm library.

[0131] The camera hardware abstraction layer can provide virtual hardware (referred to as camera HAL for short) of a front camera, a rear camera and other camera devices of the camera. The image processing algorithm library can include image elimination and repair algorithms, in other words, the image processing algorithm library contains running codes and data of the processing method for implementing the image elimination and repair provided in the embodiments of the present application.

[0132] The kernel layer 204 is a layer between hardware and software. The kernel layer includes drivers of various hardware. The kernel layer 204 can include a camera device driver, a digital signal processor driver and an image processor driver and the like. The camera device driver is used to drive the camera sensor to collect images and drive the image signal processor to pre-process the images. The digital signal processor driver is used to drive the digital signal processor to process the images. The image processor driver is used to drive the image processor to process the images.

[0133] The hardware layer 205 includes a camera sensor, an image signal processor, a digital signal processor and an image processor and the like.

[0134] It should be noted that, although the embodiments of the present application are described by taking the Android system as an example, the basic principles are also applicable to electronic devices based on iOS or Windows operating systems.

[0135] The execution subject of the processing method for image elimination and repair provided in the embodiments of the present application can be the electronic device (such as a mobile phone) described above, or a functional module and / or functional entity capable of implementing the processing method for image elimination and repair in the electronic device, and the present application can be implemented by hardware and / or software, and the specific implementation can be determined according to actual use requirements, which is not limited in the embodiments of the present application. The processing method for image elimination and repair provided in the embodiments of the present application will be described exemplarily below by taking the electronic device as an example in combination with the drawings.

[0136] In order to better understand the embodiments of the present application, the "image elimination and repair" function provided in the embodiments of the present application will be briefly described below in combination with a mobile phone interface schematic diagram:

[0137] In some embodiments, the electronic device adds an "AI elimination" option in an editing interface of a gallery application (or a photo album application), and the "AI elimination" option is an entry option for displaying the image elimination and repair function. Exemplarily, the image elimination and repair function can include intelligent circle selection and manual smearing functions. It should be noted that the various functions and function names contained in the image elimination and repair function are exemplarily described, and the embodiments of the present application are not limited thereto.

[0138] Intelligent circle selection refers to that the electronic device circles out a target according to a sliding track in response to a sliding operation of a user on a picture.

[0139] Manual smearing refers to the electronic device responding to the user's smearing action on an image and selecting the target based on the smeared area.

[0140] Figure 3 This diagram illustrates the user interface of the "AI Elimination" option in the image removal and restoration method provided in this application embodiment. Figure 3 As shown in (a), the electronic device displays an icon for the Gallery app on desktop 10. When the user clicks the Gallery app icon, as shown in (a), the user sees the following: Figure 3 As shown in (b), the electronic device updates from displaying the desktop 10 to displaying the all-photos interface 11. When the user clicks on a photo in the all-photos interface 11, as shown in (b), the electronic device updates from displaying the desktop 10 to displaying the all-photos interface 11. Figure 3 As shown in (c), the electronic device updates from displaying all photos interface 11 to displaying a single photo interface 12. The single photo interface includes editing options 13. Figure 3 As shown in (d), when editing option 13 is selected, the electronic device updates from displaying a single photo interface 12 to displaying an editing interface 14. Editing interface 14 includes the photo selected by the user and the "AI Removal" option 15. When the "AI Removal" option 15 is selected in editing interface 14, the electronic device displays the functions corresponding to the "AI Removal" option 15, such as intelligent selection and manual smoothing.

[0141] It should be noted that electronic devices can identify the selected area and the object to be removed through functions such as intelligent selection or manual smearing.

[0142] In some embodiments, the object to be eliminated can be a human body, such as a passerby in a photograph.

[0143] In other embodiments, the object to be eliminated can be an item, such as a trash can or clutter in a photograph.

[0144] According to the solution of this application, electronic devices can respond to the user's selection operation on the image, remove the object to be removed in the user-specified area (the specified area) of the image, and repair the removed area to generate a more aesthetically pleasing image.

[0145] Figure 4A This diagram illustrates a user interface used in the image removal and restoration method provided in this application to complete image removal and restoration. For example... Figure 4A As shown in (a), the electronic device displays the user-selected photo (i.e., the original image) 16 in the editing interface 14 and displays the functions corresponding to the "AI Removal" option 15 in the function option area, such as intelligent selection and manual smearing. Figure 4AAs shown in (a) and (b), when the user selects the "Smart Selection" function option 17, the electronic device can display a trajectory cursor 18 for smart selection in the original image 16. The user can drag the trajectory cursor 18 to select the objects to be eliminated in the original image 16. The user drags the trajectory cursor 18 from the starting point 19 to the ending point 20 and releases it; the selection trajectory is as follows. Figure 4A As shown by the dashed lines in (b) and (c), the selected trajectory can be closed manually by the user or automatically extended and closed by the electronic device. Figure 4A As shown in (d), the selection trajectory is closed, and the area enclosed by this trajectory is the area within the dashed box 21. The object to be eliminated is a human body, and the background is flowers and plants. Figure 4A As shown in (e), the image elimination and repair processing method provided in this application embodiment is used to eliminate and repair the object to be eliminated. The human image in the dashed frame 21 in the original image 16 is eliminated, while the background such as flowers and plants in the dashed frame 21 is still retained. The image after elimination and repair is shown in 22.

[0146] In some embodiments, after the user drags the trajectory cursor 18 from the starting point 19 to the ending point 20 and releases it, the object to be eliminated can be displayed using a dynamic flashing display method.

[0147] Figure 4B This illustration shows another user interface diagram illustrating the image removal and restoration process provided in this application embodiment, where image removal and restoration are performed. For example... Figure 4B As shown in (a) to (d), in response to the user's selection operation 23 on the original image, the electronic device recognizes that the object to be eliminated is a human image and meets the elimination conditions. The object to be eliminated 24 is highlighted. The electronic device performs image elimination and repair processing on the object to be eliminated 24 in the background, and then updates the original image. The updated image 25 does not include the object to be eliminated 24, and the eliminated area is repaired as the background image.

[0148] Figure 4C This illustration shows another user interface diagram illustrating the image removal and restoration process provided in this application embodiment, where image removal and restoration are performed. For example... Figure 4C As shown in (a) to (f), in response to the user's operation on the manual smear option 26, the electronic device starts the manual smear function; in response to the user's operation on the smear tool 27, the electronic device enables the smear tool 27; in response to the user's smear operation on the object to be eliminated 28 (trash can) in the original image, the electronic device recognizes that the object to be eliminated is not a human image and meets the elimination conditions. The electronic device performs image elimination and repair processing on the smeared area of ​​the object to be eliminated 28 in the background, and then updates and displays the original image. The updated image 29 does not include the object to be eliminated 28, and the eliminated area is repaired to the background image.

[0149] The image elimination and repair processing method provided by the embodiments of the present application is described in detail below with reference to specific embodiments and the accompanying drawings.

[0150] Figure 5 is a schematic flowchart of an image elimination and repair processing method provided by the embodiments of the present application. The method can be executed by an electronic device as shown in Figure 1 The method includes S301 to S304.

[0151] S301, in response to a selection operation (circle selection or smearing operation) of a user, determining an object to be eliminated in a first image.

[0152] Through the scheme of the present application, the electronic device can eliminate a portrait or a figure in a picture (also referred to as a first image, an original image or an image to be repaired) selected by the user through circle selection or smearing operation and repair the elimination area in response to the selection operation of the user in the picture, to generate a more beautiful picture.

[0153] The area selected by the user through circle selection or smearing is referred to as a selected area, and the selected area can include a human body or an article selected by the user in the picture through circle selection or smearing.

[0154] For example, the selection operation can be an operation of drawing a circle to select an area in the picture by sliding a finger or dragging a cursor, which can be referred to as intelligent circle selection. Accordingly, the area of intelligent circle selection can be determined as the selected area. The object in the selected area is the object to be eliminated.

[0155] For another example, the selection operation can be a click operation of the user in a certain area of the picture, which can trigger the electronic device to automatically circle the object to be eliminated at the click position. Accordingly, the area of automatic circle selection can be determined as the selected area.

[0156] For example, the selection operation can be a smearing operation of the user in a certain area of the picture. Accordingly, the area covered by smearing can be determined as the selected area.

[0157] For ease of illustration, the embodiments of the present application are exemplarily described taking the selection operation as an intelligent circle selection operation.

[0158] The object in the selected area can be a human body or an article.

[0159] In some embodiments, in response to the smart circling operation of the user, the circling track can be marked in the first image, and the selected region can be determined according to the circling track. For example, the region enclosed by the circling track can be determined as the selected region. Alternatively, the circling track can be a line track, or a closed rectangle, an ellipse, or an irregular figure. For ease of illustration, the embodiments of the present application are exemplarily described by taking the circling track as a line track.

[0160] In some embodiments, when the circling track is closed, the region enclosed by the circling track is the selected region.

[0161] In some other embodiments, when the circling track is not closed, if the distance between the start point and the end point of the circling track satisfies the preset distance condition, the electronic device can automatically connect the start point and the end point, so that the circling track forms a closed region, and then the closed region is determined as the selected region.

[0162] In some other embodiments, when the circling track is not closed, if the distance between the start point and the end point of the circling track does not satisfy the preset distance condition (for example, the distance between the start point and the end point is too large), the electronic device determines that the circling track does not satisfy the condition, and displays a prompt information. For example, the prompt information can be: the circling operation is invalid, please circle again.

[0163] S302, acquire a first mask image based on the first image, the first mask image including a first mask region corresponding to the object to be removed.

[0164] The first mask image can be referred to as a mask image or a binary image. For example, the first mask image includes a mask region in which the pixel points are marked as 1, and other regions in which the pixel points are marked as 0.

[0165] Specifically, the pixel points of the first mask region are marked as 1, and the pixel points are displayed as white. The pixel points of the other regions in the first mask image except the first mask region are marked as 0, and the pixel points are displayed as black.

[0166] In some embodiments, the electronic device can determine the selected region according to the circling track, and determine the object to be removed according to the selected region, and then the first mask image. The shape of the mask region in the first mask image is equivalent to the shape of the object to be removed.

[0167] In some other embodiments, the electronic device can input the image marked with the circling track into a certain mask generation model to obtain the first mask image. The mask region in the first mask image corresponds to the selected region enclosed by the circling track. The shape of the first mask region is similar to the shape of the region enclosed by the circling track.

[0168] S303, calculate a first area proportion (denoted as P) of the first mask region in the first mask image.

[0169] The area proportion of the first mask region in the first mask image is equal to the area proportion of the object to be eliminated in the first image.

[0170] In some embodiments, the number of pixels of the first mask image and the number of pixels of the first mask region are obtained, and the ratio of the number of pixels of the first mask region to the number of pixels of the first mask image is calculated to obtain the first area proportion P.

[0171] The number of pixels of the first mask image is the sum of the number of all pixels 1 and the number of all pixels 0.

[0172] The number of pixels of the first mask region is the number of all pixels 1 in the first mask image.

[0173] It can be understood that the area proportion of the first mask region in the first mask image is equivalent to the area proportion of the object to be eliminated in the first image.

[0174] 0

[0175] It should be noted that when P is small (e.g., P < 2%), that is, the proportion of the object to be eliminated is small, the elimination and repair are easy; when P is large (e.g., 2% ≤ P < 25%), that is, the proportion of the object to be eliminated is large, the elimination and repair are complex and require high requirements.

[0176] S304, using an image processing strategy corresponding to the first area proportion P to eliminate and repair the object to be eliminated in the first image.

[0177] For example, when P < X1, a first image processing strategy is used to eliminate and repair the object to be eliminated.

[0178] When X1 ≤ P < X2, a second image processing strategy is used to eliminate and repair the object to be eliminated.

[0179] When X2 ≤ P < X3, a third image processing strategy is used to eliminate and repair the object to be eliminated.

[0180] When P ≥ X3, the object to be eliminated is not eliminated and repaired.

[0181] 0 ≤ X1 < X2 < X3.

[0182] It should be noted that in actual implementation, the number of thresholds can be adjusted (increased or decreased) according to actual use requirements, for example, four thresholds X1, X2, X3 and X4 can be set, 0≤X1X2X3X4, so that the image processing is more refined, and the elimination and repair effect can be greatly improved. For the sake of illustration, X1, X2 and X3 are exemplarily illustrated in the following embodiments.

[0183] It should be noted that in actual implementation, the values of X1, X2 and X3 can be set according to actual use requirements, which are not limited in the present application. For the sake of illustration, X1 is 2%, X2 is 10%, and X3 is 25% in the following embodiments.

[0184] According to the scheme of the present application, for different area ratios of the to-be-eliminated object, the image processing strategy corresponding to the area ratio can be selected to eliminate and repair the to-be-eliminated object, and different image processing effects can be obtained.

[0185] It should be noted that the present application does not limit the image elimination model used by the three image processing strategies. The image elimination models used by the above three image processing strategies can be the same or different.

[0186] In some embodiments, the first image processing strategy, the second image processing strategy and the third image processing strategy may, in different cases, use the same or different image elimination models; the image processing process may, in different cases, be the same or different, for example, some perform image cropping processing, and some do not perform image cropping processing; the image processing calculation amount increases in turn, and the image processing effect is enhanced in turn.

[0187] In some embodiments, for a to-be-eliminated object with a very small area ratio, the first image processing strategy is used, image cropping processing is performed, and elimination and repair processing is performed according to the first mask image and the first image and the GAN image elimination model.

[0188] In other embodiments, for a to-be-eliminated object with a slightly larger area ratio, the second image processing strategy is used, image cropping processing is performed, and it is judged whether the background of the to-be-eliminated object is a complex background. If it is a simple background (for example, a sky or a pure color background), elimination and repair processing is performed according to the first mask image and the first image and the GAN image elimination model; if it is a complex background (for example, a crowd background), elimination and repair processing is performed according to the first mask image and the first image and the SD image elimination model.

[0189] In some embodiments, for objects to be eliminated that occupy a large area, a third image processing strategy is adopted. Instead of image cropping, it determines whether the elimination scenario has a crowd background. If it does not, elimination and repair processing is performed based on the first mask image and an image processing network model based on SD or GAN. If it does have a crowd background, elimination and repair processing is performed based on the first mask image, the second mask image, and the SD image elimination model. The first mask image includes the mask region corresponding to the object to be eliminated in the first image, and the second mask image includes the mask region corresponding to each portrait in the first image.

[0190] It should be noted that, compared to GAN image removal models, SD image removal models have higher computational requirements but superior image processing performance. SD image removal models can handle large-area and complex background removal problems.

[0191] Figure 6 The diagram illustrates three scenarios where different image processing strategies are used to eliminate and repair objects with different area proportions.

[0192] First scenario:

[0193] like Figure 6 As shown in (a), the first image includes the object to be eliminated, 31, indicated by the dashed line. The object to be eliminated is a male, and it corresponds to the first mask region 32 in the first mask image. The area ratio P of the first mask region in the first mask image is calculated, where P < 2%. Since the area ratio of the object to be eliminated is very small, the first image processing strategy can be adopted: based on the first mask image and the first image, the GAN image elimination model is used to eliminate and repair the object 31. This approach has a small computational load and the image processing effect meets basic requirements such as small color difference.

[0194] exist Figure 6 In the image shown in (a) after removal and restoration, the image to be removed (the male image) selected by the circle or smear operation in the first image is removed. It should be noted that for image removal scenarios where the object to be removed accounts for a small proportion and removal and restoration are relatively easy to achieve, the GAN image removal model can basically meet the removal and restoration requirements and save computational resources.

[0195] Second scenario:

[0196] like Figure 6As shown in (b), the first image includes the object to be eliminated, 33, indicated by the dashed line. The object to be eliminated, 33, is a trash can and corresponds to the first mask region 34 in the first mask image. The area percentage P is calculated, where 2% ≤ P < 10%. Since the area percentage of the object to be eliminated is slightly larger, the first image processing strategy can be adopted.

[0197] Case 1: If the background of the object to be eliminated 33 is a simple background (such as sky or solid color background), then the elimination and repair process is performed according to the first mask image and the first image and GAN image elimination model. The amount of computation is small and the basic requirements such as small color difference are met.

[0198] Case 2: If the background of the object to be eliminated 33 is a complex background (e.g., a crowd background), then the elimination and repair process is performed according to the first mask image and the first image and SD image elimination model. The elimination effect is good and meets the basic requirements such as small color difference.

[0199] exist Figure 6 In the image shown in (b) after removal and restoration, the image to be removed (trash can image) selected by circling or smearing operations in the first image is eliminated. By adopting the first image processing strategy, the removal and restoration requirements can be met, while saving computational resources.

[0200] Third scenario:

[0201] like Figure 6 As shown in (c), the first image includes the object to be removed, 35, indicated by the dashed line. The object to be removed is a woman, and it corresponds to the first mask region 36 in the first mask image. The area percentage P is calculated; 10% ≤ P < 25%. Since the area percentage of the object to be removed is large, a second image processing strategy with high computational complexity and good image processing effect can be used to remove and repair the object 35.

[0202] Case 1: If the background of the object to be eliminated 35 is a non-human scene, then the elimination and repair process is performed according to the first mask image and the first image and SD image elimination model. The elimination effect is good and meets the basic requirements such as small color difference.

[0203] Scenario 2: If the background of the object to be removed 35 is a crowd scene, then the removal and repair processing is performed based on the first image, the first mask image, the second mask image, and the SD image removal model. The removal effect is good and meets basic requirements such as small color difference. The second mask image contains the masked areas of all human figures in the first image.

[0204] exist Figure 6The image to be eliminated in the first image selected by the circle selection or the smearing operation (the lady image) is eliminated in the eliminated and repaired image shown in (c). It should be noted that for an image elimination scene in which the object to be eliminated accounts for a large proportion and the elimination and repair requirements are high, elimination and repair processing is performed according to the first mask image, the first image and the second mask image, and the SD image elimination model, which can well meet the elimination and repair requirements and obtain good elimination and repair effects.

[0205] It should be noted that the image elimination model used in the above three scenarios is not limited by the embodiments of the present application. For example, Figure 6 The image elimination model based on GAN can be replaced by the image elimination model based on SD in (a) in the above three scenarios. Similarly, the image elimination model based on SD can be replaced by the image elimination model based on GAN in (b) and (c) in the above three scenarios. Which image elimination model is used can be determined according to actual use requirements. Figure 6

[0206] Through the scheme of the present application, for different area proportions of the object to be eliminated selected by the user in the original image, an image processing strategy corresponding to the area proportion can be selected to eliminate and repair the object to be eliminated, and different image processing effects can be obtained. Through the image elimination and repair processing method provided by the embodiments of the present application, the portrait or the object in the circle selection region or the smearing region can be eliminated, and various image elimination requirements can be met in terms of operation amount, image processing performance, image processing effect and the like.

[0207] The image elimination and repair processing method provided by the embodiments of the present application is not only suitable for image elimination scenes with simple backgrounds, but also suitable for image elimination scenes with complex backgrounds and image elimination scenes with crowd backgrounds.

[0208] Through the method provided by the present application, according to the area proportion of the circle selection region or the smearing region in the original image, a suitable image processing strategy is selected to eliminate and repair the object to be eliminated, the image processing is more fine, and the elimination and repair effect is improved.

[0209] Image cropping processing

[0210] In some embodiments, in the scene of eliminating and repairing the object to be eliminated selected by the circle selection or the smearing in the first image, in the case where the area proportion P meets certain conditions, the first image and the first mask image can be cropped first, and then the elimination and repair processing is performed based on the cropped first image and the first mask image, which can optimize the effect of the elimination and repair processing.

[0211] ​For example, when the area percentage P is less than X2 (e.g., 10%), the first image and the first mask image can be cropped separately. The cropped mask image contains the first mask region, and the cropped first image contains the object to be eliminated. Then, based on the cropped mask image and the cropped first image, the object to be eliminated selected by circling or smearing in the first image can be eliminated and repaired. The following describes the possible implementation methods for determining the cropping region and specific cropping provided by the embodiments of this application.

[0212] 1) Calculate the number of pixels in the first mask region (denoted as S).

[0213] 2) Multiply S by the coefficient corresponding to P to calculate the number of pixels in the cropped region (denoted as T).

[0214] In some embodiments, when P < 2%, the coefficient corresponding to P is N6, and the number of pixels in the cropped region T = S * N6 is calculated. Optionally, N6 can be 17. Multiplying S by 17 can be understood as cropping the image by 6%.

[0215] In some embodiments, when 2% ≤ P < 10%, the coefficient corresponding to P is N7, and the number of pixels in the cropped region is calculated as T = S * N7. Optionally, N7 can be 10. Multiplying S by 10 can be understood as cropping the image by 10%.

[0216] 3) Calculate the width and height (H1) of the cropped area (denoted as W1) based on the number of pixels T in the cropped area. For example, you can find the square root of T and determine the width W1 and height H1 of the cropped area. For instance, if T is 360000, then the width W1 and height H1 of the cropped area are both 600.

[0217] 4) such as Figure 7 As shown in (a), the width (denoted as W2) and height (denoted as H2) of the minimum bounding rectangle 17 of the first mask region are calculated.

[0218] 5) Adjust the width W1 and height H1 of the clipping region based on the width L2 and height H2 of the minimum bounding rectangle 17.

[0219] For example, if the width W1 of the clipping region is less than the width W2 of the minimum bounding rectangle 17, then W1 = W2 + N8.

[0220] For example, if the height H1 of the clipping region is less than the height H2 of the minimum bounding rectangle 17, then H1 = H2 + N8.

[0221] For example, if the width W1 or height H1 of the cropping region is less than N9, then the width W1 or height H1 of the cropping region is set to N9. Here, N9*N9 is the input image size defined by the inpainting network.

[0222] Optionally, N8 can take 6, and N9 can take 768.

[0223] 6) As shown in (a) of Figure 7 , the center of the minimum bounding rectangle 41 of the first mask region is taken as the center, and the adjusted width L1 and height H1 are determined to determine the clipping region 42.

[0224] wherein it is assumed that the size of the first image and the first mask image are both the first size. For example, the first size is 1600*1200.

[0225] As shown in (a) and (b) of Figure 7 , the first mask image of the second size is clipped from the first mask image, and the first image of the second size is clipped from the first image according to the above steps 1) to 6). For example, the second size is 1200*800.

[0226] Mask region inflation processing

[0227] In some embodiments, in the scenario of eliminating and repairing the to-be-eliminated object in the first image by circle selection or brush selection, the first mask region in the first mask image can be dilated first, and then the elimination and repair processing is performed based on the dilated mask image, so that the effect of the elimination and repair processing can be optimized.

[0228] In the embodiments of the present application, the first mask image can be dilated to increase the mask region, so as to increase the elimination and repair range, so that the to-be-eliminated object can be more comprehensively eliminated and repaired, and the elimination and repair effect can be improved.

[0229] In some embodiments, the mask dilation ratio N10 and the initial dilation kernel size N11 can be set, the dilation kernel size is increased by 1 each time, and the first mask region is dilated to obtain the dilated first mask image (denoted as dilatedMask) until the area (pixel number) of the dilated first mask image is greater than or equal to R, R=S*N10. S represents the area (pixel number) of the first mask region.

[0230] Optionally, the dilation ratio N10 can take 1.2, and the initial dilation kernel size N11 can take 3.

[0231] Figure 8 A schematic diagram of clipping, scaling and dilating the first mask image is shown. As shown in Figure 8 , the first mask image is of the first size, the clipped first mask image is of the second size, the dilated first mask image is of the second size, and the dilated first mask region is larger than the first mask region.

[0232] It can be understood that, in the case that N10 is greater than 1, the contour of the first mask region after expansion is greater than the contour of the first mask region before expansion. In this way, by increasing the mask region, the range of elimination and repair is increased, and the elimination and repair effect is improved.

[0233] Image elimination and repair processing

[0234] Through the scheme of the present application, appropriate image processing methods can be selected according to the area proportion of the object to be eliminated for image elimination and background repair, so that the image processing is more precise, and the elimination and repair effect is improved.

[0235] It should be noted that the electronic device can adopt various possible ways to implement the above-mentioned S304 using the image processing strategy corresponding to the first area proportion P to eliminate and repair the object to be eliminated in the first image (original image) selected by circling or smearing.

[0236] The possible implementation ways of selecting appropriate image processing methods for image elimination and background repair according to the area proportion of the object to be eliminated will be described in detail below.

[0237] Exemplarily, in combination with Figure 5 As shown in FIG. 6, the above-mentioned S304 can include the following S1-S25. Figure 9 S1, determine whether P is less than 2%.

[0238] In the case that it is determined that P is less than 2%, continue to execute S2-S8, that is, use the first image processing strategy to perform image elimination and repair processing.

[0239] In the case that it is determined that P is not less than 2%, continue to execute S9, that is, further determine whether P is less than 10%.

[0240]

[0241] First image processing strategy is adopted when P is less than 2%

[0242] S2, in the case that it is determined that P is less than 2%, calculate the to-be-clipped region with a mask proportion of 6%.

[0243] Exemplarily, calculating the to-be-clipped region with a mask proportion of 6% means multiplying the number of pixels of the first mask region by 17 to obtain the number of pixels T, and then taking the square root of the number of pixels T to obtain the width W1 and the height H1. Then, according to the width W2 and the height H2 of the minimum bounding rectangle of the first mask region, the width W1 and the height H1 are adjusted (increased) to obtain the width W3 and the height H3. The width W3 and the height H3 are taken as the width and height of the to-be-clipped region. The specific calculation process of the to-be-clipped region can be referred to the above detailed description about calculating the to-be-clipped region, which will not be described here again.

[0244] ​S3, crop and scale the first image and the first mask image to obtain a cropped and scaled first image and a cropped and scaled first mask image.

[0245] First, crop the first image and the first mask image according to the calculated to-be-cropped region. The cropped first image includes the to-be-removed object, and the cropped first mask image includes a first mask region. For details of the image cropping process, refer to Figure 7 .

[0246] Then, scale the cropped first image and the cropped first mask image to N9*N9 (for example, 768*768) to meet the requirements of the removal and repair network on the input picture size.

[0247] For details of the cropping and scaling process of the first mask image, refer to Figure 16 . It should be noted that the cropping and scaling process of the first image is similar to the cropping and scaling process of the first mask image.

[0248] The size of the first image and the first mask image can be a first size, and the size of the cropped region can be a second size. The second size is smaller than the first size.

[0249] S4, perform dilation processing on the mask region of the scaled first mask image.

[0250] In the embodiments of the present application, refer back to Figure 8 , the mask region of the first mask image is dilated to increase the mask region, thereby increasing the removal and repair range, making the to-be-removed object more comprehensively removed and repaired, and improving the removal and repair effect.

[0251] S5, input the cropped and scaled first image and the cropped, scaled and dilated first mask image into the GAN image removal model for model inference to obtain a seventh image.

[0252] The size of the seventh image is a size specified by the GAN image removal model. For example, the size of the seventh image is 768*768.

[0253] In some embodiments, the GAN image removal model can be a large mask repair LAMA model.

[0254] In some embodiments, the GAN image removal model can be an aggregated contextual transformation (AOT) model.

[0255] In some embodiments, the GAN image inpainting model can be a mask-interactive generative adversarial network (MI-GAN) model.

[0256] S6, after scaling the seventh image to the second size, the third image of the first size is obtained by filling the cropped region of the first image.

[0257] In the embodiments of the present application, the seventh image is scaled (e.g., enlarged) from the third size to the second size, i.e., size restoration.

[0258] In some embodiments, the second inpainting image can be scaled by using an image super-resolution method. The super-resolution processing refers to a process of restoring a high-resolution image from a low-resolution image.

[0259] It can be understood that by scaling the seventh image to the same size (second size) as the cropped region, the scaled seventh image can be seamlessly filled into the cropped region of the first image.

[0260] S7, after scaling the cropped, scaled and expanded first mask image to the second size, the expanded and filled first mask image is obtained by filling the cropped region of the first mask image.

[0261] The present application does not limit the execution order of S6 and S7, for example, S6 can be executed first, and then S7 can be executed; S7 can be executed first, and then S6 can be executed; or S6 and S7 can be executed simultaneously.

[0262] S8, the second image is obtained according to the first image, the third image and the expanded and filled first mask image.

[0263] In the embodiments of the present application, the second image is obtained by using the following image matrix calculation formula.

[0264] Pi=Po×(1–Pm)+Pe×Pm.

[0265] Wherein, Po represents the first image, Pe represents the third image, Pm represents the expanded and filled first mask image, and Pi represents the second image.

[0266] Wherein, (1-Pm) represents taking the inverse of the pixel value of the first mask image.

[0267] Figure 10 A flowchart for implementing image inpainting and repair by using a GAN image inpainting model when P is less than 2% is shown.

[0268] As Figure 10As shown, in response to a user's selection operation (e.g., circling or smearing) on ​​the first image (original image), the electronic device determines the object to be removed and calculates the area percentage P of the object to be removed in the first image, which is less than 2%.

[0269] A first mask image is obtained based on the first image. The mask region in the first mask image is the mask region of the object to be eliminated. The size of both the first image and the first mask image is the first size.

[0270] Then, based on the mask region in the first mask image, a cropped region is calculated with a first mask percentage (e.g., 6%). The cropped region includes and is larger than the mask region, and its size is a second size. The first image and the first mask image are cropped based on the cropped region to obtain a first image of the second size and a first mask image of the second size, respectively.

[0271] Then, the first image of the second size and the first mask image of the second size are scaled to obtain the first image of the third size and the first mask image of the third size.

[0272] Then, the mask region of the first mask image with the third size is dilated to obtain the first mask image with the third size dilated.

[0273] Then, the first image at the third size and the first masked image at the third size are input into the GAN image elimination model to obtain the seventh image at the third size. The seventh image at the third size is then scaled to obtain the seventh image at the second size.

[0274] Then, the seventh image of the second size is filled into the cropped area of ​​the first image to obtain the third image of the first size.

[0275] On the other hand, the first mask image, expanded to a third size, is filled into the cropped area of ​​the first mask image of the first size to obtain the first mask image expanded and filled to the first size. Then, the pixel values ​​of the first mask image expanded and filled are inverted to obtain the fifth image. Then, the fifth image is multiplied by the first image to obtain the sixth image.

[0276] The third image and the first masked image after dilation and filling are ANDed (i.e. multiplied) to obtain the fourth image.

[0277] Then, the fourth and sixth images are fused together to obtain the second image.

[0278] like Figure 10 As shown, in the second image, the image to be eliminated (portrait) selected by circling or smearing is eliminated, and the background image of the eliminated area has been repaired.

[0279] For the to-be-eliminated object with a relatively small area ratio, the image elimination and repair requirements are relatively low, and a small amount of calculation image elimination model can be used, and the image processing effect meets the basic demand of small color difference.

[0280] The above describes the image elimination and repair process when the area ratio P of the to-be-eliminated object is very small (for example, P is less than 2%), and the following describes the image elimination and repair process when the area ratio P of the to-be-eliminated object is slightly large (for example, 2%≤P<10% or P<25%).

[0281] S9, in the case of determining that P is not less than 2%, determining whether P is less than 10%.

[0282] In the case of determining that P is less than 10%, continue to execute S10.

[0283] In the case of determining that P is not less than 10%, continue to execute S13, that is, further determine whether P is less than 25%.

[0284] Second image processing strategy is adopted when 2%≤P<10%

[0285] S10, in the case of determining that P is less than 10%, calculating the to-be-clipped region with a second mask ratio (for example, 10%).

[0286] The process of calculating the to-be-clipped region with a mask ratio of 10% can refer to the process of calculating the to-be-clipped region with a mask ratio of 6% described above.

[0287] S11, clipping the first image and the first mask image to obtain a clipped first image and a clipped first mask image.

[0288] The first image and the first mask image are both of a first size, and are both of a second size after clipping.

[0289] The implementation process of S11 is similar to the implementation process of clipping the first image and the first mask image in S3 described above, which will not be described here.

[0290] S12, determining whether the background of the to-be-eliminated object is a complex background according to the clipped first image and the clipped first mask image.

[0291] Figure 11 An example of determining whether the background of the to-be-eliminated object is a complex background is shown. As shown in Figure 11 The first mask region is dilated respectively, and then the difference is obtained to obtain a peripheral annular region. According to the feature point density in the peripheral annular region, it can be determined whether the background of the to-be-eliminated object is a complex background.

[0292] For example, the first mask region is dilated by a dilation ratio N12 to obtain dialtedMask1. And the first mask region is dilated by N13 to obtain dialtedMask2. Optionally, N12 is 15 and N13 is 75.

[0293] Then, a peripheral annular region of the first mask region is determined: AroundMask = dialtedMask2 - dialtedMask1.

[0294] Then, a feature point number FeaPtNum and a pixel number PAM of the AroundMask annular region in the first image are calculated.

[0295] Then, a feature point density FeaPtDensity in the AroundMask annular region in the first image is calculated:

[0296] FeaPtDensity = FeaPtNum / PAM.

[0297] If FeaPtDensity > N14, it is determined that the background of the object to be eliminated is a complex background.

[0298] If FeaPtDensity ≤ N14, it is determined that the background of the object to be eliminated is a simple background.

[0299] Optionally, N14 is 1%.

[0300] Optionally, the feature point number FeaPtNum can be calculated by a FAST (features from accelerated segment test) feature point extraction method, an ORB (oriented FAST and rotated BRIEF) feature point extraction method, or a SURF (speeded up robust features) feature point extraction method.

[0301] Reference Figure 11 If it is determined that the background of the object to be eliminated is not a complex background, steps S4-S8 are continued to be executed, that is, a GAN image elimination model meeting actual use requirements is used to perform image elimination and repair processing. If it is determined that the image elimination scene is a complex background, a SD image elimination model with stronger image processing performance can be used.

[0302] Exemplarily, Figure 12 A flowchart of image elimination and repair for a non-complex background is shown in the case of 2% ≤ P < 10%.

[0303] As Figure 12As shown, in response to the user's selection operation on the first image (original image), the electronic device determines the circled region and calculates the area proportion P of the object to be eliminated in the first image, 2%≤P<10%.

[0304] The first mask image is obtained based on the first image, and the mask region in the first mask image is a mask region of the object to be eliminated. The sizes of the first image and the first mask image are both the first size.

[0305] Then, based on the mask region in the first mask image, a cropped region is calculated at a second mask proportion (for example, 10%). The cropped region includes the mask region and is larger than the mask region. The size of the cropped region is the second size. The first image and the first mask image are cropped based on the cropped region, and the first image of the second size and the first mask image of the second size are obtained.

[0306] Different from Figure 10 , in Figure 12 , after image cropping, whether the background of the object to be eliminated is a complex background is determined according to the cropped first image and the first mask image.

[0307] In a case where it is determined that the background of the object to be eliminated is a complex background, the cropped first image and the first mask image are scaled, and then the mask region of the scaled first mask image is dilated. Then, the cropped and scaled first image and the cropped, scaled, and dilated first mask image are input into the SD image elimination model to obtain a seventh image.

[0308] Or, in a case where it is determined that the background of the object to be eliminated is not a complex background, the cropped first image and the first mask image are scaled, and then the mask region of the scaled first mask image is dilated. Then, the cropped and scaled first image and the cropped, scaled, and dilated first mask image are input into the GAN image elimination model to obtain a seventh image.

[0309] Then, the seventh image is filled into the cropped region of the first image to obtain a third image of the first size.

[0310] On the other hand, the dilated first mask image of the third size is filled into the cropped region of the first mask image of the first size to obtain a dilated and filled first mask image of the first size. Then, the pixel value of the dilated and filled first mask image is inverted to obtain a fifth image. Then, the fifth image is multiplied by the first image to obtain a sixth image.

[0311] The third image and the dilated and filled first mask image are ANDed (i.e., multiplied) to obtain a fourth image.

[0312] Then the fourth image and the sixth image are image fused to obtain the second image.

[0313] As shown in Figure 12 The selected portrait in the second image is eliminated by the circle selection or the smearing operation, and the background image of the eliminated area is repaired.

[0314] Through the scheme of the present application, for the image elimination scene of a non-complex background, a GAN image elimination model meeting the actual use demand can be used. For the image elimination scene of a complex background, an SD image elimination model with stronger image processing performance can be used. Through the image elimination and repair processing method provided by the embodiments of the present application, the portrait or the article in the circle selection area can be eliminated, and various image elimination demands can be met from various dimensions such as operation amount, image processing performance, and image processing effect.

[0315] Optionally, if it is determined that the background of the object to be eliminated is a complex background, then S14 can be continued to determine whether it is a crowd background elimination scene. It should be noted that for a complex background, it can be further identified whether it is a crowd background. If it is not a crowd background, an image elimination model meeting the actual use demand can be used. If it is a crowd background, in order to avoid generating a distorted portrait after eliminating the image, therefore, an image elimination model with strong image processing performance (such as an SD image elimination model) can be used according to the first image and the second mask image, so as to avoid generating a distorted portrait. The specific implementation process will be described below.

[0316] It should also be noted that in the case of 2%≤P<10%, since image cropping processing is performed in the early processing of image elimination and repair, in the later processing of image elimination and repair, the filling processing of the cropped image as in S5 and S6 needs to be completed.

[0317] The above describes the image elimination and repair process in the case of a very small area ratio P (such as P less than 2%) of the object to be eliminated, and the image elimination and repair process in the case of a slightly larger area ratio P (such as 2%≤P<10%) of the object to be eliminated. The image elimination and repair process in the case of a larger area ratio P (such as P<25%) of the object to be eliminated will be described below.

[0318] Third image processing strategy is adopted when P<25%

[0319] S13, in the case of determining that P is not less than 10%, it is determined whether P is less than 25%.

[0320] In the case of determining that P is less than 25%, S14 is continued to determine whether the current elimination scene is a crowd background elimination scene. In the case of determining that P is not less than 25%, S25 is continued.

[0321] S14, in the case of determining that P is less than 25%, judging whether the current elimination scene is a crowd background elimination scene.

[0322] The crowd background elimination scene refers to that the object to be eliminated is a portrait or a still image, and the background of the object to be eliminated is a crowd background (portraits are blocked or intersected). The specific judgment method will be described below.

[0323] In the case of judging that the current elimination scene is not a crowd background elimination scene, S15-S19 are continued to be executed, or in the case of judging that the current elimination scene is a crowd background elimination scene, S20-S24 are continued to be executed.

[0324] Firstly, the elimination and repair process of the case that the current elimination scene is not a crowd background elimination scene is described below.

[0325] S15, in the case that the current elimination scene is not a crowd background elimination scene, the first image and the first mask image are scaled to the required size of the SD image elimination model.

[0326] S16, the scaled first mask image is subjected to mask region expansion processing to obtain an expanded first mask image.

[0327] S17, the scaled first image and the scaled and expanded first mask image are input into the SD image elimination model for model inference to obtain a third image.

[0328] S18, the expanded first mask image is restored in size to the first size to obtain an expanded first mask image.

[0329] S19, according to the first image, the third image and the expanded first mask image, a second image is obtained.

[0330] In the embodiment of the present application, the second image is obtained through the following image matrix calculation formula.

[0331] Pi = Po x (1-Pm) + Pe x Pm.

[0332] Wherein, Po represents the first image, Pe represents the third image, Pm represents the expanded first mask image, and Pi represents the second image.

[0333] Wherein, (1-Pm) represents taking the pixel value of the expanded first mask image as the opposite.

[0334] Figure 13 A flowchart of image elimination and repair in the case that the current elimination scene is not a crowd background elimination scene is shown.

[0335] AsFigure 13 As shown, in response to the selection operation of the user on the first image (original image), the electronic device determines the circle selection region, and calculates the area proportion P of the to-be-removed object in the first image, 10%≤P<25%.

[0336] The first mask image is acquired based on the first image, a mask region in the first mask image is a mask region of the to-be-removed object, and the sizes of the first image and the first mask image are both the first size.

[0337] Different from Figure 10 and Figure 12 , in Figure 13 , the image cropping processing is not performed, and the image padding is not performed.

[0338] Different from Figure 10 and Figure 12 , in Figure 13 , it is judged whether the current scene is a crowd background removal scene, that is, whether the to-be-removed object is a portrait and whether the background of the to-be-removed object is a crowd background.

[0339] In a case where it is determined that the current scene is not a crowd background removal scene, the first image and the first mask image are scaled, then the mask region of the scaled first mask image is dilated, and then the scaled first image and the scaled and dilated first mask image are input into the SD image removal model to obtain a third image.

[0340] The third image and the dilated first mask image are ANDed (that is, multiplied) to obtain a fourth image.

[0341] The pixel value of the dilated first mask image is inverted to obtain a fifth image. Then, the fifth image is multiplied by the first image to obtain a sixth image.

[0342] Then, the fourth image and the sixth image are image fused to obtain a second image.

[0343] As Figure 13 shown, the first object image in the second image is removed, and the background image of the removed region is repaired.

[0344] Through the scheme of the present application, for a to-be-removed object with a larger area proportion, the removal and repair requirements are higher, and therefore a more excellent image removal model is adopted, so that a better removal and repair effect can be obtained.

[0345] The above describes the removal and repair process in a case where the current removal scene is not a crowd background removal scene. The following describes the removal and repair process in a case where the current removal scene is a crowd background removal scene.

[0346] In the embodiments of the present application, the first object image with a crowd background (which can be a human body or an article) is removed, and this image removal scenario has higher requirements because after the first object image with a crowd background is removed, a distorted portrait can be generated in the area where the first object image is removed. In order to avoid generating a distorted portrait, the present application adopts the following scheme to perform image removal and repair processing.

[0347] S20, in the crowd background removal scenario, a multi-portrait instance mask image (i.e., a second mask image) is obtained based on the first image.

[0348] S21, the first image and the multi-portrait instance mask image are scaled to the required size (third size) of the SD image removal model.

[0349] For example, the required size of the SD image removal model is 768*768.

[0350] S22, the scaled first image and the multi-portrait instance mask image are input into the SD image removal model for model inference to obtain a third image.

[0351] It should be noted that the size of the third image is the third size, for example, the size of the third image can be 768*768.

[0352] S23, the mask area of the first mask image is dilated to obtain a dilated first mask image.

[0353] The present application does not limit the execution order of S22 and S23, for example, S22 can be executed first, and then S23 can be executed; or S23 can be executed first, and then S22 can be executed; or S22 and S23 can be executed simultaneously.

[0354] S24, a second image is obtained according to the first image, the third image, and the dilated first mask image.

[0355] In the embodiments of the present application, the second image is obtained by the following image matrix calculation formula.

[0356] Pi = Po x (1-Pm) + Pe x Pm.

[0357] Wherein, Po represents the first image, Pe represents the third image, Pm represents the dilated first mask image, and Pi represents the second image.

[0358] Wherein, (1-Pm) represents taking the inverse of the pixel value of the dilated first mask image.

[0359] Figure 14A The flowchart of image removal and repair in the case of 10%≤P<25% is shown.

[0360] As Figure 14A shown, in response to the selection operation of the user on the first image (original image), the electronic device determines the circled region or the smearing region and determines the object to be eliminated, and calculates the area ratio P of the object to be eliminated in the first image, 10%≤P<25%.

[0361] It should be noted that the judgment logic of the present application can determine that the first image contains multiple portraits and the portrait in the selected region is closely adjacent to other portraits in position (i.e., closely adjacent), and this elimination scenario is a crowd background elimination scenario, i.e., a certain portrait in the crowd is eliminated and repaired, and the requirement for image elimination and repair is relatively high.

[0362] The first mask image is obtained based on the first image, and the mask region in the first mask image is the mask region of the object to be eliminated. The sizes of the first image and the first mask image are both the first size.

[0363] Figure 14A Different from Figure 13 , in Figure 14A , a multi-portrait instance mask image (i.e., a second mask image) is obtained based on the first image. The multi-portrait instance mask image includes a mask region of each portrait in the first image.

[0364] Figure 14A Different from Figure 13 , in Figure 14A , the first image and the second mask image are scaled, and then the scaled first image and the scaled second mask image are input into the SD image elimination model to obtain a third image.

[0365] Figure 14A Different from Figure 13 , in Figure 14A , the mask region of the first mask image is dilated to obtain a dilated first mask image.

[0366] The third image and the dilated first mask image are ANDed (i.e., multiplied) to obtain a fourth image.

[0367] The pixel value of the dilated first mask image is inverted to obtain a fifth image. Then, the fifth image is multiplied by the first image to obtain a sixth image.

[0368] Then, the fourth image and the sixth image are image fused to obtain a second image.

[0369] Referring to Figure 14A , the portrait in the selected region of the second image is eliminated, and the background image of the eliminated region has been repaired.

[0370] Referring toFigure 14B As shown, by the embodiments of the present application, the image removal and repair in the multi-person scene is improved, and after performing the removal and repair processing on the image area specified by the user in the photo, the new portrait is avoided to be generated, thus the present application solves the problem of removing a portrait in the multi-person scene and generating a deformed portrait again, and improves the image removal and repair experience.

[0371] It should be noted that the present application is exemplarily described by taking the removal of a portrait in a crowd background as an example, and the removal of a portrait in a crowd background is avoided to generate a deformed portrait. In actual implementation, the embodiments of the present application are also applicable to the scene of removing a portrait (for example, a garbage can) in a crowd background, that is, a portrait in a crowd background can be removed, and the problem of generating a deformed portrait can also be avoided.

[0372] S25, in the case of determining that P is greater than or equal to 25%, prompting that the selection operation is invalid.

[0373] In the case of the area selection ratio being too large, in order to avoid poor image removal and repair effect, the selection operation can be prompted to be invalid, and reoperation is suggested to ensure good image removal and repair effect and improve user experience.

[0374] By the method provided by the present application, according to the area ratio of the object to be removed in the original image, a suitable image processing strategy is selected to remove and repair the object to be removed, the image processing is more precise, and the removal and repair effect is improved.

[0375] It should be noted that the above threshold values 2%, 10%, and 25% are exemplarily described, and in actual implementation, the size of the threshold value can be adjusted according to actual use requirements, for example, the threshold value can be adjusted to 3%, 12%, 50%, or 1%, 15%, and 75%.

[0376] In actual implementation, the number of threshold values can also be increased or reduced according to actual use requirements.

[0377] Exemplarily, one threshold value can be set, for example, 25%. When the area ratio of the object to be removed in the first image is less than the threshold value 25%, one image processing strategy can be used for image removal and repair. When the area ratio of the object to be removed in the first image is greater than or equal to the threshold value 25%, another image processing strategy can be used for image removal and repair.

[0378] Exemplarily, two threshold values can be set, for example, 10% and 25%. In this way, the calculation amount can be reduced while ensuring the image removal and repair effect.

[0379] Exemplarily, four threshold values, for example, 2%, 10%, 15%, and 25%, can be set, so that the image processing can be more refined, and the image elimination and repair effect can be improved.

[0380] The image elimination and repair processing method provided by the embodiments of the present application can be applied to an image elimination scene with a simple background, and can also be applied to an image elimination scene with a complex background.

[0381] In the scheme of the present application, the elimination and repair method based on the GAN image elimination model and the elimination and repair method based on the SD image elimination model are effectively combined, so that the elimination and repair effect and performance can be improved.

[0382] Image elimination scenario of judging whether it is a crowd background

[0383] In the above embodiments, the elimination scene of the crowd background is described, and the possible implementation manner of determining whether the image elimination scene is the crowd background provided by the present application is described in detail.

[0384] Figure 14C An exemplary flowchart for determining whether the image elimination scene is the crowd background provided by the embodiments of the present application is shown.

[0385] S401, input the first image into a portrait instance segmentation network or a panorama segmentation network model to obtain a second mask image. The second mask image includes a mask region of Y portraits. The second mask image can also be referred to as a portrait instance segmentation mask image.

[0386] S402, determine whether Y is greater than a threshold N15.

[0387] Optionally, N15 is 3.

[0388] If the second mask image is empty (Y=0), that is, there is no mask region or no region marked as 1, it indicates that there is no portrait in the first image. Therefore, it can be determined that the scene does not belong to the crowd background elimination scene.

[0389] If the second mask image is not empty, that is, there is a mask region or a region marked as 1, it indicates that there is at least one portrait in the first image.

[0390] In some embodiments, if the second mask image is not empty (that is, there is a portrait), and the number of portrait instances Y is less than N15, it can be determined that the scene does not belong to the crowd background elimination scene.

[0391] In some embodiments, if the second mask image is not empty (that is, there is a portrait), and the number of portrait instances is greater than or equal to N15, S403 is continued to be executed.

[0392] S403, traverse to calculate the pixel number (denoted as Q1) of the intersection area (intersectionArea) and the pixel number (denoted as Q2) of the union area (unionArea) between the first mask region (i.e. the mask region of the object to be eliminated) and the i-th portrait mask region in the second mask image, and the intersection-over-union ratio (denoted as IoU). i starts from 1 and takes a value up to Y.

[0393] wherein, IoU = Q1 / Q2.

[0394] S404, determine whether Q1 / Q2 is greater than a threshold N16.

[0395] Optionally, N16 = 0.8.

[0396] S405, if Q1 / Q2 ≥ N16, determine that the first mask region is a single-person region, then stop the traversal calculation and record the i-th portrait mask region.

[0397] If no portrait instance mask satisfying the condition is found after the traversal, it is determined that the first mask region does not contain a portrait, and then it can be determined that the scene does not belong to the elimination scene of the crowd background.

[0398] S406, traverse to calculate the distance L between the i-th portrait mask region and the j-th portrait mask region in the second mask image, j starts from 1 and takes a value up to Y.

[0399] After recording the single-person portrait mask region, it can be determined whether the scene is the elimination scene of the crowd background according to the close adjacency state of the recorded single-person portrait mask and the remaining portrait masks.

[0400] S407, determine whether L is less than D, D = WxN17.

[0401] Optionally, N17 can be 0.5.

[0402] wherein, W represents the width of the minimum bounding rectangle of the single-person portrait mask.

[0403] Specifically, it is determined whether the minimum bounding rectangle of the recorded i-th portrait mask intersects with the minimum bounding rectangle of a certain remaining portrait mask or the minimum distance is less than D.

[0404] S408, if the minimum bounding rectangle of the recorded i-th portrait mask intersects with the minimum bounding rectangle of a certain remaining portrait mask or the minimum distance is less than WxN17, it is determined that the i-th portrait mask and the remaining portrait mask have a close adjacency state, and then the traversal calculation is stopped. In this case, it can be determined that the scene belongs to the elimination scene of the crowd background.

[0405] S409. If no human face instance mask meets the conditions after traversal, it is determined that the scene is not tightly connected and therefore does not belong to the elimination scene with a crowd background (i.e., the elimination scene without a crowd background).

[0406] In this embodiment of the application, in the scenario of eliminating and repairing objects selected by circling or smearing in the first image, it is determined whether the current scene is an elimination scenario with a crowd background based on the first mask image and the mask image of multiple portrait instances (i.e., the second mask image). If it is determined that the current scene is an elimination scenario with a crowd background, an elimination and repair method suitable for crowd backgrounds is adopted to perform image elimination and repair, so as to avoid generating deformed portraits. This can optimize the effect of elimination and repair processing.

[0407] Determine if the elimination condition is met.

[0408] In some embodiments, in scenarios where objects to be eliminated are selected by circling or smearing in the original image, in order to avoid the elimination of part of the human body image due to the circling or smearing area being located on the human body, resulting in a fragmented human body image and affecting the elimination effect, the solution of this application can first determine whether the circling or smearing area meets the elimination conditions. If the circling or smearing area meets the elimination conditions, then the elimination and repair process is performed. If the circling or smearing area does not meet the elimination conditions, then the elimination and repair process is not performed. This can optimize the effect of elimination and repair processing.

[0409] For example, combined Figure 5 ,like Figure 15 As shown, after S302 and before S303, the method also includes S305 and S306.

[0410] S305. Obtain a second mask image and a third mask image based on the first image. The second mask image includes mask regions of multiple human figures, and the third mask image includes mask regions corresponding to the selected regions.

[0411] The second mask image includes the mask region corresponding to each portrait in the first image.

[0412] In some embodiments, the first image can be input into a human instance segmentation network to obtain a second mask image. The second mask image can also be called a human instance segmentation mask image (HumanInstanceMask). The human instance segmentation network is a network model used to identify human figures or human bodies in the first image and generate a mask image for each human figure.

[0413] Wherein, after inputting the first image into the portrait instance segmentation network model, a mask image of each portrait instance in the first image can be obtained, and by combining the mask images of all the portrait instances in the first image, a second mask image can be obtained.

[0414] In some other embodiments, the first image can also be input into a panorama segmentation network model to obtain a portrait instance segmentation mask image. Wherein, the panorama segmentation network model is a network model for generating a mask image of each person or object after identifying the person or object in the first image.

[0415] Wherein, after inputting the first image into the panorama segmentation network model, a mask image of each person or object in the first image can be obtained, and by combining the mask images of all the persons in the first image, a second mask image can be obtained.

[0416] Figure 16 A schematic diagram of obtaining a second mask image according to a first image is shown. As shown in Figure 16 The first image includes three portrait instances: portrait 1, portrait 2 and portrait 3; correspondingly, the second mask image generated according to the first image includes three portrait mask regions: mask 1, mask 2 and mask 3, which are the mask regions corresponding to the three portrait instances respectively.

[0417] It should be noted that if the second mask image is empty, i.e. there is no mask region or no region marked as 1, it means that there is no portrait in the first image. If the second mask image is not empty, i.e. there is a mask region or a region marked as 1, it means that there is at least one portrait in the first image.

[0418] S306, according to the third mask image and the second mask image, judging whether the circle selection region or the smearing region meets the elimination condition.

[0419] It should be noted that the purpose of judging whether the circle selection region or the smearing region meets the elimination condition is to avoid the partial image of the human body being eliminated due to the circle selection region or the smearing region being located in the human body part, forming a disfigured human body image and affecting the elimination effect.

[0420] Figure 17A flowchart is shown for determining whether the selected region or the smearing region meets the elimination condition according to the third mask image and the second mask image. It can be understood that in the case where it is determined that the selected region or the smearing region meets the elimination condition, further selection of the elimination and repair method is allowed, and the image elimination and repair processing is performed on the object to be eliminated. In the case where it is determined that the selected region or the smearing region does not meet the elimination condition, it can be prompted that the selection operation is invalid. In this way, it can be avoided that part of the image of the human body is eliminated due to the selected region being located on the human body part, and thus the elimination and repair effect for the human image can be improved.

[0421] In some embodiments, if the second mask image is empty, that is, there is no human image in the first image, the object to be eliminated is not a human body, that is, the object to be eliminated can be a garbage can or an article such as sundries, which meets the elimination condition, and thus the elimination and repair processing can be performed on the object to be eliminated in the selected region of the first image.

[0422] If the second mask image is not empty, that is, there is a human image in the first image, the following parameters are obtained:

[0423] (1) The area S of the mask region of the third mask image (the mask region corresponds to the selected region in the first image) can be determined by counting the number of pixels of the first mask region. For example, as shown in Figure 18 , the white region in the third mask image is the mask region, and the number of pixels of the mask region is calculated, that is, the number of pixels S of the selected region or the smearing region is obtained.

[0424] (2) The area Si of each human image mask region of the second mask image is calculated. For example, as shown in Figure 18 , the second mask image includes three human image mask regions: mask 1, mask 2 and mask 3, the number of pixels of mask 1 is S1, the number of pixels of mask 2 is S2, and the number of pixels of mask 3 is S3. Si includes S1, S2 and S3.

[0425] (3) The area S' of the intersection region between the first mask region and each human image mask region in the second mask image is calculated. The number of pixels of the intersection region between the first mask region and mask 1 is S1', the number of pixels of the intersection region between the first mask region and mask 2 is S2', and the number of pixels of the intersection region between the first mask region and mask 3 is S3'. Si' includes S1', S2' and S3'.

[0426] wherein the first mask region and mask 2 have an intersection region (as Figure 18The pixel number S2' of the intersection region is greater than zero. The first mask region has no intersection region with mask 1 and mask 3, the pixel number S2' of the intersection region is 0, and S3' is 0.

[0427] In the embodiments of the present application, after the parameters S, Si (including S1, S1 and S3) and Si' (including S1', S2' and S3') are calculated, whether the selected region or the smearing region meets the elimination condition is determined in the following manner. i is taken as 1, 2 and 3 in turn.

[0428] If Si' / Si ∈ (N1, N2) and Si' / S > N3 are met, it is determined that the selected region or the smearing region does not meet the elimination condition, and the elimination processing is not performed on the object to be eliminated.

[0429] Alternatively, N1 can be 1%, N2 can be 50%, and N3 can be 80%. For the convenience of description, the following embodiments are exemplarily described with N1 being 1%, N2 being 50%, and N3 being 80%. That is, when Si' / Si ∈ (1%, 50%) and Si' / S > 80% are met, it is determined that the selected region or the smearing region does not meet the elimination condition.

[0430] It should be noted that Si' / S can be understood as the area ratio of the intersection region Si' to the selected region or the smearing region S. If the area ratio is less than or equal to 80%, it means that part of the selected region or the smearing region falls within the portrait region, and the object to be eliminated can be a portrait or other objects. If the area ratio is greater than 80%, it means that most or all of the selected region falls within the portrait region, and the object to be eliminated can be more accurately identified as a portrait.

[0431] Si' / Si can be understood as the area ratio of the intersection region Si' to the portrait Si. If the area ratio is within the range of (1%, 50%), it means that the selected region or the smearing region covers part of the portrait region, and the elimination processing should not be performed on the object to be eliminated to avoid the false elimination of part of the human body region, cause the human body image to be incomplete, and affect the image elimination effect.

[0432] That is, if Si' / S > 80% and Si' / Si ∈ (1%, 50%) are met, it is determined that the object to be eliminated is a human body and the selected region or the smearing region is part of the human body, and it is considered that the elimination condition is not met, and the elimination processing is not performed on the object to be eliminated.

[0433] If Si' / Si does not belong to (1%, 50%), it is determined that the elimination condition is met, and the elimination processing can be performed on the object to be eliminated.

[0434] It should be noted that the calculation can be performed on each portrait mask area. If the elimination conditions are not met, the calculation can be stopped and an invalid selection operation will be displayed. If the conditions are met after traversing and calculating all portrait mask areas, the selected or painted area is determined to meet the elimination conditions, and the elimination process can be performed on the object to be eliminated. By performing traversal calculations, the amount of computation can be reduced.

[0435] The solution proposed in this application can determine whether the selected or painted area meets the elimination conditions in response to the user's selection operation on the first image. If the selected or painted area meets the elimination conditions, the object to be eliminated is eliminated. If the selected or painted area does not meet the elimination conditions, the object to be eliminated is not eliminated, so as to prevent accidental elimination, such as preventing accidental elimination of human body parts in the image.

[0436] The following illustrations, with reference to the accompanying drawings, illustrate scenarios where the selected or painted area meets the elimination criteria and scenarios where the selected or painted area does not meet the elimination criteria.

[0437] Scenes where the selected area (circled area or painted area) meets the elimination criteria.

[0438] The following is for reference. Figure 18 This diagram illustrates how to determine if a selected or painted area meets the elimination criteria. For example... Figure 18 As shown, the selected area in the first image is indicated by the dashed line. The number of pixels S of the object to be eliminated is determined by the mask area in the third mask image; the number of pixels S1, S2, and S3 of each portrait is determined by the mask area of ​​each portrait in the second mask image; the number of pixels S1′, S2′, and S3′ of the intersection area of ​​the first mask area and each portrait mask area is calculated. S1′ = 0, S3′ = 0.

[0439] like Figure 18 As shown, starting with i = 1, the traversal begins. Since S1′ = 0, S1′ / S1 = 0, and S1′ / S = 0. The traversal continues. Then, i = 2, S2′ / S2 > 50%, and S2′ / S > 80%. The traversal continues. Then, i = 3, since S3′ = 0, S3′ / S3 = 0, and S3′ / S = 0. The traversal calculation ends. Si′ / Si does not belong to (1%, 50%), satisfying the elimination condition. Therefore, the elimination process can be performed on the object to be eliminated.

[0440] Scenes where the selected area (circled area or painted area) does not meet the elimination criteria.

[0441] The following is for reference. Figure 19 This diagram illustrates how to determine if a selected or painted area does not meet the elimination criteria. For example... Figure 19As shown in the first image, the selected region as shown by the dashed line can determine the pixel number S of the object to be eliminated through the mask region in the third mask image, and the pixel number S1, S2, S3 of each person image can be determined through each person image mask region in the second mask image. The pixel number S1', S2', S3' of the intersection region of the first mask region and each person image mask region can be calculated.

[0442] As shown in the first image, the selected region as shown by the dashed line can determine the pixel number S of the object to be eliminated through the mask region in the third mask image, and the pixel number S1, S2, S3 of each person image can be determined through each person image mask region in the second mask image. The pixel number S1', S2', S3' of the intersection region of the first mask region and each person image mask region can be calculated. Figure 19 As shown in the first image, the selected region as shown by the dashed line can determine the pixel number S of the object to be eliminated through the mask region in the third mask image, and the pixel number S1, S2, S3 of each person image can be determined through each person image mask region in the second mask image. The pixel number S1', S2', S3' of the intersection region of the first mask region and each person image mask region can be calculated.

[0443] As shown in the first image, the selected region as shown by the dashed line can determine the pixel number S of the object to be eliminated through the mask region in the third mask image, and the pixel number S1, S2, S3 of each person image can be determined through each person image mask region in the second mask image. The pixel number S1', S2', S3' of the intersection region of the first mask region and each person image mask region can be calculated.

[0444] As shown in the first image, the selected region as shown by the dashed line can determine the pixel number S of the object to be eliminated through the mask region in the third mask image, and the pixel number S1, S2, S3 of each person image can be determined through each person image mask region in the second mask image. The pixel number S1', S2', S3' of the intersection region of the first mask region and each person image mask region can be calculated. Figure 18 If the electronic device determines that the area ratio of mask 3 in the second mask image is less than 1%, the electronic device can not perform the traversal calculation on mask 3.

[0445] As shown in the first image, the selected region as shown by the dashed line can determine the pixel number S of the object to be eliminated through the mask region in the third mask image, and the pixel number S1, S2, S3 of each person image can be determined through each person image mask region in the second mask image. The pixel number S1', S2', S3' of the intersection region of the first mask region and each person image mask region can be calculated.

[0446] It should be noted that various possible ways can be used to determine whether the selected region or the smearing region meets the elimination condition, which is not limited in the present application. For example, the present application also provides the following implementation manner to determine whether the selected region or the smearing region meets the elimination condition.

[0447] Exemplarily, after calculating the pixel number S of the object to be eliminated, the pixel number Si (including S1, S1 and S3) of each portrait and the pixel number Si' (including S1', S2' and S3') of the intersection region and other parameters, it is determined whether the selected region or the smearing region meets the elimination condition in the following manner.

[0448] In some embodiments, if the ratio of the pixel number Si' of the intersection region to the pixel number Si of the portrait region is within the range of (N1, N2) and the BBOX proportion is greater than N5, it is determined that the selected region or the smearing region does not meet the elimination condition. In this case, the selected region or the smearing region usually contains a human body and will cause an elimination effect on the human body region, so the elimination process is not performed on the object to be eliminated.

[0449] The BBOX proportion is the width of the bounding rectangle of the intersection region of the portrait i, divided by the width of the mask bounding rectangle of the portrait i.

[0450] Wherein, the intersection region of the portrait i refers to the intersection region between the mask region of the portrait i and the first mask region.

[0451] Let the BBOX proportion be Y(i). Y(i) = BBOX(i').width / BBOX(i).width.

[0452] Wherein, BBOX represents the bounding rectangle. BBOX(i).width is the width of the mask bounding rectangle of each portrait. BBOX(i').width represents the width of the bounding rectangle of the intersection region of each portrait.

[0453] Figure 20 A schematic diagram of the BBOX proportion is shown as Figure 20 As shown, it can be determined that mask2 has an intersection region with the first mask region, and the intersection region proportion is Y(2) = BBOX(2').width / BBOX(2).width.

[0454] Optionally, N5 can be 30%.

[0455] It can be understood that when the BBOX proportion is greater than 30%, it means that more than 30% of the selected region or the smearing region falls within the region of the human body image. If the ratio of the pixel number Si' of the intersection region to the pixel number Si of the portrait region is within the range of (1%, 50%), it can be determined that the object to be eliminated is a human body and the selected region is part of the human body, so it can be determined that the selected region or the smearing region does not meet the elimination condition. In this case, the selected region or the smearing region usually contains a human body and will cause an elimination effect on the human body region, so the elimination process is not performed on the object to be eliminated.

[0456] In other embodiments, if the elimination conditions are met after the traversal calculation is completed, this usually means that the selected area or the smeared area does not contain a human body (i.e., the object to be eliminated is not a human body), or the selected area or the smeared area contains a human body but will not have an elimination effect on the human body area. Therefore, the object to be eliminated can be eliminated.

[0457] Figure 21 A comparative diagram is shown to determine whether the selected area meets the elimination criteria.

[0458] like Figure 21 As shown in (a), the selected region 21 surrounds the human body image. Through the above judgment process, it can be determined that the selected region meets the elimination conditions, so elimination and repair processing is performed.

[0459] like Figure 21 As shown in (b) above, the dashed box 22 is the selected area in the first image. The selected area does not completely surround the human figure, but only covers a part of the human body. Through the above judgment process, it can be determined that the selected area does not meet the elimination condition. In this case, the electronic device can prompt that the selection operation is invalid.

[0460] In this embodiment of the application, if the selected area (e.g., the circled area or the painted area) does not meet the elimination conditions, the electronic device may display a prompt message, such as "This photo does not support elimination".

[0461] For example, such as Figure 22A As shown in (a) and (b) of the image, in the scenario of AI elimination through intelligent selection, in response to the user's selection operation in the first image, the electronic device displays the selection trajectory 51 in the first image. Through the solution of this application, the electronic device determines that the selected area corresponding to the selection trajectory 51 is a human body area, which does not meet the elimination conditions, as shown in (a) and (b). Figure 22A As shown in (c), the electronic device displays the message 52 "Prompt: This photo does not support deletion".

[0462] For example, such as Figure 22B As shown in (a) and (b) of the image, in a scenario where AI removal is performed by manual smearing, in response to the user's manual smearing operation in the first image, the electronic device displays smear mark 53 in the first image. Using the solution of this application, the electronic device determines that the area corresponding to smear mark 53 is a human body area and does not meet the removal conditions, as shown in (a) and (b). Figure 22B As shown in (c), the electronic device displays the message 54 "Tip: This photo does not support deletion".

[0463] The proposed solution responds to a user's selection or smearing operation on the first image, and if the selected or smeared area meets the elimination conditions, image elimination and repair are then performed.

[0464] It should be noted that in the embodiments of the present application, "greater than" can be replaced by "greater than or equal to", "less than or equal to" can be replaced by "less than", or "greater than or equal to" can be replaced by "greater than", and "less than" can be replaced by "less than or equal to".

[0465] The various embodiments described herein can be independent solutions or combined according to inherent logic, and all fall within the protection scope of the present application.

[0466] The above mainly describes the solutions provided by the embodiments of the present application from the perspective of method steps. It can be understood that, in order to realize the above functions, the electronic device implementing the method contains the corresponding hardware structure and / or software module for executing each function. Those skilled in the art should realize that, in combination with the units and algorithm steps of the examples described in the embodiments disclosed herein, the present application can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is realized in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the protection scope of the present application.

[0467] The embodiments of the present application can divide the electronic device into functional modules according to the above method examples. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be realized in the form of hardware or software functional module. It should be noted that the division of modules in the embodiments of the present application is illustrative, and is only a logical function division. Actual implementation can have other feasible division manners. The following takes dividing each functional module according to each function as an example for description.

[0468] Figure 23 A schematic block diagram of the image elimination and repair processing apparatus 500 provided by the embodiments of the present application. The apparatus 500 can be used to execute the actions performed by the electronic device in the above method embodiments. The apparatus 500 includes an image shooting unit 510, an image display unit 520 and an image processing unit 530.

[0469] The image shooting unit 510 is configured to, after a camera application is enabled, shoot a first image in response to a first operation of a user; the first image includes a first object image, a second object image and a background image.

[0470] The image display unit 520 is configured to display the first image.

[0471] The image processing unit 530 is configured to determine a selected region and an object to be eliminated in response to a first operation of a user on the first image; the selected region includes the first object image, and the first object image is the object to be eliminated.

[0472] The image processing unit 530 is further configured to obtain a first mask image and a second mask image according to the first image; the mask region in the first mask image is a mask region of the object to be eliminated, and the second mask image includes a mask region of each person in the first image.

[0473] The image processing unit 530 is further configured to perform image processing on the first image and the first mask image or on the first image and the second mask image according to a first area ratio of the object to be eliminated in the first image, to obtain a third image; the third image includes the background image but does not include the first object image.

[0474] The image processing unit 530 is further configured to perform image processing on the first image, the third image and the first mask image to obtain a second image; the second image includes the second object image and the background image but does not include the first object image.

[0475] The image display unit 520 is configured to display the second image as an update of the first image.

[0476] The image processing device provided by the embodiment of the present application can determine a selected region and perform image elimination and background repair on a first object image in the selected region in response to a user operation on a first image after the first image is captured. The area ratio of an object to be eliminated in the first image can be used to select image processing on the first image and a first mask image or image processing on the first image and a second mask image to generate a third image (which does not include the first object image and includes a background image). Then, a second image (which does not include the first object image and includes a second object image and a background image) is generated based on the first image, the third image and the first mask image. Through the solution of the present application, a user can directly select a region on a photo after the photo is captured, and the selected region can be eliminated by one key. The area ratio of the object to be eliminated is used to select a suitable image processing manner for image elimination and background repair, so that the image processing is more accurate, and the elimination and repair effect is improved.

[0477] The apparatus 500 according to the embodiment of the present application can correspond to performing the method described in the embodiment of the present application, and the above and other operations and / or functions of the units in the apparatus 500 are respectively used to implement the corresponding flow of the method, and for brevity, will not be described here.

[0478] The application further provides a chip coupled with the memory, the chip being configured to read and execute the computer program or instructions stored in the memory to perform the method in each of the embodiments.

[0479] The application further provides an electronic device comprising a chip configured to read and execute the computer program or instructions stored in the memory so that the method in each of the embodiments is performed.

[0480] The embodiment further provides a computer readable storage medium storing computer instructions, when the computer instructions are run on an electronic device, the electronic device performs the related method steps to implement the processing method of image elimination and repair in the above embodiments.

[0481] The embodiment further provides a computer program product, the computer readable storage medium stores program codes, when the computer program product is run on a computer, the computer performs the related steps to implement the processing method of image elimination and repair in the above embodiments.

[0482] In the embodiments, the electronic device, the computer readable storage medium, the computer program product or the chip are used to perform the corresponding method provided above, and thus the beneficial effects achieved by the electronic device, the computer readable storage medium, the computer program product or the chip can refer to the beneficial effects of the corresponding method provided above, which will not be described herein again.

[0483] In the several embodiments provided in the present application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are only schematic. The division of the modules or units is only a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the displayed or discussed coupling or direct coupling or communication connection between the units can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0484] The term "and / or" used herein is a description of an association relationship between associated objects, which means that there can be three relationships, for example, A and / or B, which means that there can be three cases of A alone, A and B together, and B alone. The symbol " / " in this paper represents the relationship of or, for example, A / B represents A or B.

[0485] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. In the description of embodiments in this application, unless otherwise stated, "multiple" means two or more; for example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0486] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A processing method for image removal and repair, characterized in that, The method comprises: in response to a first operation of a user, enabling an image removal function; in response to a second operation of the user on a first image, highlighting a selected region in the first image; identifying an object to be removed in the selected region; determining a target image processing strategy according to a first area ratio of the object to be removed in the first image; performing removal and repair processing on the object to be removed according to the target image processing strategy to obtain a second image; displaying the second image; wherein the object to be removed is not included in the selected region of the second image, and the area after the object to be removed is removed is repaired as a background image of the object to be removed.

2. The method of claim 1, wherein, The method further comprises: when the first area ratio is less than a first threshold, a first image processing strategy is determined as the target image processing strategy; when the first area ratio is greater than or equal to the first threshold and less than a second threshold, it is determined whether the background image of the object to be removed is a complex background image; if not, a second image processing strategy is determined as the target image processing strategy; if yes, a third image processing strategy is determined as the target image processing strategy; when the first area ratio is greater than or equal to the second threshold and less than a third threshold, it is determined whether the background image of the object to be removed is a crowd background image; if not, a fourth image processing strategy is determined as the target image processing strategy; if yes, a fifth image processing strategy is determined as the target image processing strategy; wherein the first threshold is less than the second threshold, and the second threshold is less than the third threshold.

3. The method of claim 2, wherein, The method further comprises: inputting the first image and a target mask image into a target image removal model to obtain a third image; the target image removal model is determined according to the target image processing strategy; the target mask image is a mask image obtained based on the first image; obtaining the second image according to the third image, a first mask image, and the first image; a mask region in the first mask image is a mask region of the object to be removed.

4. The method of claim 3, wherein, The method further comprises: in a case of using the first image processing strategy, inputting the first image and a first mask image into a first image removal model to obtain the third image; in a case of using the second image processing strategy for a non-complex background image, inputting the first image and the first mask image into a second image removal model to obtain the third image; in a case of using the third image processing strategy for a complex background image, inputting the first image and the first mask image into a third image removal model to obtain the third image; In a case where the fourth image processing strategy is adopted for a non-human crowd background image, the first image and the first mask image are input into a fourth image elimination model to obtain the third image; In a case where the fifth image processing strategy is adopted for a human crowd background image, the first image and the second mask image are input into a fifth image elimination model to obtain the third image; The second mask image includes a mask region of each portrait in the first image.

5. The method of claim 3, wherein, The second image is obtained according to the third image, the first mask image and the first image, including: The third image and the first mask image are subjected to AND operation to obtain a fourth image; The first image and a fifth image are subjected to AND operation to obtain a sixth image; the fifth image is an image obtained by inverting pixel values of the first mask image; The fourth image and the sixth image are subjected to image fusion to obtain the second image.

6. The method of claim 5, wherein, The third image and the first mask image are subjected to AND operation, including: the third image and the first mask image after inflation are subjected to AND operation; The fifth image is an image obtained by inverting pixel values of the first mask image after inflation.

7. The method according to any one of claims 1 to 6, characterized in that, After the object to be eliminated in the selected region is identified, the method further includes: In a case where the object to be eliminated is identified as a portrait and the selected region covers a partial image of a human body, it is determined whether the current elimination operation meets an elimination condition according to a second mask image and a third mask image; a mask region of the third mask image is a mask region corresponding to the selected region; In a case where it is determined that the current elimination operation meets the elimination condition, a first area ratio of the object to be eliminated in the first image is obtained.

8. The method of claim 3 or 4, wherein, The third image is obtained by inputting the first image and the target mask image into a target image elimination model, including: In a case where the first area ratio is less than or equal to a second threshold, image cropping processing is performed on the first image and the target mask image, the cropped first image and the cropped target mask image are input into the target image elimination model to obtain a seventh image; The seventh image is filled into a cropped region of the first image to obtain the third image.

9. The method of claim 8, wherein, The third image is obtained by inputting the first image and the target mask image into a target image elimination model, including: The mask region of the cropped target mask image is subjected to second inflation processing; The cropped first image and the cropped and inflated target mask image are input into the target image elimination model to obtain the seventh image.

10. The method of claim 2, wherein, The background image of the object to be eliminated is identified as a complex background image, including: The mask region of the first mask image is subjected to first inflation processing to obtain a first inflation image; The mask region of the first mask image is subjected to second inflation processing to obtain a second inflation image; the inflation ratios of the first inflation processing and the second inflation processing are different; The second inflation image and the first inflation image are subtracted to obtain a first annular image; According to a ratio of the number of feature points of the first annular image to the number of pixels of the first annular image, a feature point density of the first annular image is obtained; If the feature point density of the first annular image is greater than a fourth threshold value, it is determined that the background image of the object to be removed is a complex background image; If the feature point density of the first annular image is less than or equal to the fourth threshold value, it is determined that the background image of the object to be removed is not a complex background image.

11. The method of claim 2, wherein, The identification of whether the background image of the object to be removed is a crowd background image comprises: If the number of portrait instances in the second mask image is less than a fifth threshold value, it is determined that the background image of the object to be removed is not a crowd background image; If the number of portrait instances in the second mask image is greater than or equal to the fifth threshold value, a first portrait mask region in the second mask image is determined according to the first mask image and the second mask image, if the first portrait mask region intersects with at least one other portrait mask region in the second mask image or the distance is less than a sixth threshold value, it is determined that the background image of the object to be removed is a crowd background image; if the distance between the first portrait mask region and at least one portrait mask region in the second mask image is greater than or equal to the sixth threshold value, it is determined that the background image of the object to be removed is not a crowd background image.

12. The method according to any one of claims 1 to 11, characterized in that, The highlighting of the selected region in the first image in response to the second operation of the user on the first image comprises: In response to a circle selection operation of the user on the first image, a closed region formed by a moving track of the circle selection operation is determined as the selected region; or, In response to a smearing operation of the user on the first image, the smearing region is determined as the selected region.

13. The method according to any one of claims 1 to 12, characterized in that, The method further comprises: According to a ratio of the number of pixels of the mask region in the first mask image to the number of pixels of the first mask image, the first area ratio is determined. The mask region in the first mask image is a mask region of the object to be removed.

14. The method of claim 2, wherein, The method further comprises: In a case where the first area ratio is greater than or equal to the third threshold value, it is determined that the removal and repair processing is not performed, and a first prompt information is displayed, the first prompt information is used to prompt the user that this operation is invalid.

15. An electronic device, comprising: The electronic device comprises one or more processors and a memory; The memory is coupled with the one or more processors, the memory is used to store computer program codes, the computer program codes comprise computer instructions, and the one or more processors invoke the computer instructions to enable the electronic device to perform the method in any one of claims 1 to 14.

16. A chip system, characterized by The chip system is applied to an electronic device, and the chip system comprises one or more processors, and the one or more processors are used to invoke computer instructions to enable the electronic device to perform the method in any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises instructions, when the instructions run on an electronic device, enable the electronic device to perform the method in any one of claims 1 to 14.

Images