Method for performing image restoration, storage medium, and computing device

By calculating the overlap of relative masks and generating appropriate masks, and combining AI technology for image repair, the problems of low image repair efficiency and distortion in the prior art are solved, and a more realistic image repair effect is achieved.

CN119090760APending Publication Date: 2024-12-06CANVA PTY LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411042818.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-06-27
Filing Date
2024-07-31
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Existing digital image repair techniques are inefficient and prone to image distortion when processing large areas, especially when the user-selected areas overlap with the foreground elements, it is difficult to maintain the realistic effect of the image.

Method used

By calculating the relative mask overlap and comparing according to the threshold, a merge mask and a reduction mask are generated, and image repair processing is combined with AI technology to ensure the retention of background and foreground elements and avoid distortion.

Benefits of technology

It improves the efficiency and quality of image repair, can fill unwanted areas more realistically, maintain the original features of the image, and reduce image distortion.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090760A_ABST
    Figure CN119090760A_ABST
Patent Text Reader

Abstract

The described embodiments generally relate to computer-implemented methods, non-transitory computer-readable storage media, and computing devices for performing image inpainting. The method comprises the following steps: accessing a first image; receiving a selected area of the first image; identifying a foreground region of the first image; generating a merged mask based on a union set of the user selected region and the foreground region; performing image restoration processing on a region of the first image corresponding to the merge mask to generate a second image, the second image being an image-restored image; generating a reduced mask based on the user-selected region subtracting the foreground region; and combining regions of the first image and the second image corresponding to the reduced mask to produce an output image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The described embodiments relate to systems and methods for performing image inpainting. In particular, the described embodiments relate to systems and methods for performing image inpainting of digital images. Background Art

[0002] Inpainting is a term used to describe an image processing technique that removes unwanted image elements and replaces them with new elements to preserve the realism of the original image. In the past, inpainting was performed manually on physical images by painting or otherwise covering the unwanted image elements with physical media. With the widespread adoption of digital image processing tools, digital image inpainting has become possible.

[0003] Digital image inpainting can be performed manually using digital image editing software to fill in areas of an image. This can be a very long and tedious task if high-quality results are required, especially when dealing with large areas. This is because this image inpainting method may require pixel-level processing of the image in order to replace the unwanted areas with realistic fills. Some automated methods have been developed, but these often produce unrealistic results and undesirably distort the image.

[0004] It would be desirable to address or ameliorate one or more disadvantages or shortcomings associated with existing systems and methods for performing image inpainting, or at least provide a useful alternative thereto.

[0005] Throughout the specification, the word "comprise" or variations such as "comprises" or "comprising" will be understood to imply the inclusion of stated elements, integers or steps, or groups of elements, integers or steps, but not the exclusion of any other elements, integers or steps, or groups of elements, integers or steps.

[0006] Any discussion of documents, acts, materials, devices, articles or the like included in this specification should not be taken as an admission that any or all of these matters form part of the prior art base or were common general knowledge in the field relevant to the present disclosure as it existed before the priority date of each appended claim. Summary of the invention

[0007] Some embodiments relate to a computer-implemented method for performing image restoration, the method comprising:

[0008] accessing a first image;

[0009] receiving a selected region of a first image;

[0010] identifying a foreground region of the first image;

[0011] generating a merge mask based on a union of the user-selected region and the foreground region;

[0012] Performing an image restoration process on a region of the first image corresponding to the merge mask to generate a second image, the second image being an image restoration image;

[0013] generating a reduced mask based on the user selected area minus the foreground area; and

[0014] The first image is combined with the region of the second image corresponding to the downscaling mask to produce an output image.

[0015] Some embodiments further include storing the output image to a memory location.

[0016] Some embodiments further include transmitting the output image to an external device.

[0017] Some embodiments further include:

[0018] Calculate the relative mask overlap between the user-selected area and the foreground area;

[0019] comparing the relative mask overlap to at least one threshold; and

[0020] The following steps are continued: generating a merge mask, performing an image inpainting process, generating a downscaling mask and combining the images based on the result of the comparison, wherein these steps form part of a foreground protection process.

[0021] According to some embodiments, the relative mask overlap is calculated using the following equation:

[0022]

[0023] Where RMO is the relative mask overlap, Num_pixels is the function that gets the number of pixels, US_area is the user selected area, and FG_area is the foreground area.

[0024] According to some embodiments, comparing the relative mask overlap to at least one threshold comprises comparing the relative mask overlap to a lower threshold, and performing foreground protection processing based on determining that the relative mask overlap is greater than the lower threshold.

[0025] In some embodiments, the lower threshold is 0%.

[0026] In some embodiments, based on determining that the relative mask overlap is less than or equal to a lower threshold, instead of executing a foreground protection algorithm, an image inpainting process is performed on a region of the first image corresponding to the user-selected region to generate an output image.

[0027] According to some embodiments, comparing the relative mask overlap to at least one threshold comprises comparing the relative mask overlap to an upper threshold, and performing foreground protection processing based on determining that the relative mask overlap is less than the upper threshold.

[0028] In some embodiments, the upper threshold is between 10% and 90%.

[0029] In some embodiments, the upper threshold is between 40% and 70%.

[0030] In some embodiments, the upper threshold is 50%.

[0031] According to some embodiments, based on determining that the relative mask overlap is greater than or equal to an upper threshold, instead of performing a foreground protection process, an image restoration process is performed on a region of the first image corresponding to the user-selected region to generate an output image.

[0032] Some embodiments further comprise the steps of dilating the merge mask and performing an image restoration process on a region of the first image corresponding to the dilated merge mask.

[0033] In some embodiments, the binning mask is dilated by between 1 and 9 pixels.

[0034] In some embodiments, the binning mask is dilated by between 3 and 7 pixels.

[0035] In some embodiments, the binning mask is expanded by 5 pixels.

[0036] Some embodiments further include performing a second image restoration process on an area of ​​the first image corresponding to the user-selected area to generate a second image.

[0037] Some embodiments further include performing a smoothing process on the output image before outputting the image to smooth an edge between the first image and the second image.

[0038] According to some implementations, the smoothing is performed using an alpha compositing process.

[0039] In some implementations, performing the image restoration process includes performing an AI-based image restoration process.

[0040] According to some embodiments, performing the image inpainting process includes performing a LaMa image inpainting process.

[0041] In some implementations, identifying a foreground region of the first image includes performing a background subtraction process on the first image.

[0042] Some embodiments relate to a non-transitory computer-readable storage medium storing instructions that, when executed by a processing device, cause the processing device to perform the methods described in some other described embodiments.

[0043] Some embodiments relate to a computing device comprising:

[0044] a non-transitory computer-readable storage medium as described in some other described embodiments; and

[0045] A processor configured to execute instructions stored in a non-transitory computer-readable storage medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The various figures in the drawings depict only example embodiments of the present disclosure and are not to be considered limiting of its scope.

[0047] Figures 1A to 1D showing a first series of example images showing the results of image restoration using prior art known techniques compared to the methods of some embodiments described below;

[0048] FIG. 2A to FIG. 2D showing a second series of example images showing results of image restoration using prior art known techniques compared to the methods of some embodiments described below;

[0049] Figure 3 is a schematic diagram of a system for performing image restoration according to some embodiments;

[0050] Figure 4 is a process flow diagram of a method for performing image restoration according to some embodiments;

[0051] FIG. 5A to FIG. 5K are example diagrams illustrating how to perform Figure 4 the steps of the method;

[0052] FIG. 6A to FIG. 6C shows a series of example images showing the results of performing a background separation step according to some embodiments;

[0053] FIG. 7A to FIG. 7D shows a series of example images showing the results of performing a relative mask overlap calculation step according to some embodiments;

[0054] FIG. 8A to FIG. 8D shows a series of example images showing the results of performing a merge mask generation step according to some embodiments;

[0055] 9A to 9Cshows a series of example images illustrating the results of performing a merged mask image inpainting step according to some embodiments;

[0056] FIG. 10A to FIG. 10C shows a series of example images showing the results of performing a user selected area image restoration step according to some embodiments;

[0057] FIG. 11A to FIG. 11D shows a series of example images illustrating the results of performing a reduction mask generation step according to some embodiments;

[0058] Figures 12A to 12D shows a series of example images showing the results of performing a reduced mask image inpainting step according to some embodiments; and

[0059] FIG. 13A to FIG. 13D A series of example images are shown, illustrating the results of performing a smoothed image merging step according to some embodiments. DETAILED DESCRIPTION

[0060] The described embodiments relate to systems and methods for performing image inpainting. In particular, the described embodiments relate to systems and methods for performing image inpainting of digital images.

[0061] Image inpainting refers to removing areas of an image, such as undesirable image elements, and filling the removed areas so that the image looks realistic and as if the undesirable image elements were never there. Automatic digital techniques have been developed that allow a user to select one or more areas of an image for removal, and which will automatically fill the selected areas in a manner that replaces the area with a realistic substitute. However, these techniques often distort surrounding image areas, especially when the area selected for removal overlaps with foreground elements of the image.

[0062] Figures 1A to 1C and FIG. 2A to FIG. 2C An example of distortion that may occur using prior art techniques is shown.

[0063] Figure 1AAn example image 100 is shown having a foreground image element 105 and a background image element 110. In the example shown, the foreground image element 105 includes a person located near a camera capturing the image and is the primary subject of the image, while the background image element 110 includes a train, people, and a cityscape located behind the person, serving as a background. The image 100 also includes an unwanted image element 115, which is not the intended subject of the image, but is a person partially captured behind the subject of the image. The unwanted image element 115 is directly adjacent to and in contact with an edge of the partial foreground image element 105.

[0064] Figure 1B An image 120 is shown. Image 120 has the same foreground image element 105 and background image element 110 as image 100. The difference between image 100 and image 120 is that image 120 includes a user-selected area 125 applied over the unwanted image element 115. According to some embodiments, a user who wishes to edit image 100 may interact with a user interface component of a computing device displaying image 100 to indicate which area of ​​image 100 they consider to be an unwanted image element and wish to remove from the image. For example, a user may use a brush tool, a tracking tool, or other tool to digitally select, track, circle, or "paint" over the unwanted image element through a user interface to produce a user-selected area 125, which may be used as a mask to perform an image restoration method. However, a user may inadvertently select an area that they do not want to remove from image 100 as part of the user-selected area 125. For example, a user may inadvertently select an area of ​​the foreground image element 105. In the illustrated embodiment, the user-selected area 125 overlaps a portion of the foreground image element 105 corresponding to the back and neck of the subject.

[0065] Figure 1CAn image 140 is shown. Image 140 has the same foreground image elements 105 and background image elements 110 as image 100. The difference between image 100 and image 140 is that image 140 has had image inpainting processing performed on a user-selected region (i.e., user-selected region 125). In image 140, image inpainting has been performed by a previously known image inpainting technique, using an algorithm to fill in the area indicated by user-selected region 125 so that the image looks as if the unwanted image elements were never there. For example, this can be done using a machine learning or AI-based image inpainting method, such as resolution-robust large mask image inpainting (LaMa) using Fourier convolution, co-modulated Generative Adversarial Networks (CoModGAN), or Guided Language-to-Image Diffusion for Generation and Editing (GLIDE).

[0066] In the illustrated example, the image restoration technique has removed the unwanted image element 115, but has distorted the foreground image element 105 to produce a distorted foreground portion 150. As shown in the illustrated example, the neck, shoulders and back of the object forming the foreground image element 105 have been blurred and distorted in the area indicated as the distorted foreground portion 150. The technique also distorts the background image element 110 to produce a distorted background portion 145. As shown in the illustrated example, the color of the distorted background portion 145 does not match the surrounding area of ​​the background image element 110 because the foreground image element 105 has infiltrated into the background image element 110. The more the user selected area 125 overlaps the foreground image portion 105, the more these effects will increase. The user's intention will most likely be to keep the foreground image element 105 unchanged and fill the background image portion 110 more realistically.

[0067] Figure 2A Another example image 200 is shown having a foreground image element 205 and a background image element 210. In the example shown, the foreground image element 205 includes a person who is located near the camera that captured the image and is the main subject of the image, while the background image element 210 includes the walls of the room in which the person is located, serving as a background. The image 200 also includes an unwanted image element 215, which is a plurality of picture frames that are partially captured in the image behind the subject of the image. The unwanted image element 215 is directly adjacent to and in contact with the edge of the partial foreground image element 205.

[0068] Figure 2BAn image 220 is shown. Image 220 has the same foreground image element 205 and background image element 210 as image 200. The difference between image 200 and image 220 is that image 220 includes a user-selected area 225 applied over the unwanted image element 215. According to some embodiments, a user who wishes to edit image 200 may interact with a user interface component of a computing device displaying image 200 to indicate which area of ​​image 200 they consider to be an unwanted image element and wish to remove from the image. For example, a user may use a brush tool, a tracking tool, or other tool to digitally select, track, circle, or "paint" over the unwanted image element through a user interface to produce a user-selected area 225, which may be used as a mask to perform an image restoration method. However, a user may inadvertently select an area that they do not want to remove from image 200 as part of the user-selected area 225. For example, a user may inadvertently select an area of ​​the foreground image element 205. In the illustrated embodiment, the user-selected area 225 overlaps a portion of the foreground image element 205 corresponding to the hair and face of the subject.

[0069] Figure 2C Image 240 is shown. Image 240 has the same foreground image elements 205 and background image elements 210 as image 200. The difference between image 200 and image 240 is that image 240 has had an image restoration process performed on the image area selected by user selected area 225. In image 240, image restoration has been performed by a previously known image restoration technique, using an algorithm to fill in the area indicated by user selected area 225 so that the image appears as if the unwanted image elements were never there. For example, this can be done using a machine learning or AI-based image restoration method such as LaMa, CoModGAN, or GLIDE.

[0070] In the example shown, the image restoration technique has removed the unwanted image element 215, but has distorted the foreground image element 205 to produce a distorted foreground portion 250. As shown in the example shown, the face and hair of the subject forming the foreground image element 205 have been blurred and distorted in the area indicated as the distorted foreground portion 250. The technique also distorts the background image element 210 to produce a distorted background portion 245. As shown in the example shown, the background image element 210 has been blurred and distorted in the area indicated as the distorted background portion 245, and the color of the foreground image element 205 has bled into the distorted background portion 245. The more the user selected area 225 overlaps the foreground image portion 205, the more these effects will increase. The user's intention will most likely be to keep the foreground image element 205 unchanged and fill in the background image portion 210 more realistically.

[0071] Figures 1A to 1C and FIG. 2A to FIG. 2C The examples shown in illustrate some limitations of known image inpainting methods. In cases where unwanted image elements overlap, touch, or are close to foreground image elements, the area selected by the user for image inpainting may touch or overlap the foreground image elements, resulting in undesirable effects. These undesirable effects limit the usability of known image inpainting tools because the user needs to brush or trace close to the foreground image elements in order to completely cover or select the unwanted image elements.

[0072] The described embodiments relate to an image restoration method that preserves background and foreground image elements to avoid Figure 1C and Figure 2C The distortion shown in , even when areas of these image elements are selected. Figure 1D and Figure 2D 1 and 2 show the results of performing image restoration according to some described embodiments. As can be seen from these images, unwanted image elements 115 and 215 have been removed from regions 165 and 265, respectively, leaving foreground image elements 105 and 205 and background image elements 110 and 210 undistorted and preserved, thereby producing an image similar to Figure 1C and Figure 2C The effect is more realistic than the image shown in .

[0073] Figure 3 is a schematic diagram illustrating an example system 300 that may be used to perform an image restoration method according to some described embodiments.

[0074] The system 300 includes a user computing device 310 that can be used by a user who wishes to edit one or more images. Specifically, the user computing device 310 can be used by the user to perform image restoration on one or more images using the methods described below. In the illustrated embodiment, the system 300 also includes a server system 320. The user computing device 310 can communicate with the server system 320 via a network 330. However, in some embodiments, the user computing device 310 can be configured to independently perform the described methods without access to the network 330 or the server system 320.

[0075] For example, the user computing device 310 may be a computing device such as a personal computer, a laptop computer, a desktop computer, a tablet computer, or a smart phone. The user computing device 310 includes a processor 311 configured to read and execute program code. The processor 311 may include one or more data processors for executing instructions, and may include one or more of the following: a microprocessor, a microcontroller-based platform, a suitable integrated circuit, and one or more application-specific integrated circuits (ASICs).

[0076] The user computing device 310 also includes at least one memory 312. The memory 312 may include one or more memory storage locations, which may include volatile and non-volatile memory, and may be in the form of ROM, RAM, flash memory, or other memory types. The memory 312 may also include system memory, such as BIOS.

[0077] The memory 312 is arranged to be accessible by the processor 311 and is arranged to store data that can be read and written by the processor 311. The memory 312 may also contain program code 314 that can be executed by the processor 311 to enable the processor 311 to perform various functions. For example, the program code 314 may include an image editing application 315. The processor 321 executing the image editing application 315 may be caused to perform an image repair method, as described below with reference to Figures 4 to 13D described in further detail.

[0078] According to some embodiments, the image editing application 315 can be a standalone application in the form of a smartphone application or a desktop application that provides a set of image editing functions. In some embodiments, the image editing application 315 can be a web browser application (such as Chrome, Safari, Internet Explorer, Opera, or an alternative web browser application) that can be configured to access a web page that provides image editing functions through an appropriate uniform resource locator (URL).

[0079] Program code 314 may include Figure 3 Additional applications not shown in the figure may include operating system applications, which may be a mobile operating system if the user computing device 310 is a mobile device, a desktop operating system if the user computing device 310 is a desktop device, or an alternative operating system.

[0080] The user computing device 310 may also include user input and output peripherals 316. For example, these may include one or more of the following: a display screen, a touch screen display, a mouse, a keyboard, a speaker, a microphone, and a camera. The user I / O 316 may be used to receive data and instructions from a user and transmit information to a user.

[0081] The user computing device 310 also includes a communication interface 317 to facilitate communication between the user computing device 310 and other remote or external devices. The communication module 317 can allow wired or wireless communication between the user computing device 310 and the external device, and can utilize Wi-Fi, USB, Bluetooth, or other communication protocols. According to some embodiments, for example, the communication module 317 can facilitate communication between the user computing device 310 and the server system 320.

[0082] The network 330 may include one or more local or wide area networks that facilitate communication between elements of the system 300. For example, according to some embodiments, the network 330 may be the Internet. However, the network 330 may include at least a portion of any one or more networks having one or more nodes that send, receive, forward, generate, buffer, store, route, switch, process, or a combination thereof, one or more messages, packets, signals, some combination thereof, and the like. The network 330 may include, for example, one or more of the following: a wireless network, a wired network, the Internet, an intranet, a public network, a packet-switched network, a circuit-switched network, an ad hoc network, an infrastructure network, a public-switched telephone network (PSTN), a cable network, a cellular network, a satellite network, a fiber optic network, or some combination thereof.

[0083] The server system 320 may include one or more computing devices and / or server devices, such as one or more servers, databases, and / or processing devices communicating over a network, wherein the computing device hosts one or more applications, libraries, APIs, or other software elements. The components of the server system 300 may provide a server-side functional set to one or more client applications, such as the image editing application 315. The server-side functional set may include operations such as user account management, login, and content creation functions such as image editing, saving, publishing, and sharing functions. According to some embodiments, the server system 320 may include a cloud-based server system. Although a single server system 320 is shown, the server system 320 may include multiple systems of servers, databases, and / or processing devices. According to some described embodiments, the server system 320 may host one or more components of a platform for performing image restoration.

[0084] The server system 320 may include memory 322 and at least one processor 321. The processor 321 may include one or more data processors for executing instructions, and may include one or more of the following: a microprocessor, a microcontroller-based platform, a suitable integrated circuit, and one or more application-specific integrated circuits (ASICs). The memory 322 may include one or more memory storage locations, and may be in the form of ROM, RAM, flash memory, or other memory types.

[0085] Memory 322 is arranged to be accessible by processor 321 and contains data 323 that processor 321 is configured to read and write. Data 323 may store data such as user account data, image data, and data related to image editing tools (such as machine learning models trained to perform image editing functions).

[0086] The memory 322 also includes program code 324 that can be executed by the processor 321 to cause the processor 321 to perform a workflow. For example, the program code 324 includes a server application 325 that can be executed by the processor 321 to cause the server system 320 to perform a server-side function set. According to some embodiments, such as in the case where the image editing application 315 is a web browser, the server application 325 may include a web server, such as Apache, IIS, NGINX, GWS, or an alternative web server. In some embodiments, the server application 325 may include an application server that is specifically configured to interact with the image editing application 315. The server system 320 can be provided with both a web server module and an application server module.

[0087] The program code 324 may also include one or more code modules, such as one or more of a background separation module 318 and an image restoration module 319. As described in further detail below, executing the background separation module 318 may cause the processor 321 to perform a background separation process to separate the background and foreground of the input image. For example, this may be accomplished with a machine learning model or a pipeline consisting of a combination of a machine learning model and a traditional image processing technique. The background separation module may output an alpha channel in an area where the foreground is not completely opaque, as well as color information of the separated foreground elements. Executing the image restoration module 319 may cause the processor 321 to perform an image restoration process. For example, the processor 321 may be caused to perform a machine learning or AI-based image restoration method, such as LaMa, CoModGAN, or a diffusion model, such as a publicly available stable diffusion model. According to some embodiments, the processor 321 may be caused to perform an algorithm-based image restoration process, such as patch matching.

[0088] The background separation module 318 and / or the image restoration module 319 may be software modules, such as add-ons or plug-ins, that operate in conjunction with the server application 325 to extend its functionality set. In alternative embodiments, the modules 318 and / or 319 may be local to the server application 325. In further alternative embodiments, the modules 318 and / or 319 may be stand-alone applications (running on the server system 320 or an alternative server system) that communicate with the server application 325.

[0089] Although modules 318 and 319 have been described and shown as being part of / installed in server system 320, the set of functionality provided by modules 318 and 319 may alternatively be provided by user computing device 310, for example as an addition or extension to image editing application 315, as a separate stand-alone server application that communicates with image editing application 315, or as a local portion of image editing application 315.

[0090] The server system 320 also includes a communication interface 327 to facilitate communication between the server system 320 and other remote or external devices. The communication module 327 can allow for wired or wireless communication between the server system 320 and the external device, and can utilize Wi-Fi, USB, Bluetooth, or other communication protocols. According to some embodiments, for example, the communication module 327 can facilitate communication between the server system 320 and the user computing device 310.

[0091] Server system 320 may include additional functional components beyond those illustrated and described, such as one or more firewalls (and / or other network security components), load balancers (for managing access to server applications 325), and / or other components.

[0092] Figure 4 is a process flow diagram of a method 400 for performing image restoration according to some embodiments. In some embodiments, the method 400 may be performed at least in part by a processor 311 executing an image editing application 315. In some embodiments, the method 400 may be performed at least in part by a processor 321 executing a server application 325. Although certain steps of the method 400 have been described as being performed by specific elements of the system 300 (such as by the processor 311 or the processor 321), in some embodiments, these steps may be performed by different elements.

[0093] In step 405, processor 311 executing image editing application 315 receives an image selected by a user for editing. This may be the result of the user capturing an image for editing using a camera forming part of user I / O 316, or the result of the user selecting an image from a memory location. The memory location may be within data 313 stored locally in memory 312 on user computing device 310, or within data 323 stored remotely in memory 322 in server system 320. Depending on where the image editing process is to be performed, a copy of the retrieved image may be stored to a second memory location to allow processor 311 and / or processor 321 to efficiently access the image file. According to some embodiments, the selected image may be displayed within a user interface of image editing application 315, which may be displayed on a display screen forming part of user I / O 316. Image editing application 315 may display a plurality of editing tools for the user to select to perform image editing functions.

[0094] An example image that may be received in step 405 is Figure 1A and Figure 2A as shown above, and can be Figure 5A and Fig. 6A , as described in further detail below.

[0095] In step 410, the processor 311 executing the image editing application 315 may receive an indication that the user has selected an image inpainting tool to perform an image inpainting process on the received image. According to some embodiments, the selection of the image inpainting tool may cause the processor 311 to change the cursor or selection method to a brush style tool, thereby allowing the user to brush or paint on one or more areas of the image that they wish to remove. In some embodiments, the selection of the image inpainting tool may cause the processor 311 to change the cursor or selection method to a tracking style tool, thereby allowing the user to track the perimeter of or circle one or more areas of the image that they wish to remove.

[0096] In step 415, the processor 311 executing the image editing application 315 receives a user input corresponding to a selected area of ​​the image that the user wants to remove using the selected image restoration tool (referred to as the user selected area).

[0097] An example image of the user selected area that may be received in step 415 is shown in FIG. Figure 1B and Figure 2B as shown above, and can be Figure 5B , as described in further detail below.

[0098] In step 420, the processor 311 executing the image editing application 315 is caused to process the image to separate the background of the image from the foreground of the image. In some embodiments, the processor 311 performs step 420 by transmitting the image to be processed to the server system 320 and causing the processor 321 to execute the background separation module 318. According to some embodiments, this may cause the processor 321 to perform a background removal process on the image. According to some embodiments, this may cause the processor 321 to perform a foreground removal process on the image. In some embodiments, the processor 321 may further be caused to generate a foreground mask and / or a background mask based on the result of the separation process.

[0099] Example image with background removal applied Figure 6B As described in further detail below. An example image showing a foreground mask is shown in Figure 5C and Figure 6C , as described in further detail below.

[0100] In step 425, the processor 311 executing the image editing application 315 is caused to determine the amount of overlap between the user selected area received in step 415 and the foreground and / or background identified in step 420. According to some embodiments, the processor 311 may calculate the relative mask overlap (RMO) of the user selected area (US_area) and the foreground area (FG_area) based on the following equation:

[0101]

[0102] The numerator of the fraction calculates the intersection of the pixels of US_area and FG_area. The denominator calculates the number of pixels of US_area. Additionally or alternatively, the processor 311 may calculate the relative mask overlap (RMO) of the user selected area (US_area) and the background area (BG_area).

[0103] An example image showing the overlap between the user selected area and the foreground area is shown in Figure 5D and Fig.7D , as described in further detail below.

[0104] In step 430, the processor 311 executing the image editing application 315 is caused to compare the determined value of the RMO with one or more thresholds to determine which portions of the image the user intends to perform image repair on. According to some embodiments, the processor 311 may compare the RMO with a lower threshold. In some embodiments, the lower threshold may be 0%. According to some embodiments, the processor 311 may compare the RMO with an upper threshold. The upper threshold may be between 20% and 90%. In some embodiments, the upper threshold may be between 40% and 60%. For example, in some embodiments, the upper threshold may be 50%. In some embodiments, the upper threshold may be calculated by the processor 311 based on the size of the user selected area or the size of the user selected area compared to the accuracy of the selection tool used by the user to select the user selected area. In the case where the user selected area is small, especially when compared to the tool size, the accuracy of the selection may be low, and the upper threshold may be selected as a higher percentage because it is more difficult for the user to accurately select a small area selected for the image. In some embodiments, the threshold may be simply retrieved from the data 313 or the data 323.

[0105] In the case where RMO is less than or equal to the lower threshold, this indicates that the user selected area is completely in the background area of ​​the image and the user intends to remove the area of ​​the background. The processor 311 executing the image editing application 315 proceeds to step 435 to perform an image restoration process on the image, which can be performed by the processor 311, which transmits the image to be restored to the server system 320 and causes the processor 321 to execute the image restoration module 319. This can cause the processor 321 to perform the image restoration process on the user selected area of ​​the image.

[0106] In the case where RMO is greater than or equal to the upper threshold, this indicates that the user selected area is mostly in the foreground area of ​​the image, and the user intends to remove the foreground area. According to some embodiments, the processor 311 can determine whether the foreground area is completely surrounded by the user selected area. If so, the user may intend to remove the foreground area, and the processor 311 executing the image editing application 315 can proceed to step 435 to perform or cause the server system 320 to perform image restoration processing on the user selected area of ​​the image. Alternatively, in the case where the user selected area is mostly on the foreground area but does not cover the entire foreground area, this can indicate that the user intends to perform image restoration on a portion of the foreground area, and the processor 311 can continue to execute or cause the server system 320 to perform image restoration processing only on the foreground area before combining the foreground area that has been image restored with the original background area.

[0107] In the case where RMO is greater than the lower threshold but less than the upper threshold, this may indicate that the user wants to remove something in the background area of ​​the image but happened to be slightly brushed or traced over the foreground area. In this case, processor 311 may activate a foreground protection algorithm, as described below with reference to steps 440 to 470.

[0108] In step 440, the processor 311 executing the image editing application 315 is caused to calculate the union of the user selected area received in step 415 and the foreground area determined in step 420. A new mask, also called a merged mask, is created based on the result of the union.

[0109] An example image showing the union of the user selected region and the foreground region forming the merge mask is shown in Figure 5E and Figure 8C , as described in further detail below.

[0110] In optional step 445, the processor 311 executing the image editing application 315 is caused to expand the merge mask to ensure that any image elements that form part of the foreground area or the user-selected area have been adequately captured. According to some embodiments, the merge mask can be expanded by a predetermined amount, such as a predetermined number of pixels, which can be retrieved from a memory location (such as from data 313 or data 323). For example, the merge mask can be expanded by between 1 and 10 pixels. In some embodiments, the merge mask can be expanded by between 3 and 7 pixels. In some embodiments, the merge mask can be expanded by 5 pixels. In some alternative embodiments, the number of pixels used to expand the image can be calculated based on the size of the image and can be a percentage of the number of pixels in the image.

[0111] An example image showing the enlarged merged mask is shown in Fig. 5F and Fig.8D , as described in further detail below.

[0112] In step 450, the processor 311 executing the image editing application 315 is caused to perform image restoration on the area of ​​the original image received in step 405 that is spatially aligned with the enlarged merge mask determined in step 445 (or the merge mask determined in step 440). Step 450 can be performed by the processor 311, which transmits the image to be image restored to the server system 320 and causes the processor 321 to execute the image restoration module 319. This can cause the processor 321 to perform the image restoration process on the mask area. The result is a relatively realistic filling of the mask area. In an ideal case, the image looks realistic, as if the mask area of ​​the image has never appeared.

[0113] Example images showing image inpainting with merged masks are shown in Figure 5G and Fig. 9C , as described in further detail below.

[0114] In optional step 455, processor 311 executing image editing application 315 is caused to perform a second inpainting step by inpainting an area of ​​the inpainted image generated in step 450 that is associated with the user-selected area received in step 415. Since the user-selected area is the area of ​​the image from which the unwanted image elements are removed, performing step 455 may produce a higher resolution and more realistic fill for that area. This is because the inpainting process generally produces higher resolution and realistic results when the size of the area being inpainted is small compared to inpainting a larger area. Step 455 may be performed by processor 311 transmitting the image to be inpainted to server system 320 and causing processor 321 to execute inpainting module 319. This may cause processor 321 to perform the inpainting process on the masked area.

[0115] In step 460, the processor 311 executing the image editing application 315 generates a downscaling mask that is the user selected area received in step 415 minus the foreground area calculated in step 420. In other words, the portion of the user selected area that overlaps the foreground area is removed from the user selected area to form the downscaling mask.

[0116] An example image representing a downscaling mask is Fig.5I and Fig.11D , as described in further detail below.

[0117] In step 465, the processor 311 executing the image editing application 315 is caused to combine the original image received in step 405 with the inpainted image generated in step 450 or step 455 using the reduction mask. Specifically, the area of ​​the original image corresponding to the reduction mask is replaced with the inpainted image. This allows the background area of ​​the original image selected by the user in step 415 to be filled with the inpainted image, while the foreground image remains unchanged.

[0118] An example image representing a combined image is shown in Figure 5K and Fig.12D , as described in further detail below.

[0119] In optional step 470, the processor 311 executing the image editing application 315 is caused to perform smoothing on the combined image generated in step 465. A more realistic result can be achieved if the transition between the original image and the inpainted image fades in at the edge of the reduction mask. This can be achieved by performing an alpha composite of the foreground area overlapping the user selected area, so that a few pixels at the border of the mask are taken from the foreground area and added to the combined image as generated in step 465. This can produce a smoother transition. Although steps 465 and 470 are shown in separate boxes, according to some embodiments, smoothing can be performed when the images are combined in step 465, rather than as an additional step 470.

[0120] Example images showing alpha compositing between the combined image and the foreground area are shown in Fig.13D , as described in further detail below.

[0121] FIG. 5A to FIG. 5K is a diagrammatic representation of the stages of performing method 400 on an image.

[0122] Figure 5A An example image 500 is shown, which may be an image received by the processor 311 executing the image editing application 315 in step 405 of the method 400, as described above. The image 500 includes a foreground element 501, which is a person who is the subject of the captured image and appears closest to the viewer. The image 500 includes background elements 502, which include a brick wall and some other people. The image 500 also includes an unwanted image element 503, which is a person who appears behind the foreground element 501 and is not the intended subject of the image.

[0123] Figure 5B An example image 505 is shown, which may be an image displayed to a user via user I / O 316 after performing step 415 of method 400 as described above. As step 415, the user selects an area of ​​the original image 500 that they want removed, which is displayed as user selected area 506 along with elements of the previously displayed image 500, such as foreground element 501 and background element 502. In this case, user selected area 506 overlaps with unwanted image element 503. However, user selected area 506 also partially overlaps with foreground image element 501.

[0124] Figure 5CAn example image 510 is shown, which shows the result of the processor 311 executing the image editing application 315 to perform step 420, as described above. In step 420, a background separation step is performed to separate the original image 500 into a foreground area 511 and a background area 512.

[0125] Figure 5D An example image 515 is shown, which shows the degree of overlap of the foreground area 511 with the user selected area 506, which can be calculated by the processor 311 executing the image editing application 315 to perform step 425, as described above. As described in further detail above, in step 425, the processor 311 is caused to calculate the degree of overlap between the foreground area 511 and the user selected area 506 to determine the type of image restoration processing to be performed on the image 500. With respect to the illustrated embodiment, it is assumed that the amount of overlap is between the lower threshold amount and the upper threshold amount, which means that the processor 311 determines that the foreground protection algorithm should be performed, as described above with respect to steps 440 to 470.

[0126] Figure 5E An example image 520 is shown, which shows the result of executing the image editing application 315 to perform step 440, as described above. In step 440, the processor 311 generates a merge mask 521, which is a union of the foreground area 511 and the user selected area 506, as shown in FIG. Figure 5D shown.

[0127] Fig. 5F An example image 525 is shown, which shows the result of executing the image editing application 315 to perform step 445, as described above. In step 445, the processor 311 generates an enlarged merge mask 526 by enlarging or expanding the merge mask, such as Figure 5E shown.

[0128] Figure 5G An example image 530 is shown, which shows the result of executing the image editing application 315 to perform step 450, as described above. In step 450, the processor 311 performs image restoration on the original image 500 in the area indicated by the enlarged merge mask 526. The resulting image 530 includes background elements 502 as in the original image 500. However, the foreground elements 501 and the unwanted image elements 503 have been image restored to produce the image restored area 531. In other words, the foreground elements 501 and the unwanted image elements 503 have been replaced or filled with the background image generated by the processor 311 to create a realistic image that creates the impression that the foreground elements 501 and the unwanted image elements 503 were never in the image.

[0129] Figure 5H An example image 535 is shown showing the intersection between the foreground region 511 and the user selected region 506, which can be used by the processor 311 executing the image editing application 315 to perform step 460, as described above. In step 460, the processor 311 generates a reduced mask that is the user selected region 506 with the foreground region 511 reduced.

[0130] Fig.5I An example image 540 is shown, which shows the result of the processor 311 executing the image editing application 315 to perform step 460, as described above. In step 460, as Figure 5H As shown, the processor 311 generates a reduced mask 536 by subtracting the foreground area 511 from the user-selected area 506 .

[0131] Figure 5J An example image 545 is shown showing a reduction mask 536 superimposed on the inpainted image 530, which can be used by the processor 311 executing the image editing application 315 to perform step 465, as described above. In step 465, the processor 311 combines the original image 500 with the inpainted image 530 in the area defined by the reduction mask 536. In some embodiments, this can be done by copying the area of ​​the image 540 that overlaps the reduction mask 536 and pasting it onto the same area of ​​the original image 500.

[0132] Figure 5K An example image 550 is shown, which shows the result of the processor 311 executing the image editing application 315 to perform step 465, as described above. In step 465, the processor 311 combines the original image 500 with the inpainted image 530 to produce a final image 545, wherein the unwanted image elements 503 are removed to produce the inpainted area 546, while the foreground elements 501 are retained.

[0133] FIG. 6A to FIG. 6C A series of example images are shown which illustrate the results of processor 311 executing image editing application 315 to perform step 420 (step 420 is the background separation step), as described above.

[0134] Fig. 6AAn example image 600 is shown, which may be an image received by processor 311 executing image editing application 315 in step 405 of method 400, as described above. Image 600 includes foreground element 605, which is a person who is the subject of the captured image and appears closest to the viewer. Image 600 includes background element 610, which includes the floor and other people. Image 600 also includes unwanted image element 615, which is a person who appears behind foreground element 605 and is not the intended subject of the image.

[0135] Figure 6B An example image 630 is shown, which represents the result of processor 311 executing image editing application 315 to perform a background removal step on image 600, which may be accomplished by processor 311 transmitting the image to be processed to server system 320 and causing processor 321 to execute background separation module 318. In image 630, foreground elements 605 have been retained, but background elements 610, including unwanted image elements 615, have been removed, leaving only background area 635.

[0136] Figure 6C An example image 660 is shown, which represents the result of processor 311 executing image editing application 315 to perform a foreground removal step on image 630, which may be accomplished by processor 311 transmitting the image to be processed to server system 320 and causing processor 321 to execute background separation module 318. In image 660, foreground elements 605 have been removed, leaving only foreground region 665 and background region 670. Foreground region 665 and background region 670 may be used as masks by processor 311 in further processing steps.

[0137] FIG. 7A to FIG. 7D A series of example images are shown which illustrate the results of processor 311 executing image editing application 315 to perform step 425 (step 425 is the relative mask overlap calculation step), as described above.

[0138] Fig. 7A An original image 600 is shown, as described above.

[0139] Figure 7B An example image 700 is shown having a user selected area 710 that is selected by a user of the user computing device 310 as an area of ​​the original image 600 that they wish to remove. The user selected area 710 corresponds to the unwanted image element 615. However, the user selected area 710 also overlaps the foreground element 605.

[0140] Figure 7CAn example image 660 is shown, which is divided into a foreground region 665 and a background region 670, as described above.

[0141] Fig.7D An image 750 is shown, which shows a superimposed Figure 7C The foreground area 665 Figure 7B The processor 311 may use this image to perform step 425 to determine the amount of overlap 760 between the user selected area 710 and the foreground area 665 .

[0142] FIG. 8A to FIG. 8D A series of example images are shown showing the results of processor 311 executing image editing application 315 to perform steps 440 and 445 (steps 440 and 445 are the merged mask generation step and the enlarged merged mask generation step as described above).

[0143] Fig. 8A An image 700 is shown showing a user selected area 710, as described above.

[0144] Figure 8B A separated image 660 is shown, which is separated into a foreground region 665 and a background region 670, as described above.

[0145] Figure 8C 8 shows an image 800 showing a merge mask 810 generated by processor 311 executing image editing application 315 to perform step 440, as described above. In step 440, processor 311 calculates Fig. 8A The user selected area 710 shown and Figure 8B The union of the foreground regions 665 is shown to generate a merged mask 810.

[0146] Fig.8D 850 is shown, and the image 850 shows the enlarged merge mask 860, which is generated by the processor 311 executing the image editing application 315 to perform step 440, as described above. In step 445, the processor 311 Figure 8C The merge mask 810 is dilated as shown in to generate the dilated merge mask 860.

[0147] 9A to 9C A series of example images are shown which illustrate the results of processor 311 executing image editing application 315 to perform step 450 (step 450 is the image inpainting step of merging masks as described above).

[0148] Fig. 9AAn original image 600 is shown, as described above.

[0149] Fig. 9B An image 850 is shown showing an enlarged merge mask 860 , as described above.

[0150] Fig. 9C 900 is shown, and image 900 is generated by processor 311 executing image editing application 315 to perform step 450, as described above. In step 450, processor 311 generates an image 900 by executing image editing application 315. Fig. 9B The enlarged merge mask 860 shown is shown to be performed in the area indicated by Fig. 9A Inpainting of image 600 is shown. This produces an inpainted image 900, in which foreground elements 605 and unwanted image elements 615 have been inpainted to produce inpainted regions 910. Background elements 610 remain unchanged.

[0151] FIG. 10A to FIG. 10C A series of example images are shown which illustrate the results of processor 311 executing image editing application 315 to perform step 455 (step 455 is the image inpainting step of the user selected mask as described above).

[0152] Fig. 10A An inpainted image 900 is shown, as described above.

[0153] Fig. 10B An image 700 is shown showing a user selected area 710, as described above.

[0154] Fig. 10C 1000 is shown, and image 1000 is generated by processor 311 executing image editing application 315 to perform step 455, as described above. In step 455, processor 311 generates an image 1000 by executing image editing application 315. Fig. 10B The following is performed in the area indicated by the user selected area 710 shown. Fig. 10A This produces a further inpainted image 1000, wherein the area corresponding to the user selected area 710 has been inpainted again to produce a further inpainted area 1010. As described above, this second inpainting step may produce a higher resolution and / or realistic result in the image area corresponding to the user selected area 710.

[0155] FIG. 11A to FIG. 11D A series of example images are shown which illustrate the results of processor 311 executing image editing application 315 to perform step 460 (step 460 being the downscaling mask generation step as described above).

[0156] Fig.11A An original image 600 is shown, as described above.

[0157] Fig. 11B An image 700 is shown showing a user selected area 710, as described above.

[0158] Fig. 11C A separated image 660 is shown, separated into a foreground region 665 and a background region 670, as described above.

[0159] Fig.11D 1100 is shown, and image 1100 is generated by processor 311 executing image editing application 315 to perform step 460, as described above. In step 460, processor 311 generates a Fig. 11B The user selected area 665 shown is minus the Fig. 11C The foreground area 710 is shown to generate a reduced mask 1110.

[0160] FIG. 12A to FIG. 12D A series of example images are shown which illustrate the results of processor 311 executing image editing application 315 to perform step 465 (step 465 is the image inpainting step of reducing the mask as described above).

[0161] Fig. 12A An original image 600 is shown, as described above.

[0162] Fig. 12B An image 1100 is shown showing a reduced mask 1110 , as described above.

[0163] Fig. 12C An inpainted image 1000 is shown, as described above.

[0164] Fig.12D 1200 is shown, and image 1200 is generated by processor 311 executing image editing application 315 to perform step 465, as described above. In step 465, processor 311 uses Fig. 12B The reduced mask 1110 shown in FIG. Fig. 12A The original image 600 shown in FIG. Fig. 12C 1000. Specifically, the region of the original image 600 corresponding to the user-selected region 710 is replaced by the corresponding region of the inpainted image 1000. The resulting image 1200 includes the background elements 610 and the foreground elements 605 of the image 600, but the unwanted image elements 615 have been replaced by the inpainted region 1010 from the image 1000.

[0165] FIG. 13A to FIG. 13D A series of example images are shown which illustrate the results of processor 311 executing image editing application 315 to perform step 470 (step 470 is the smooth image merging step described above).

[0166] Fig.13A An original image 600 is shown, as described above.

[0167] Fig. 13B A separated image 660 is shown, which is separated into a foreground region 665 and a background region 670, as described above.

[0168] Fig. 13C A combined image 1200 as described above is shown.

[0169] Fig.13D 1300 is shown, and image 1300 is generated by processor 311 executing image editing application 315 to perform step 470, as described above. In step 470, processor 311 performs a Fig.13A The alpha synthesis of the foreground element 605 is Fig. 13C The combined image 1200 shown is smoothed by gradually smoothing the foreground region 665 and the background region 670. Fig.13A The pixels of the foreground element 605 are added to the combined image 13C, which can produce a smoother and more realistic image. The resulting image 1300 has a smoother and more realistic transition 1305 between the foreground element 605 and the image repair area 1010. Figure 4 As described above, the smoothing process performed in step 470 may be performed simultaneously with the process of combining the original image 600 and the inpainted image 1000. As a result, when the smoothing process is performed, the image 1300 may be directly generated by combining the original image 600 with the inpainted image 1000 using an alpha compositing process, rather than first generating the image 1200.

[0170] Those skilled in the art will appreciate that numerous changes and / or modifications may be made to the above embodiments without departing from the broad scope of the present disclosure. Therefore, the present embodiments are considered in all aspects to be illustrative and not restrictive.

Claims

1. A computer-implemented method for performing image restoration, the method comprising: accessing a first image; receiving a selected area of ​​the first image; identifying a foreground area of ​​the first image; generating a merge mask based on a union of the user-selected region and the foreground region; Performing an image restoration process on a region of the first image corresponding to the merge mask to generate a second image, wherein the second image is an image restored; generating a reduced mask based on the user-selected area minus the foreground area; as well as The first image is combined with a region of the second image corresponding to the downscaling mask to produce an output image.

2. The method of claim 1, further comprising storing the output image to a memory location. The method of claim 1 , further comprising transmitting the output image to an external device.

4. The method according to any one of claims 1 to 3, further comprising: Calculating a relative mask overlap between the user-selected area and the foreground area; comparing the relative mask overlap to at least one threshold; as well as The following steps are continued: generating a merge mask, performing an image inpainting process, generating a downscaling mask and combining the images based on the result of the comparison, wherein these steps form part of a foreground protection process.

5. The method according to claim 4, wherein: The relative mask overlap is calculated using the following equation: Wherein, RMO is the relative mask overlap, Num_pixels is the function to obtain the number of pixels, US_area is the user selected area, and FG_area is the foreground area.

6. The method according to claim 4 or claim 5, wherein: Comparing the relative mask overlap to at least one threshold includes comparing the relative mask overlap to a lower threshold, and performing the foreground protection process based on determining that the relative mask overlap is greater than the lower threshold.

7. The method according to claim 6, wherein: The lower threshold is 0%.

8. The method according to claim 6 or claim 7, wherein: Based on determining that the relative mask overlap is less than or equal to the lower threshold, instead of executing the foreground protection algorithm, an image restoration process is performed on the area of ​​the first image corresponding to the user-selected area to generate the output image.

9. The method according to any one of claims 4 to 8, wherein: Comparing the relative mask overlap to at least one threshold includes comparing the relative mask overlap to an upper threshold, and performing the foreground protection process based on determining that the relative mask overlap is less than the upper threshold.

10. The method according to claim 9, wherein: The upper threshold is between 10% and 90%.

11. The method according to claim 9 or claim 10, wherein: The upper threshold is between 40% and 70%.

12. The method according to any one of claims 9 to 11, wherein: The upper threshold is 50%.

13. The method according to any one of claims 9 to 12, wherein: Based on determining that the relative mask overlap is greater than or equal to the upper threshold, instead of performing the foreground protection process, an image restoration process is performed on the area of ​​the first image corresponding to the user-selected area to generate the output image.

14. The method according to any one of claims 1 to 13, further comprising the following steps: The merge mask is expanded, and the image restoration process is performed on a region of the first image corresponding to the expanded merge mask.

15. The method according to claim 14, wherein: The merge mask is dilated by between 1 and 9 pixels.

16. The method according to claim 14 or claim 15, wherein: The merge mask is dilated by between 3 and 7 pixels.

17. The method according to any one of claims 14 to 16, wherein: Dilate the merge mask by 5 pixels.

18. The method according to any one of claims 1 to 17, further comprising performing a second image restoration process on an area of ​​the first image corresponding to the user-selected area to generate the second image.

19. The method according to any one of claims 1 to 18, further comprising, before outputting the image, performing a smoothing process on the output image to smooth an edge between the first image and the second image.

20. The method according to claim 19, wherein: The smoothing is performed using an alpha compositing process.

21. The method according to any one of claims 1 to 20, wherein: Performing the image restoration process includes performing an AI-based image restoration process.

22. The method according to claim 21, wherein: Performing the image restoration process includes performing a LaMa image restoration process.

23. The method according to any one of claims 1 to 22, wherein: Identifying a foreground region of the first image includes performing a background subtraction process on the first image.

24. A non-transitory computer-readable storage medium storing instructions which, when executed by a processing device, cause the processing device to perform the method according to any one of claims 1 to 23.

25. A computing device comprising: The non-transitory computer-readable storage medium of claim 24; as well as A processor configured to execute the instructions stored in the non-transitory computer-readable storage medium.