Bounds-aware object removal and content padding

Abstract

Embodiments of the present disclosure relate to boundary-aware object removal and content filling. Systems and methods for removing objects from an image are disclosed. An image processing application identifies a boundary of each object in a set of objects in an image. In some cases, the identification uses deep learning. The image processing application identifies a complete boundary for each object in the set of objects by providing the objects to a trained model. The image processing application determines a set of masks. Each mask corresponds to an object in the set of objects and represents an image region defined by an intersection of a boundary of the object and a boundary of a target object that is to be removed from the image. The image processing application updates each mask by performing content filling on the corresponding regions, respectively. The image processing application creates an output image by merging each updated mask with a portion of the image.

Description

Technical Field

[0001] The present disclosure generally relates to removing objects from digital images. More specifically, but not by way of limitation, the present disclosure relates to removing objects from images by using object boundary detection in combination with content-aware fill. Background Art

[0002] During editing of an image such as a photograph, a graphic designer may wish to remove a target object. For example, a designer may want to remove a toy that appears in the middle of a garden photo, or remove a yard sign from a house photo.

[0003] However, existing object removal software tools have drawbacks. For example, some solutions can only remove objects that appear in front of a simple background, because such solutions simply remove the unwanted area and automatically fill the resulting holes with samples from other parts of the image.

[0004] Other solutions improve upon this approach by more thoroughly analyzing the area around the hole to determine an appropriate fill for the missing content. However, such solutions fail to determine the boundaries of objects occluded by the target object, resulting in inaccurate fills of content. For example, such solutions cannot accurately recreate leaves occluded by the ball because the boundaries of the leaves cannot be analyzed and the missing portions of the boundaries recreated. Therefore, such solutions will fill in the missing portions of the leaves in a way that the leaves do not look realistic.

[0005] Therefore, a solution is needed to reliably and accurately remove and fill objects. Summary of the invention

[0006] Systems and methods for removing objects from an image are disclosed. An exemplary method accesses an image including a set of objects. The method identifies a complete boundary for each object in the set of objects by providing the objects to a trained model. The method receives a selection of a target object to be removed from the objects. The method also determines a set of masks. Each mask corresponds to an object in the set of objects and represents an image region defined by an intersection of a boundary of the object and a boundary of the target object. The method updates each mask by performing content filling on the corresponding regions, respectively. The method creates an output image by merging each updated mask with a portion of the image.

[0007] These illustrative embodiments are not mentioned to limit or define the present disclosure, but to provide examples to aid in understanding the present disclosure.Additional embodiments are discussed in the Detailed Description, and further description is provided there. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The features, embodiments and advantages of the present disclosure may be better understood when the following detailed description is read with reference to the accompanying drawings.

[0009] Figure 1 is a diagram depicting an example of an application component for removing an object from an image according to an embodiment of the present disclosure;

[0010] Figure 2 An example of a system implementation for removing an object from an image according to an embodiment of the present disclosure is depicted;

[0011] Figure 3 An example of a process for removing an object from an image according to an embodiment of the present disclosure is depicted;

[0012] Figure 4 Depicting examples of intermediate and final results of an object removal environment according to an embodiment of the present disclosure;

[0013] Figure 5 An example of a process for training a conditional generative adversarial network to detect boundaries of an object according to an embodiment of the present disclosure is depicted;

[0014] Figure 6 Describes an embodiment of the present disclosure. Figure 5 Examples of training data used in ; and

[0015] Figure 7 An example of a computing system for removing objects from an image in accordance with an embodiment of the present disclosure is depicted. DETAILED DESCRIPTION

[0016] As discussed above, existing solutions for removing objects have drawbacks that include unrealistic content filling of the removed objects. In contrast, the embodiments described herein facilitate more accurate object removal by using deep learning to detect the boundaries of objects present in the input image, finding the intersections between those objects, and applying content filling to each intersection. Thus, the disclosed solution creates a more aesthetically pleasing result because the boundaries of the underlying objects are recreated and the removed objects are appropriately filled.

[0017] For discussion purposes, the following non-limiting example is introduced. An image processing application executing on a computing device accesses a photographic image. The image depicts a garden table on a lawn and a wine bottle on the table. The upper region of the wine bottle obscures the lawn, and the lower region of the wine bottle obscures the table. The application receives input from a graphic designer that the wine bottle should be removed from the image.

[0018] The image processing application identifies objects within the image, specifically the table, the lawn, and the wine bottle. The image processing application then identifies the boundaries or outlines of the detected objects. In some cases, deep learning such as conditional generative adversarial networks is used. Continuing with the example, the application determines a first region or mask corresponding to the intersection or overlap between the bottle and the lawn and a second region or mask corresponding to the overlap between the bottle and the table.

[0019] With each region detected, the image processing application fills or completes the mask, for example, using content-aware fill based on the content of the object on which the shape is based. Content-aware fill involves recreating appropriate content based on the existing content of the object to fill the empty spaces of the mask. For example, to fill the lawn area previously covered by the upper portion of the wine bottle, the image processing application uses content from the rest of the lawn to intelligently complete the mask. The application fills the first mask without considering the second mask, and vice versa. Once both masks are filled, the application combines the regions into an output image, which is the original image without the wine bottle.

[0020] The disclosed solution provides technical improvements in the field of image object removal. For example, certain embodiments use an improved method for generating training data for a conditional generative adversarial network, and then use the trained network to identify the boundaries of occluded objects where complete boundaries are not available. More specifically, an image processing application generates a robust training data set that includes images depicting objects with known boundaries, as well as modified images whose boundaries have been artificially occluded. The conditional generative adversarial network is then trained with this data set. By using the network, certain embodiments can complete the missing boundaries of an object. In addition, by applying content-aware fill to regions from different overlapping objects separately, an embodiment provides more realistic object removal than previous solutions.

[0021] In addition, the disclosed solution provides a technical improvement over the following solutions: predicting segmentation labels (e.g., the type of objects present in the image) but failing to predict the boundaries of each object in the image and determine each intersection between the object and the target object. Instead, the disclosed solution predicts segmentation labels for missing areas, thereby generating information about the predicted object locations and shape details of the target object. The disclosed solution can combine the segmentation mask with the input image to make a complete prediction. By guiding the segmentation process in this way, the disclosed solution combines the strengths of deep generative models with segmentation information, resulting in more realistic predictions, especially for boundaries between different objects. In addition, compared to previous solutions that are only able to make a single prediction given an input image, the disclosed solution provides interactive and multimodal predictions.

[0022] As disclosed herein, an "image" refers to an electronic representation of a photograph, image, text, user interface element, etc. Examples of images include document, Document or Document. An image can consist of one or more objects.

[0023] As disclosed herein, an "object" refers to an independent component of an image. Example objects include shapes, letters, text boxes, background objects, etc. Each object has a boundary.

[0024] As used herein, the "border" or "outline" of an object refers to the outer perimeter around the object. For example, the border around an orange may be circular or nearly circular. The borders of more complex objects may be non-linear, such as is the case with leaves or animals.

[0025] As used herein, a "region" or "mask" refers to a data structure that identifies or specifies certain pixels of an image for a particular purpose. For example, a mask may identify pixels in a hole to be filled.

[0026] Turning now to the accompanying drawings, Figure 1 is a diagram depicting an example of application components for removing objects from an image according to an embodiment of the present disclosure. Figure 1 An image processing environment 100 is depicted, which includes an input image 110, a computing system 101, and an output image 140. The computing system 101 executes an image processing application 102 and uses a machine learning model 105. An example of the computing system 101 is shown in FIG. Figure 6 is shown in more detail in .

[0027] Input image 110 includes first object 112 (square) and second object 113 (ellipse). It can be seen that object 113 partially occludes object 112 in the lower right corner of object 112. Because input image 110 can be a digitized photo or a rendered image containing pixels, image processing application 102 determines object 112 or 113 and its boundaries. Then, image processing application 102 receives a designation of object 113 as a target object. In turn, image processing application 102 determines the boundaries of objects 112 and 113, determines the intersection between objects 112 and 113, and applies content fill to the intersection. It can be seen that output image 140 shows object 142 corresponding to object 112.

[0028] For an input image with more objects, the image processing application 102 identifies more masks and applies content filling to the corresponding object intersections, respectively. The image processing application 102 then merges the intersections or masks into an output image 140. Figure 2Another example including more objects is depicted.

[0029] In some cases, the image processing application 102 can use the machine learning model 105 to perform one or more tasks. For example, as depicted, the machine learning model 105 is a conditional generative adversarial model that includes a generative model 106 and a discriminative model 107. The image processing application 102 can train the generative model 106 and the discriminative model 107 by using training data 108. Figure 5 An exemplary training process is discussed, but other processes may be used.

[0030] However, different machine learning models may also be present in the image processing environment 100. Other examples of suitable models employed by the machine learning model 105 are prediction models, classification models, generative models, and discriminative models. In addition, the image processing application 102 may also use additional models, for example, to perform different tasks. For example, the image processing application 102 may use a first machine learning model to perform object detection and a second machine learning model to perform object boundary detection. Similarly, machine learning may be used to perform content-aware fill.

[0031] Figure 2 An example of a system implementation for removing objects from an image according to an embodiment of the present disclosure is depicted. Figure 2 An object removal environment 200 is depicted that includes an object detection module 250, an object boundary module 252, an object intersection module 253, and a content fill module 254. These modules are separated for purposes of example; the functionality described herein may be implemented across any number of modules included in the image processing application 102. Additionally, one or more modules may use machine learning capabilities.

[0032] The object removal environment 200 also depicts the input and output of various stages. More specifically, the object removal environment 200 includes an input image 211, a mask 260, a mask 261, and an output image 240. In some cases, the image processing application 102 may output intermediate steps for debugging or for illustrative purposes.

[0033] Object detection module 250 receives input image 211, which includes object 212, object 213, and object 214. Object 213 is designated as a target object or an object to be removed from input image 211. It can be seen that object 213 intersects object 212 and object 214. Object boundary module 252 detects the boundary of objects 212 and 213. For example, two objects are shown. In the case where input image 211 includes more than two objects, object boundary module 252 determines the boundary for each additional object.

[0034] The object intersection module 253 determines whether the object 212 or the object 214 intersects with the object 213. As depicted in the input image 211, both the object 212 and the object 214 intersect with the object 213. The object intersection module 253 determines the intersection of the objects by analyzing the vectors representing the objects 212-214. The object intersection module 253 outputs a mask 260 and a mask 261, where the mask 260 represents the intersection between the object 212 and the object 213, and the mask 261 represents the intersection between 214 and 213. In the case where the input image 211 includes more than two objects intersecting with the target object, additional intersections can be determined.

[0035] Content fill module 254 receives masks 261 and 262 and performs filling of the region. As discussed further, different filling methods, such as content-aware filling, can be used. It can be seen that output image 240 includes object 242 corresponding to object 212 and object 243 corresponding to object 214. Objects 242 and 243 are filled and appear roughly similar to if the object was drawn without object 213 in the front.

[0036] Figure 3 An example of a process 300 for removing an object from an image according to an embodiment of the present disclosure is depicted. The image processing application 102 may implement the process 300, but other software applications and computing systems may also implement the process 300. For example, Figure 4 discuss Figure 3 .

[0037] At block 301, process 300 involves accessing an image including an object. Image processing application 102 accesses an input image. Image processing application 102 may receive the input image from an external device or file. Alternatively, image processing application 102 may create the input image in response to a user drawing command.

[0038] Figure 4 Depicted are examples of intermediate and final results of an object removal environment according to an embodiment of the present disclosure. Figure 4 Depicted are an input image 401 , intersection regions 402 - 403 , filled images 404 - 405 , and an output image 406 . At block 301 , the image processing application 102 accesses the input image 401 .

[0039] As can be seen, input image 401 depicts a swimming pool 410 located on grass 411, with wooden object 412 obscuring a portion of swimming pool 410 and a portion of grass 411. Image processing application 102 uses object detection module 250 to identify swimming pool 410 and grass 411. Different methods can be used for object identification. For example, machine learning model 105 is a trained classification model that can detect the presence of an image based on a training set of known images (e.g., objects, plants, animals, faces, cars, etc.). Additional methods for object identification include template matching, linear discriminant analysis (LDA), support vector machines (SVM).

[0040] At block 302, process 300 involves identifying a complete boundary for each object. In some cases, image processing application 102 may use a trained conditional generative adversarial network to detect the boundaries of the object. More specifically, image processing application 102 provides each object to the model, and the model in turn provides an output representing the complete boundary of the object.

[0041] Some object boundaries may be complete (eg, not occluded by another object). Other boundaries (such as the boundaries of an object occluded by a target object) may be incomplete. The image processing application 102 detects these boundaries.

[0042] Generative adversarial networks were first proposed by Ian J. Goodfellow et al. of the Department of Information and Operations Research (departement d'informatique et de recherche oprationnelle) of the University of Montreal in 2014 in "Generative Adversarial Networks". Generative adversarial networks use a generator to generate new data instances and a discriminator to evaluate whether these instances belong to the training dataset.

[0043] Discriminative models attempt to classify input data. For example, a discriminative model can be trained to predict whether an email message is SPAM or non-SPAM, or whether an image depicts a dog or not. More specifically, a discriminative model maps features (detected in the input) to labels.

[0044] In contrast, generative models try to determine how to reach a specific prediction. For example, a generative model tries to understand why a picture of a dog is classified as a dog. Instead of predicting a label given an input, a generative model tries to predict features associated with a specific label (e.g., dog).

[0045] By using the trained model, the boundaries of complex shapes can be recreated. For example, in some cases, recreating the boundaries of regular shapes may be relatively simple (e.g., linear completion), but the boundaries of other objects are more difficult to recreate. For example, the boundary of a leaf is incomplete because the boundary is occluded by a target object that is to be removed. By using the trained conditional generative adversarial network, the image processing application 102 recreates the boundary of the leaf, which may be complex and nonlinear. Reference Figure 5 An exemplary process for training a conditional adversarial network is described.

[0046] Continuing with the example, the image processing application 102 identifies the boundary for the swimming pool 410 and the lawn 411 .

[0047] At block 303, process 300 involves receiving a selection of an object as a target object to be removed from the image. Image processing application 102 may receive the selection of the target object from a user interface or from another input. Based on identifying the object as a target object, image processing application 102 proceeds to block 304 to identify intersections between boundaries of other objects and boundaries of the target object. Continuing with this example, image processing application 102 receives an identification that wooden object 412 is a target object to be removed.

[0048] At block 304, process 300 involves determining a set of masks. Each mask may correspond to an object and represent an image region defined by the intersection of the boundary of the object and the boundary of the target object. Continuing with this example, image processing application 102 identifies masks 420 (shown in image 402) and 421 (shown in image 403). More specifically, mask 420 corresponding to swimming pool 410 represents the intersection region between swimming pool 410 and wooden object 412. Mask 421 corresponding to grass 411 represents the intersection region between grass 411 and wooden object 412.

[0049] At block 305, process 300 involves updating the set of masks by performing content fill on the corresponding regions, respectively. Filling can be performed using different methods. In an example, image processing application 102 fills the corresponding regions or holes identified by each mask using content-aware fill. Different types of content-aware fill are possible.

[0050] In general, a content-aware fill algorithm seeks to approximately reconstruct the hole (the portion represented by the mask) by rearranging small square pixel patches from the rest of the object that exists (e.g., the sampled area) and stitching them together. More specifically, content-aware fill identifies approximate matching patches from the sampled area for the mask. Candidate patches from the object are selected, tested, and refined in an iterative manner. The resulting reconstruction can fill the area represented by the mask in a way that makes it appear that the portion represented by the mask (e.g., the intersection between the particular object and the target object) has never been occluded.

[0051] In some cases, the content-aware fill can be aware of object boundaries. For example, the content-aware fill method can analyze the area within the boundary of the shape to be filled and ignore the area outside the boundary, which can produce better results in some cases.

[0052] In some cases, a patch-based synthesizer can be used to perform content filling. In an example, a first user input and a second user input are received, the first user input specifies a hole to be filled in an original image; the second user input indicates a sampling region of the original image from which pixels can be sampled, the sampling region excluding the hole. A constraint mask is used to store the sampling region, and the patch-based synthesizer is configured to synthesize a target image from patches sampled in the sampling region specified by the constraint mask.

[0053] For example, a patch-based synthesizer can perform patch synthesis using a random algorithm to generate and evaluate candidate patches and identify approximate nearest neighbor matches between image patches. In order to construct a given target image (e.g., a hole) using image patches transformed from a source image, a data structure called a nearest neighbor field (NNF) can be used to manage the mapping between patches in the source image and the target image. The NNF includes a transformation for each pixel in the target image. The transformation may include a similarity transformation. For a given pixel, the transformation in the NNF for the pixel identifies the corresponding source patch, and the similarity of the source patch to the target patch associated with the pixel can be tested. The goal of the patch-based synthesizer is to identify the source patch (e.g., from a valid sampling area) that best matches each target patch (i.e., the nearest neighbor). The NNF domain can be updated during various stages of the synthesis process to keep track of the nearest neighbor source patch for each target patch.

[0054] The patch-based synthesizer may involve an iterative process of initialization, search, voting, and enlargement for each scale of the multi-scale solution. For example, for each target pixel from the target image (e.g., a hole), the initialization component assigns a randomly generated transformation as an initialization. These transformations may include similarity transformations. The similarity transformation parameters may be user-selected, predetermined, some combination thereof, etc. Typically, the randomly generated transformations are bounded on applicable similarity transformation parameters (e.g., translation, scaling, rotation, and / or mirroring search domains). Sometimes, the patch validity component determines the validity of each candidate patch. For candidate source patches that fail the patch validity test, the initialization component assigns a new randomly generated transformation to replace the candidate patch that failed the test, and repeats the patch validity test. If the patch validity test fails the test for a predetermined number of times (e.g., 256), a candidate patch may be generated by reducing the effective sampling area (e.g., enlarging the hole), defining the corresponding search domain, or using a randomly generated simple translation instead of a full similarity transformation. In this way, this alternative technique can be used to generate valid candidate patches.

[0055] Typically, a patch-based synthesizer performs a search to identify candidate patches that improve the NNF. A patch validity component determines the validity of each candidate patch. If a candidate source patch fails the patch validity test, the candidate patch is not used to improve the NNF. Candidate patches that pass patch validity are evaluated to determine whether a given candidate patch is a closer match for a particular target patch than an existing nearest neighbor in the NNF (e.g., whether the candidate patch reduces the patch distance). In other words, the NNF is the minimum of the NNF and the previous NNF.

[0056] In some embodiments, the propagation search component and the random search component can identify candidate patches in a manner that is conducive to deterministic fill synthesis. In conventional techniques, a designated hole can be divided into several sub-segments for parallel processing by different threads. In one example, the hole can be divided into three sub-segments, and each of the three threads processes the corresponding sub-segment in parallel. In conventional techniques, a particular thread processes each pixel in the assigned sub-segment in scan line order. For example, for a given pixel, the propagation search component propagates solutions for adjacent pixels and selects the best solution, the random search component identifies solutions for randomly identified pixels and selects the best solution, and the assigned thread moves to the next pixel in scan line order. However, because some threads may end generating fills for assigned sub-segments faster than other threads, patches sampled from incomplete fills for adjacent sub-segments are typically used to generate fills for sub-segments.

[0057] In some embodiments, a designated hole may be divided into more subdivisions than threads, and multiple threads may be assigned to process only non-adjacent subdivisions in parallel. In a simple example, assume that the hole is divided into six blocks 0-5. For even iterations of a patch-based synthesizer, three threads may process alternating blocks (e.g., 0, 2, 4) in scanline order. During odd iterations, threads may process alternating blocks in reverse scanline order (e.g., 1, 3, 5). Because adjacent subdivisions have already been filled when any thread finishes processing a particular subdivision, the time at which each thread finishes processing its assigned subdivision is not important. Therefore, assigning multiple threads to process non-adjacent subdivisions in parallel may produce deterministic results.

[0058] In other embodiments, wavefront techniques may be applied to identify candidate patches to facilitate deterministic fill synthesis. In general, wavefront processing is a technique for processing multidimensional grids in which specific cells in the grid depend on other cells in the grid. By starting in a corner, processing proceeds in a diagonal scan across the grid, which is similar to a wavefront. In the context of patch-based synthesis, search can be implemented using wavefronts instead of in scan line order (e.g., a propagation search component can propagate solutions for neighboring pixels above and for neighboring pixels on the left). In addition, the random number generator for randomly identifying pixels of the random search component can be modified. Random number generators are typically designed to generate a known sequence of uniform numbers when a seed is given. For wavefront processing that produces deterministic patch synthesis, patch-based synthesizer iterations and random search iterations accept (x, y) as input to generate uniform numbers. In this way, for a given (x, y) pixel value, a given sequence of calls to the random number generator will produce the same result. In this way, a deterministic set of candidate patches can be identified, thereby facilitating deterministic fill synthesis. A person of ordinary skill in the art will understand other variations.

[0059] Typically, a patch-based synthesizer performs voting to generate a proposed target image. Typically, patch voting is performed to accumulate the pixel colors of each intersecting adjacent patch, and the color votes are weighted averaged. The proposed target image can be passed to the user interface to be presented as a preview. As described above, during each subsequent iteration, the proposed target image is updated, and the updated target image can be passed to the front end for each iteration. The result is a progressively updated, real-time preview. These progressive updates can provide the user with fast, real-time feedback and an earlier opportunity to make any desired changes to achieve the desired fill.

[0060] The patch-based synthesizer performs upscaling (e.g., upscaling the current NNF via an upscaling component to serve as a baseline during a subsequent iteration at the next scale). Since this upscaling can produce invalid patches, the patch validity component advantageously determines the validity of candidate patches corresponding to the upscaling NNF. During subsequent patch-based synthesizer iterations, candidate patches that pass patch validity are evaluated to determine whether a given candidate patch is a closer match for a particular target patch than a corresponding candidate patch generated from a randomly initialized NNF.

[0061] Typically, for subsequent incremental scales, the flow through the patch-based synthesizer is repeated until a full-resolution solution is generated and passed to the front end for presentation to the user. In some embodiments, the patch-based synthesizer may be interrupted when some components detect an updated (e.g., by a user) sampling area and / or applicable translation, scaling, rotation, and / or mirroring search domain. In this scenario, as described in more detail above, the patch-based synthesizer may salvage existing calculations to increase speed and avoid unnecessary recalculations, and may automatically begin processing the updated sampling area and / or search domain. Additionally and / or alternatively, as described in more detail above, the patch-based synthesizer may pass a suggested target image for presentation as a preview, and interrupt its process before completing the population to facilitate user input. As described in more detail above, a user indication to continue processing may trigger the patch-based synthesizer to calculate the remaining resolution.

[0062] However, in some cases, the holes filled by content-aware filling include incorrect image content, producing strange and unnatural results. For example, by restricting candidate patches to two-dimensional translations, the potential quality of the reconstructed target image is limited. However, CAF techniques are already computationally expensive, and simply expanding the search domain results in an unacceptable increase in computation time.

[0063] Thus, in some cases, user input may be requested. The interactive system includes a user interface that allows the user to specify holes and sampling areas for filling the holes. The sampling areas may be stored in a constraint mask that generally excludes the holes. The brush tool may facilitate user input to add to or subtract from the sampling area. Upon detecting that the user input is complete, the interactive system may automatically pass the resulting constraint mask to a backend content-aware fill engine to synthesize the content-aware fill using the specified constraint mask.

[0064] Some embodiments may use an improved patch validity test for patch-based synthesis applications. The introduction of patch rotation and scaling increases the complexity of determining whether a candidate patch is a valid patch that falls within a sampling region. To test the validity of a given patch that includes multiple pixels, one or more simplified tests in a series of simplified tests may be performed to determine whether each pixel of the patch falls within a sampling region (e.g., specified by a constraint mask). A hole dilation test for patch validity may be performed by dilating a hole in a constraint mask to generate a reduced constraint mask, and performing a lookup to determine whether a representative pixel in the patch falls within the region specified by the reduced constraint mask. Patches that pass this test are valid. A non-dilation test for patch invalidity may be performed by finding whether a representative pixel of the patch falls within a hole (e.g., falls outside the sampling region specified by the constraint mask). Patches that meet this criterion are invalid. A comprehensive pixel test for patch validity may be performed by finding whether each pixel in the patch falls within the sampling region specified by the constraint mask. Due to the relatively large computational requirements of such a comprehensive pixel test, sometimes only those patches whose validity cannot be determined using either of the other two tests are tested by the comprehensive pixel test. Patches whose pixels pass the synthesis test are valid. One or more patch validity tests can be incorporated into an interactive system to automatically synthesize content-aware fills.

[0065] For example, testing the validity of the patch may include using a similarity transform. A similarity transform is a shape-preserving transform that may include one or more translations, rotations, scaling, or reflections (i.e., mirroring). More specifically, the example method involves generating a similarity-transformed candidate patch for reconstructing a target image, validating the candidate patch as a valid patch that falls within a sampling region of a source image based on an evaluation of representative pixels of the similarity-transformed candidate patch, and automatically reconstructing the target image using the validated candidate patch. In some cases, the candidate patch includes pixels from the source image. In some cases, wherein validating the candidate patch includes performing a first patch validity test, the first patch validity test includes dilating a hole in the sampling region to generate a reduced sampling region, and validating the candidate patch based on performing a lookup to determine that the representative pixels fall within the reduced sampling region. In some cases, dilating the sampling region includes increasing the width of the dilated strip based on the width of the candidate patch, the allowed patch rotation, the maximum allowed scaling factor, and the supersampling rate.

[0066] Continuing with this example, as can be seen in images 404-405, image processing application 102 has performed fills on regions 430 and 431. More specifically, region 430 depicts fills on previously occluded portions of pool 410, and region 431 depicts fills on previously occluded portions of grass 411.

[0067] At block 306, process 300 involves creating an output image by merging each updated mask with a portion of the image. A portion of the image is an area of ​​the image that is not represented by any mask. Image processing application 102 creates an output image by combining the filled images. Continuing with this example, as can be seen in output image 406, image processing application 102 merges images 404 and 405, including regions 430 and 431, such that output image 406 represents the removal of the target object, namely wooden object 412.

[0068] As discussed, embodiments may use a conditional generative adversarial network to perform object boundary detection, for example, as performed at block 302 of process 300 . Figure 5 An example of a process for training such a network is shown.

[0069] Figure 5 An example of a process for training a conditional adversarial network to detect the boundary of an object according to an embodiment of the present disclosure is depicted. The machine learning model 105 can be a conditional generative adversarial network, which includes a generative model 106 and a discriminative model 107. Training the generative adversarial network involves simultaneously optimizing the generative model 106 and the discriminative model 107, where the generative model 106 captures the data distribution and the discriminative model 107 estimates the probability that the sample comes from the training data rather than the generative network.

[0070] At block 501, process 500 involves determining a boundary for an object within an image. Figure 6 discuss Figure 5 ,because Figure 6 Example training images are depicted. However, other training images are possible.

[0071] Figure 6 Described is a method according to an embodiment of the present disclosure. Figure 5 An example of the training data used in . Figure 6 Depicted are image 601, mask 602, and image 603. Image processing application 102 receives image 601 and determines boundaries of objects in the image. Specifically, image processing application 102 determines boundaries for depicted leaves, depicted lizards, etc. Mask 602 depicts the determined boundaries.

[0072] At block 502, process 500 involves creating a training set of images by applying a different randomly generated mask to the border of each image so that the mask ends up covering a portion of the border. The image processing application 102 randomly determines a mask that interrupts a portion of the leaf, as depicted by point 610 within image 603. In this way, the image processing application 102 forces the conditional generative adversarial network to predict the now missing border of the leaf. Blocks 501-502 repeat until a sufficiently large training data set is created. Different types of images can be used to deepen or widen the training data set. For example, multiple images of leaves with missing borders can be used to deepen the training set.

[0073] At box 503, process 500 involves providing an image stream to the discriminative model, the image stream including the training image set and the pseudo image generated by the generative model. In some cases, the image processing application 102 creates an image stream including the generated training images and one or more pseudo images generated by the generative model. During training, the generative model 106 competes with an opponent, the discriminative model 107. The generative model 106 can be thought of as similar to a counterfeiter team that attempts to produce counterfeit money and use it without detection, while the discriminative model is similar to the police, which attempts to detect counterfeit money. Competition in this game encourages both teams to improve their methods until the counterfeit products are indistinguishable from the real products.

[0074] At block 504, process 500 involves updating one or more parameters of the generative model or the discriminative model so that the loss function is minimized. Both the generative model 106 and the discriminative model 107 may have one or more trainable parameters or weights. The generative adversarial network is trained by updating one or more of these parameters. For example, the image processing application 102 may simultaneously update the discriminative distribution so that it distinguishes samples of data from the generative distribution from samples of the generative distribution.

[0075] More specifically, the image processing application 102 receives identification of boundaries in the image from the discriminant model and for each image in the stream. In turn, the image processing application 102 calculates a loss function based on the difference between the boundaries and the reference boundaries and updates the parameters of the model accordingly. The discriminant model can identify that the fake image is not real and therefore does not update any parameters based on the fake image. The image processing application 102 can use the trained machine learning model 105 in conjunction with the process 300 to predict the boundaries of the object.

[0076] Figure 7An example of a computing system for removing an object from an image according to an embodiment of the present disclosure is depicted. Computing device 700 is an example of computing system 101 and can implement image processing application 102. Any suitable computing system can be used to perform the operations described herein. The example of the computing device 700 depicted includes a processor 702 that is communicatively coupled to one or more memory devices 704. The processor 702 executes computer executable program code 730 stored in the memory device 704, accesses data 720 stored in the memory device 704, or both. Examples of processor 702 include a microprocessor, an application specific integrated circuit ("ASIC"), a field programmable gate array ("FPGA"), or any other suitable processing device. The processor 702 may include any number of processing devices or cores, including a single processing device. The functionality of the computing device may be implemented in hardware, software, firmware, or a combination thereof.

[0077] The memory device 704 includes any suitable non-transitory computer-readable medium for storing data, program code, or both. Computer-readable media may include any electronic, optical, magnetic, or other storage device capable of providing computer-readable instructions or other program code to the processor. Non-limiting examples of computer-readable media include flash memory, ROM, RAM, ASIC, or any other medium from which a processing device can read instructions. Instructions may include processor-specific instructions generated by a compiler or interpreter from code written in any suitable computer programming language, including, for example, C, C++, C#, Visual Basic, Java, or a scripting language.

[0078] The computing device 700 may also include a number of external or internal devices, such as input or output devices. For example, the computing device 700 is shown as having one or more input / output ("I / O") interfaces 708. The I / O interfaces 708 may receive input from an input device or provide output to an output device. One or more buses 707 are also included in the computing device 700. The bus 707 communicatively couples one or more components of the corresponding computing device 700.

[0079] The computing device 700 executes program code 730, which configures the processor 702 to perform one or more operations described herein. For example, the program code 730 causes the processor to execute Figure 2 Or the operation described in 3.

[0080] The computing device 700 also includes a network interface device 710. The network interface device 710 includes any device or group of devices suitable for establishing a wired or wireless data connection to one or more data networks. The network interface device 710 can be a wireless device and have an antenna 714. The computing device 700 can use the network interface device 710 to communicate with one or more other computing devices implementing computing devices or other functionality via a data network.

[0081] Computing device 700 may also include display device 712. Display device 712 may be an LCD, LED, touch screen, or other device operable to display information about computing device 700. For example, the information may include an operating state of the computing device, a network state, and the like.

[0082] General considerations

[0083] Many specific details are set forth herein to provide a thorough understanding of the claimed subject matter. However, those skilled in the art will appreciate that the claimed subject matter may be practiced without these specific details. In other cases, methods, devices, or systems known to those of ordinary skill in the art are not described in detail in order to avoid obscuring the claimed subject matter.

[0084] Unless otherwise specifically noted, it should be understood that throughout the specification, discussions utilizing terms such as "processing," "computing," "calculating," "determining," and "identifying" refer to actions or processes of a computing device, such as one or more computers or similar electronic computing devices, that manipulate or transform data represented as physical electronic or magnetic quantities within a memory, register, or other information storage device, transmission device, or display device of a computing platform.

[0085] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable component arrangement that provides a result conditioned on one or more inputs. Suitable computing devices include a multi-purpose microprocessor-based computer system that accesses stored software that programs or configures the computing system from a general-purpose computing device to a dedicated computing device that implements one or more embodiments of the present subject matter. Any suitable programming, scripting, or other type of language or language combination may be used to implement the teachings contained herein with software to be used to program or configure a computing device.

[0086] Embodiments of the methods disclosed herein may be performed in the operation of such a computing device. The order of the blocks presented in the above examples may be changed—for example, the blocks may be reordered, combined, and / or divided into sub-blocks. Certain blocks or processes may be performed in parallel.

[0087] The use of "suitable for" or "configured to" herein is meant to be open and inclusive language that does not exclude devices that are suitable for or configured to perform additional tasks or steps. Additionally, the use of "based on" is meant to be open and inclusive, as a process, step, calculation, or other action that is "based on" one or more recited conditions or values ​​may, in practice, be based on additional conditions or values ​​other than those recited. The headings, lists, and numbers included herein are for ease of explanation only and are not intended to be limiting.

[0088] Although the subject matter has been described in detail with respect to a specific embodiment of the subject matter, it should be understood that those skilled in the art can easily generate changes, modifications and equivalents of such embodiments after understanding the foregoing. Therefore, it should be understood that the present disclosure is presented for the purpose of illustration rather than limitation, and does not exclude the inclusion of such modifications, modifications and / or additions to the subject matter that are clear to those of ordinary skill in the art.

Claims

1. A method for removing an object from an image, the method comprising: accessing an image comprising a plurality of objects; identifying a complete boundary for each object of the plurality of objects by providing each object to a trained model; receiving a selection of a target object from the plurality of objects, the target object to be removed from the image; determining a set of masks, wherein each mask corresponds to an object of the plurality of objects and represents a region of the image defined by an intersection of the complete boundary of the object and the complete boundary of the target object; Updating each mask by performing content filling on the corresponding regions respectively; as well as An output image is created by merging each of the updated masks with the portion of the image. 2 . The method of claim 1 , wherein the portion of the image is an area of ​​the image that is not represented by any of the masks.

3. The method of claim 1, wherein the trained model is a conditional adversarial network.

4. The method of claim 1 , wherein accessing the image further comprises: The plurality of objects are detected in the image.

5. The method of claim 1, wherein the filling comprises: A content-aware fill is applied to each mask separately based on the corresponding object.

6. The method of claim 1 , wherein creating the output image comprises: A layer set is created from the mask, and each layer in the layer set is merged into the output image.

7. The method of claim 1 , wherein filling each of the masks further comprises: dividing each mask into a plurality of segments; Perform content-aware fill on each segment; as well as Each segment is combined into the mask.

8. The method of claim 1 , wherein filling each of the masks comprises: generating a candidate patch, the candidate patch comprising pixels from the object represented by the mask; verifying the candidate patch; as well as The pixels of the mask are automatically reconstructed using the verified candidate patches.

9. A computer-implemented method for training a conditional generative adversarial network to predict boundaries of an object, the method comprising: determining reference boundaries for objects within the image; Creating a set of training images from the images by: applying different randomly generated masks to the reference boundary such that the masks ultimately occlude a portion of the reference boundary; The conditional generative adversarial network consisting of a generative model and a discriminative model is trained in the following way: providing an image stream to the discriminative model, the image stream comprising the training image set and pseudo images generated by the generative model, wherein the pseudo images are randomly generated; and One or more parameters of the generative model or the discriminative model are updated so that the loss function is minimized.

10. The method according to claim 9, wherein the training further comprises: For each image in the image stream, a conditional generative adversarial network is iteratively trained in the following manner: receiving, from the discriminative model and for each image, an identification of a boundary in the image; calculating a loss function based on a difference between the boundary and the reference boundary; and The parameters of the discriminant model are updated so that the loss function is minimized.

11. The method of claim 9, wherein the discriminative model identifies that the fake image is not real, and wherein updating the parameters is not based on the fake image.

12. The method according to claim 9, further comprising: accessing an image including a first object and a second object; Identify the boundaries for each object by: providing the object to the conditional generative adversarial network; receiving a complete boundary for the object from the conditional generative adversarial network; determining a set of masks, wherein each mask represents a region of the image defined by an intersection of the boundary of the first object and the boundary of the second object; updating the mask set by filling each mask in the masks; as well as An output image is created by merging each of the updated masks into an output image.

13. A system for image processing, comprising: one or more processing devices; as well as A non-transitory computer readable medium communicatively coupled to the one or more processing devices, wherein the one or more processing devices are configured to execute instructions and thereby perform operations including: accessing an image comprising a plurality of objects; identifying a complete boundary for each object of the plurality of objects by providing each object to a trained model; receiving a selection of a target object from the plurality of objects, the target object to be removed from the image; determining a set of masks, wherein each mask corresponds to an object of the plurality of objects and represents a region of the image defined by an intersection of the complete boundary of the object and the complete boundary of the target object; Updating each mask by performing content filling on the corresponding regions respectively; as well as An output image is created by merging each of the updated masks with the portion of the image.

14. The system of claim 13, wherein the portion of the image is an area of ​​the image that is not represented by any of the masks.

15. The system of claim 13, wherein the trained model is a conditional adversarial network.

16. The system of claim 13, wherein accessing the image further comprises: The plurality of objects are detected in the image.

17. The system of claim 13, wherein the filling comprises: A content-aware fill is applied to each mask separately based on the corresponding object.

18. The system of claim 13, wherein creating the output image comprises: A layer set is created from the mask, and each layer in the layer set is merged into the output image.

19. The system of claim 13, wherein filling each of the masks further comprises: dividing each mask into a plurality of segments; Perform content-aware fill on each segment; as well as Each segment is combined into the mask.

20. The system of claim 13, wherein filling each of the masks comprises: generating a candidate patch, the candidate patch comprising pixels from the object represented by the mask; verifying the candidate patch; as well as The pixels of the mask are automatically reconstructed using the verified candidate patches.

Images