Guided Contextual Attention Maps for Remediation Tasks

Abstract

Systems and methods for data augmentation can utilize one or more machine learning models and contextual attention data to perform more realistic and efficient data augmentation. For example, systems and methods for inpainting can utilize machine learning models to generate predictive contextual attention data and blend the predictive contextual attention data with acquired contextual attention data to determine replacement data for augmenting an image to replace one or more occlusions. The acquired contextual attention data can include user-guided contextual attention.

Description

[Technical Field]

[0001] The present disclosure relates generally to processing image data. More particularly, the present disclosure relates to a machine learning-based model for high-quality image inpainting that can combine contextual attention with user guidance for improved inpainting. [Background technology]

[0002] Images (e.g., photographs) and other forms of data often contain junk data. As one example, junk data may correspond to artifacts resulting from processing an image to reduce noise in the image. As another example, junk data may correspond to a human figure in the foreground of a landscape or an unknown person in the background of a family photograph. As another example, junk data may correspond to an unsightly object in an otherwise pure background.

[0003] Thus, the garbage may correspond to an object that obscures or obscures other portions of an image, such as a depicted scene. However, replacing the garbage with replacement data (e.g., replacement image data that depicts the obscured portion of the image obscured by the garbage, a process also known as "inpainting") is a difficult problem that is inherently non-deterministic. In other words, multiple possible solutions may be determined from the same image, resulting in a difficult problem.

[0004] Some current systems and methods include a "clone tool" that allows a user to select the source location of copied pixels to repair an area the user wants to remove. Meanwhile, solutions suggested in papers written by the machine learning and computer vision community include contextual attention maps, i.e., components that attempt to predict the source location of pixels using machine learning. However, both techniques have their own deficiencies. Summary of the Invention [Means for solving the problem]

[0005] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the description that follows, or may be learned from the description, or may be learned by practice of the embodiments.

[0006] One exemplary aspect of the present disclosure is directed to a computer-implemented method for training an inpainting model. The method may include receiving, by a computing system including one or more processors, an input image and a ground truth image. In some implementations, the ground truth image may depict a scene, and the input image may depict a scene having one or more occlusions. The method may include processing, by the computing system, the ground truth image with a context attention model to generate a context attention output. The method may include processing, by the computing system, the input image and the context attention output with an augmented model to generate a predicted image. The method may include evaluating, by the computing system, a loss function that evaluates a difference between the predicted image and the ground truth image. In some implementations, the method may include adjusting, by the computing system, one or more parameters of the augmented model based at least in part on the loss function.

[0007] In some implementations, the augmented model can include a predictive model, a blending model, and an occlusion model. The method can include processing the input image and the context attention output using the augmented model, including processing the input image by a computing system with the predictive model to generate predicted context attention data, processing the predicted context attention data and the context attention output by the computing system with the blending model to generate blended data, and processing the blended data and the input image by the computing system to generate a predicted image. In some implementations, the blending model can be trained to randomly blend the predicted context attention data and the context attention output. The input image can be generated by adding one or more occlusions to a ground truth image. In some implementations, the context attention model can include a convolutional neural network and one or more context attention blocks. The context attention model may be trained by processing, by a computing system, one or more training images with the context attention model to generate a training context attention output, processing, by the computing system, the training context attention output with the inpainting model to generate training augmented images, evaluating, by the computing system, a training loss function that evaluates differences between the training augmented images and the ground truth images, and adjusting, by the computing system, one or more context attention parameters of the context attention model based at least in part on the training loss function. In some implementations, the one or more occlusions may include one or more humans. The method may include receiving, by the computing system, one or more inputs describing a selection result of a portion of the input image.In some implementations, the predicted image may be generated based at least in part on one or more inputs.

[0008] Another exemplary aspect of the present disclosure is directed to a computing system for repair. The computing system may include one or more processors and one or more non-transitory computer-readable media that collectively store instructions, which, when executed by the one or more processors, cause the computing system to perform operations. The operations may include obtaining an input image. In some implementations, the input image may include one or more objects for replacement. The operations may include obtaining one or more user inputs. In some implementations, the one or more user inputs may include one or more contextual attention inputs. The operations may include processing the input image and the one or more user inputs with an augmented model to generate an augmented image. In some implementations, the augmented model may be trained to replace one or more objects via ground truth training. The operations may include providing the augmented image to a user.

[0009] In some implementations, the operation may include obtaining one or more second user inputs, where the one or more second user inputs may describe a selection result of a portion of the input image. In some implementations, the augmented image may be generated based at least in part on the one or more second user inputs. The augmented model may include a blending model trained to blend the predicted contextual attention data and the contextual attention data of the user input formula. In some implementations, the predictive contextual attention data may be generated based at least in part on the input image, and the contextual attention data of the user input formula may be based at least in part on one or more user inputs. The predicted contextual attention data and the contextual attention data of the user input formula may be blended at least in part based on blending weights, where the blending weights may be manually controlled by a user. In some implementations, the predicted contextual attention data and the contextual attention data of the user input formula may be blended at least in part based on blending weights, where the blending weights may be adjusted during ground truth training. The computing system may include a visual display for displaying the augmented image. In some implementations, the operations can include providing an input image for display to provide a visual interface to a user. The one or more user inputs can be obtained using a touchscreen display. The operations can include providing a representation of the one or more user inputs superimposed on the input image.

[0010] Another example aspect of the present disclosure is directed to one or more non-transitory computer-readable media collectively storing instructions that, when executed by one or more processors, cause a computing system to perform operations. The operations may include receiving input data from a user. In some implementations, the input data may include one or more images. The operations may include obtaining contextual attention data. In some implementations, the contextual attention data may include selected pixel data. The operations may include processing the input data with a predictive model to generate predicted contextual attention data. The operations may include processing the contextual attention data and the predicted contextual attention data with a blending model to generate blended data. The operations may include processing the input data and the blended data with an occlusion model to replace one or more pixels from the one or more images.

[0011] In some implementations, the one or more images may be frames of video, and the output of the occlusion model may be an augmented video output. The occlusion model may be trained using a perceptual loss function. The operation may include adding one or more pixels back to the one or more images to generate a rendered image. In some implementations, the contextual attention data may include a flattened input tensor. The contextual attention data may be generated at least in part based on a lasso gesture input by a user.

[0012] Another exemplary aspect of the present disclosure is directed to a computer-implemented method for training an augmented model. The method may include receiving, by a computing system including one or more processors, an input image and a ground truth image. In some implementations, the ground truth image may depict a scene, and the input image may depict a scene having one or more occlusions. The method may include processing, by the computing system, the ground truth image with a context attention model to generate a context attention output. The method may include processing, by the computing system, the input image and the context attention output with the augmented model to generate a predicted image. The method may include evaluating, by the computing system, a loss function that evaluates a difference between the predicted image and the ground truth image, and adjusting, by the computing system, one or more parameters of the augmented model based at least in part on the loss function.

[0013] In some implementations, the augmented model may include a predictive model, a blending model, and an occlusion model. The operations may include processing the input image and the context attention output using the augmented model, including processing the input image by a computing system with the predictive model to generate predicted context attention data, processing the predicted context attention data and the context attention output by the computing system with the blending model to generate blended data, and processing the blended data and the input image by the computing system with the occlusion model to generate a predicted image. In some implementations, the blending model may be trained to randomly blend the predicted context attention data and the context attention output. The occlusion model may be trained using a perceptual loss function. The input image may be generated by adding one or more occlusions to a ground truth image. In some implementations, the context attention model may include a convolutional neural network and one or more context attention blocks. The context attention model may be trained by processing, by a computing system, one or more training images with the context attention model to generate a training context attention output, processing, by the computing system, the training context attention output with the inpainting model to generate training augmented images, evaluating, by the computing system, a training loss function that evaluates differences between the training augmented images and the ground truth images, and adjusting, by the computing system, one or more context attention parameters of the context attention model based at least in part on the training loss function. In some implementations, the one or more occlusions may include one or more humans.

[0014] In some implementations, the method may include receiving, by a computing system, one or more inputs describing a selection result of a portion of the input image, and the predicted image may be generated based at least in part on the one or more inputs.

[0015] Another exemplary aspect of the present disclosure is directed to a computer-implemented method. The method can include obtaining an input image. The input image can include one or more objects for replacement. The method can include obtaining one or more contextual attention inputs. The method can include processing the input image and the one or more contextual attention inputs with an augmented model to generate an augmented image. In some implementations, the augmented model can be trained to replace the one or more objects via ground truth training. The method can include providing the augmented image to a user.

[0016] In some implementations, the augmented model can be trained using a computer-implemented method. The one or more contextual attention inputs can be user inputs. The one or more user inputs can be acquired using a touchscreen display. The method can include providing a representation of the one or more user inputs superimposed on the input image. In some implementations, the contextual attention data can be generated at least in part based on a lasso gesture input by a user. The method can include acquiring one or more second user inputs. In some implementations, the one or more second user inputs can describe a selection result of a portion of the input image, and the augmented image can be generated at least in part based on the one or more second user inputs. The augmented model can include a blending model trained to blend predictive contextual attention data and contextual attention data of a user input expression, where the predictive contextual attention data can be generated at least in part based on the input image, and the contextual attention data of the user input expression can be based at least in part on the one or more user inputs. In some implementations, the predictive contextual attention data and the user-entered contextual attention data can be blended based at least in part on blending weights, which can be manually controlled by a user.

[0017] In some implementations, the predicted contextual attention data and the user-input contextual attention data can be blended based at least in part on blending weights, and the blending weights can be adjusted during ground truth training. The method can include providing input images for display to provide a visual interface to a user. The input images can be frames of a video, and the output of the augmented model can be an augmented video output. In some implementations, the contextual attention input can include a flattened input tensor.

[0018] In some implementations, one or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more processors, cause a computing system to perform a computer-implemented method.

[0019] In some implementations, a computing system may include one or more processors and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform a computer-implemented method.

[0020] Other aspects of the present disclosure are directed to various systems, apparatus, non-transitory computer-readable media, user interfaces, and electronic devices.

[0021] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the present disclosure and, together with the description, serve to explain associated principles.

[0022] Detailed descriptions of embodiments directed to persons skilled in the art are set forth herein, which refer to the accompanying figures. [Brief explanation of the drawings]

[0023] [Figure 1A] FIG. 1 is a block diagram of an exemplary computing system for performing image augmentation, according to an exemplary embodiment of the present disclosure. [Figure 1B] FIG. 1 is a block diagram of an exemplary computing device that performs image augmentation, according to an exemplary embodiment of the present disclosure. [Figure 1C] FIG. 1 is a block diagram of an exemplary computing device that performs image augmentation, according to an exemplary embodiment of the present disclosure. [Figure 2] FIG. 1 is a block diagram of an exemplary inpainting model for training, according to an exemplary embodiment of the present disclosure. [Figure 3] FIG. 1 is a block diagram of an exemplary contextual attention remediation model, according to an exemplary embodiment of the present disclosure. [Figure 4] FIG. 1 is a block diagram of an exemplary contextual attention remediation model with blending, according to an exemplary embodiment of the present disclosure. [Figure 5] FIG. 1 is a block diagram of an exemplary context attention mapping, according to an exemplary embodiment of the present disclosure. [Figure 6] FIG. 1 is a flowchart diagram of an example method for performing extended model training, according to an example embodiment of the present disclosure. [Figure 7] FIG. 1 is a flowchart diagram of an exemplary method for performing image augmentation based on context attention, according to an exemplary embodiment of the present disclosure. [Figure 8] FIG. 1 is a flowchart diagram of an exemplary method for performing image augmentation, according to an exemplary embodiment of the present disclosure. [Figure 9] FIG. 1 is a diagram of an exemplary repair process, according to an exemplary embodiment of the present disclosure. [Figure 10] FIG. 1 is a diagram of an exemplary context attention inpainting image, according to an exemplary embodiment of the present disclosure. [Figure 11] FIG. 1 is a diagram of an exemplary context attention inpainting image, according to an exemplary embodiment of the present disclosure. [Figure 12] FIG. 10 is an illustration of an exemplary blended restoration result, according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0024] Reference numbers repeated across the figures are intended to identify like features in various implementations.

[0025] overview Generally, the present disclosure is directed to utilizing machine learning models for inpainting. Images may often have one or more occlusions that a user desires to remove or replace with predicted pixels. For example, a photo taken at a landmark may include signs, people, and other objects that a user desires to remove. The systems and methods disclosed herein may be utilized to remove one or more unwanted objects from an image. The systems and methods may obtain input data and contextual attention data. The input data may include one or more images with one or more objects for cloning or removal, and the contextual attention data may include data describing the objects to be removed and / or areas of the image to be cloned. The input data and contextual attention data may then be processed by an augmented model to generate an augmented image. The augmented image may comprise features from one or more images of the input data, but may include cloned portions of the input image in place of one or more objects for removal.

[0026] The systems and methods disclosed herein can utilize one or more machine learning models to predict obscured data and replace occlusions with predicted data to generate augmented data. In some implementations, the one or more machine learning models can be trained using ground truth training. Systems and methods for training an inpainting model can include receiving input data and ground truth data. The ground truth data can include unobstructed data, and the input data can include data with one or more occlusions. The input data can be generated by adding the occlusions to the ground truth data. The ground truth data can be processed using a context attention model to generate a context attention output. The input data and the context attention output can then be processed using an augmented model to generate predicted data. The predicted data can include data in which one or more occlusions have been replaced by predicted replacement data. The predicted data and the ground truth data can then be compared to evaluate a loss function. Based on the evaluation, one or more parameters of the augmented model can be adjusted. The process can be repeated iteratively to train the augmented model to accurately inpaint the obscured data.

[0027] The trained model can then be used for various inpainting tasks. The systems and methods disclosed herein can be used to augment various data types, including image data, audio data, and / or three-dimensional point cloud data. Systems and methods for inpainting using one or more machine learning models can include acquiring input data, where the input data includes one or more occlusions for removal. The systems and methods can further include acquiring one or more user inputs. The one or more user inputs can include one or more contextual attention inputs and may include one or more selections of portions of the input data to be cloned. The input data and the one or more user inputs can be processed using the augmentation model to generate augmented data. The augmented data can include replacement data for one or more occlusions. The systems and methods can further include providing the augmented data to a user.

[0028] In some implementations, the systems and methods may be utilized for image inpainting. For example, a user may desire to replace one or more objects in an image with predicted data to remove unwanted humans, trash, buildings, etc. Utilizing the systems and methods for image inpainting may include receiving input data from a user, where the input data includes one or more images. Contextual attention data may also be obtained. The contextual attention data may include selected pixel data, where the selected pixel data includes pixels of the input image. The input data may be processed using a predictive model to generate predicted contextual attention data. The predicted contextual attention data and the contextual attention data may be processed using a blending model to generate blended data. The blended data and the input image may then be processed using an occlusion model to generate an augmented image. The augmented image may include the input image with one or more occlusions replaced with predicted data.

[0029] Training a machine learning model can involve receiving or acquiring input data and ground truth data. The input data and ground truth data can include input images and ground truth images, respectively. The ground truth images can depict a scene, and the input images can depict a scene with one or more occlusions. In some implementations, the input images can be generated by adding occlusions to the ground truth images. Training one or more machine learning models can include acquiring multiple training sets, each training set including input data and ground truth data. Alternatively and / or additionally, the input data and ground truth data can include audio data, in which a portion of the audio data for the input data is obscured compared to the ground truth data. The occlusions can be occlusions created by humans, animals, buildings, and / or image capture. In some implementations, the data can be audio data, and the occlusions can be random sounds, equipment-generated noise, white noise, etc.

[0030] The ground truth data may be processed using a context attention model to generate a context attention output. The context attention model may include one or more context attention sub-blocks. In some implementations, the context attention model may include a feed-forward fully convolutional neural network. The context attention model may include one or more softmax sub-blocks.

[0031] In some implementations, the context attention model may be trained by processing one or more training images with the context attention model to generate a training context attention output. The training context attention output may be processed with the inpainting model to generate training augmented images. Differences between the training augmented images and the ground truth images may then be used to evaluate a training loss function. The training loss function evaluation may then be used to adjust one or more context attention parameters of the context attention model.

[0032] The context attention data may include a portion of ground truth image data generated by processing the ground truth image with the context attention sub-block and the convolutional neural network. Alternatively and / or additionally, the context attention data may include user input data comprising one or more selections of portions of the input image to be cloned.

[0033] The input data and the context attention output can be processed with the augmented model to generate predicted data, which can include augmented image data that replaces one or more occlusions with predicted replacement data.

[0034] In some implementations, the augmented model can include a prediction model, a blending model, and an occlusion model. The prediction model can process the input data to generate a context attention prediction. The blending model can process the predicted context attention data and the context attention output to generate blended data. The blended data can then be processed with the input data to generate a predicted image that includes the occlusion object replaced by the cloned data.

[0035] The blending model may be trained to randomly blend the predictive contextual attention data and the contextual output data, which may be generated based on user input or may be automatically generated based on one or more datasets.

[0036] The predicted data and the ground truth data may then be compared to evaluate a loss function. The loss function may be a perceptual loss function. Alternatively, the loss function may be a Kullback-Leibler (KL) divergence loss function, a VGG loss function, or a GAN loss function. The loss function may then be used to adjust one or more parameters of the augmented model. In some implementations, one or more parameters of at least one of the predicted model and the blended model may be adjusted in response to an evaluation of the loss function.

[0037] A computing system that utilizes one or more machine learning models for remediation can include one or more processors and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations can include a method for remediation.

[0038] The system and method can include acquiring input data. The input data can include one or more input images. The input images can include one or more objects for removal. The objects can be occlusions that obscure one or more portions of a scene depicted in the image. The occlusions can include image imperfections, unwanted people or objects captured in the image, and / or previously placed occlusions.

[0039] One or more user inputs may then be obtained. The one or more user inputs may include one or more contextual attention inputs. For example, in some implementations, the one or more user inputs may include a selection of at least one of an occlusion or a portion of the image that contains pixels that the user desires to clone.

[0040] In some implementations, the first user input can include one or more selections of occlusions for replacement. The system and method can then obtain one or more second user inputs. The one or more second user inputs can include one or more selections of portions of the input data to clone to replace the one or more occlusions.

[0041] The input data and one or more user inputs may be processed with an augmented model to generate augmented data. The augmented model may be trained to remove and replace one or more objects via ground truth training. In some implementations, the augmented model generates a predicted contextual attention output, blends the predicted contextual attention output with a user-selected portion of the input data, and generates the augmented data using the blended data and the input data. The blending weights may be chosen randomly, automatically determined, or manually selected. In some implementations, the blending weights may be manually adjusted by a user via one or more inputs to a user interface. The blending weights may be pre-selected or chosen via a slide-scale function, so that a user can see various possible options based on the blending weights. Alternatively and / or additionally, the blending weights may be machine-learned. For example, in some implementations, the blending weights may be determined and adjusted based on ground truth training and / or retrained via reinforcement training. The blending weights may be adjusted based on previous user interactions, such that the blending model is adjusted to be tailored toward user preferences. Blending can include semantically blending the data. In some implementations, blending can involve attention pooling.

[0042] The augmented data may then be provided to the user. The augmented data may include an augmented image in which portions of the input image are replaced with the predicted data to replace one or more objects in the input image. The augmented image may be provided via a visual display, the visual display being part of a computing device. In some implementations, the computing device may be a mobile computing device.

[0043] In some implementations, the systems and methods disclosed herein may be implemented in a mobile computing device. For example, input data may be acquired from one or more sensors stored on the mobile computing device (e.g., image data acquired using one or more image sensors, audio data acquired using one or more audio sensors, etc.). The one or more user inputs may be one or more touch inputs on a touchscreen display, and augmented data may be provided via one or more output components of the mobile computing device (e.g., augmented images displayed on a visual display, augmented audio played through one or more speakers, etc.).

[0044] In some implementations, the systems and methods disclosed herein may be provided via a user interface. For example, the user interface may include one or more visual interface functions. More particularly, the input data may include an image that may be provided for display. The user interface may then allow a user to select one or more portions of the image to select for replacement. The user interface may also allow a user to select one or more portions of the image to clone to replace one or more previously selected portions. The input image and user input may then be processed to generate an augmented image, which is then provided for display via the visual interface.

[0045] Moreover, in some implementations, when user input is provided, one or more depictions may be displayed on the input image. For example, in response to selecting an occlusion for removal, a silhouette of the occlusion may be displayed superimposed on the input image to provide visual feedback of the selection. The silhouette may be colored in one or more colors, may be opaque or partially transparent, and / or may be labeled. Alternatively and / or additionally, the user input may be displayed similar to drawing on the input image, such that the first user input and / or the second user input may be displayed similar to a brush tool response in a paint application. The first user input and the second user input may be displayed in different colors or styles.

[0046] In some implementations, the systems and methods disclosed herein can be used to move objects in an image and may be used to generate videos of objects moving throughout a scene. For example, a user may select one or more objects to be segmented and moved to different parts of the image. Portions of the image that originally contained one or more objects can then be inpainted using the systems and methods disclosed herein. These systems and methods can also be used to resize objects in an image and inpaint the now-extended portions of the image.

[0047] One or more objects or occlusions may be selected by automatic recognition, lasso gestures, manual brush input, or through various other selection techniques. In some implementations, the lasso gestures may be assisted by an object recognition process. The automatic recognition may involve training one or more machine learning models to identify the foreground and background of an image. The one or more machine learning models may be trained to further determine people or objects of interest. The one or more machine learning models may also learn from past user interactions which objects or features (e.g., trash cans, logos, image or facial imperfections, etc.) users frequently select for removal. The one or more machine learning models may also be trained to learn which people and objects are acceptable to keep in future input images.

[0048] In some implementations, the systems and methods disclosed herein may include flattening an input tensor, processing the flattened input tensor with a context attention matrix to generate a transformed input, which may then be unflattened to generate an output tensor.

[0049] The context attention model and the augmented model may be trained simultaneously or separately. In some implementations, the context attention model may be trained before training the augmented model.

[0050] In some implementations, the augmentation model may utilize an input mask target for augmentation. Additionally and / or alternatively, the systems and methods disclosed herein may utilize an image segmentation mask to segment portions of an image for image inpainting tasks.

[0051] The systems and methods of the present disclosure provide several technical effects and advantages. As an example, the inpainting machine learning system can aid computational performance by improving the parameters of predictions completed relative to the creation of replacement data. Accordingly, the inpainting performed may be of higher quality (e.g., more accurate) than previous techniques, which represents an improvement in the performance of the computing system. Furthermore, the proposed approach may eliminate the need to create such a large spectrum of predictions to be evaluated, as required by some existing techniques. Eliminating the need to create many different predictions can result in savings in computing resources, such as processor usage, memory usage, and / or network bandwidth usage. The use of ground truth data also removes some of the clutter from training, making it more efficient and thereby saving computing resources. The trained system may reduce the amount of computing resources utilized relative to previous systems.

[0052] A machine learning implementation may also add more efficiency by eliminating the need to manually edit out every occurrence of junk data in an image. The system may also eliminate the need for coders to write lengthy code, run the code, improve the code, and continuously monitor performance.

[0053] Additionally, the systems and methods described herein may be used in any process where images are used as input to a system to provide the system with higher quality input images. Non-limiting examples of possible applications include medical images such as x-rays or scans of a patient, monitoring the condition of pieces of machinery where images of pieces of machinery are periodically collected and used to determine when parts may need repair or replacement, and autonomous vehicles making decisions about their path and speed based on images they collect of their surroundings.

[0054] The reduced computational costs of the systems and methods disclosed herein may enable the systems and methods disclosed herein to be used with mobile computing devices, allowing image restoration to be completed in the palm of a user's hand.

[0055] Referring now to the figures, exemplary embodiments of the present disclosure will be described in further detail.

[0056] Exemplary Devices and Systems 1A illustrates a block diagram of an exemplary computing system 100 for performing data augmentation, according to an exemplary embodiment of the present disclosure. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150, which are communicatively coupled via a network 180.

[0057] The user computing device 102 may be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a game console or game controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0058] The user computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple operatively connected processors. The memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 that are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0059] In some implementations, the user computing device 102 can store or include one or more extension models 120. For example, the extension models 120 can be or otherwise include various machine learning models, such as neural networks (e.g., deep neural networks) or other types of machine learning models including nonlinear and / or linear models. The neural networks can include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. An exemplary extension model 120 is described with reference to FIGS. 2-5.

[0060] In some implementations, one or more augmented models 120 can be received from a server computing system 130 over a network 180, stored in a user computing device memory 114, and then used or otherwise implemented by one or more processors 112. In some implementations, a user computing device 102 can implement multiple parallel instances of a single augmented model 120 (e.g., perform parallel data augmentation across multiple instances of occlusion).

[0061] More specifically, the systems and methods disclosed herein can obtain input data and contextual attention data, which can then be processed using an augmentation model to generate augmented data (e.g., an image can be augmented to remove one or more occlusions). The augmentation model can be trained using ground truth training and can include a predictive model and a blending model. The predictive model can process the input data to generate predicted contextual attention data. The blending model can then process the predicted contextual attention data and the obtained contextual attention data to generate blended data. The blending data and the input data can then be processed using an occlusion model included in the augmentation model to generate augmented data that includes replacement data in place of one or more occlusions.

[0062] Additionally or alternatively, one or more extension models 140 may be included in or otherwise stored and implemented by a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the extension models 140 may be implemented by the server computing system 130 as part of a web service (e.g., a repair service). Thus, one or more models 120 may be stored and implemented at the user computing device 102 and / or one or more models 140 may be stored and implemented at the server computing system 130.

[0063] The user computing device 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component may be useful for implementing a virtual keyboard. Other exemplary user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0064] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple operatively connected processors. The memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0065] In some implementations, server computing system 130 includes or is otherwise implemented by one or more server computing devices. In instances where server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a serial computing architecture, a parallel computing architecture, or some combination thereof.

[0066] As described above, the server computing system 130 may store or otherwise include one or more machine learning-based extended models 140. For example, the models 140 may be or otherwise include various machine learning-based models. Exemplary machine learning-based models include neural networks or other multi-layer nonlinear models. Exemplary neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Exemplary models 140 are described with reference to FIGS. 2-5.

[0067] The user computing device 102 and / or the server computing system 130 can train the models 120 and / or 140 through interaction with a training computing system 150 that is communicatively coupled via a network 180. The training computing system 150 may be separate from the server computing system 130 or may be part of the server computing system 130.

[0068] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple operably connected processors. The memory 154 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 may store data 156 and instructions 158 that are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0069] The training computing system 150 may include a model trainer 160 that trains the machine learning models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130 using various training or learning techniques, such as, for example, backpropagation. For example, a loss function may be backpropagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques may be used to iteratively update the parameters over a number of training iterations.

[0070] In some implementations, performing error backpropagation may include performing truncated backpropagation through time. The model trainer 160 may perform several generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the model being trained.

[0071] In particular, model trainer 160 can train augmented model 120 and / or 140 based on a set of training data 162. Training data 162 can include, for example, a training data set including input data and ground truth data. In some implementations, the input data can be generated by adding one or more occlusions to the ground truth data.

[0072] In some implementations, if the user has given consent, the training examples may be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 may be trained by the training computing system 150 against user-specific data received from the user computing device 102. In some instances, this process may be referred to as personalizing the model.

[0073] Model trainer 160 includes computer logic utilized to provide desired functionality. Model trainer 160 may be implemented in hardware, firmware, and / or software controlling a general-purpose processor. For example, in some implementations, model trainer 160 includes program files stored on a storage device, loaded into memory, and executed by one or more processors. In other implementations, model trainer 160 includes one or more sets of computer-executable instructions stored on a tangible computer-readable storage medium, such as RAM, a hard disk, or an optical or magnetic medium.

[0074] Network 180 may be any type of communications network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications over network 180 may be carried over any type of wired and / or wireless connection using a wide variety of communications protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0075] The machine learning models described herein may be used in a variety of tasks, applications, and / or use cases.

[0076] In some implementations, an input to a machine learning model of the present disclosure may be image data. The machine learning model may process the image data to generate an output. As an example, the machine learning model may process the image data to generate an image segmentation output. As another example, the machine learning model may process the image data to generate an image data modification output (e.g., alteration of the image data, etc.). As another example, the machine learning model may process the image data to generate an encoded image data output (e.g., an encoded and / or compressed representation of the image data, etc.). As another example, the machine learning model may process the image data to generate an upscaled image data output.

[0077] In some implementations, the input to the machine learning model of the present disclosure may be audio data. The machine learning model may process the audio data to generate an output. As another example, the machine learning model may process the audio data to generate a latent embedding output. As another example, the machine learning model may process the audio data to generate an encoded audio output (e.g., an encoded and / or compressed representation of the audio data, etc.). As another example, the machine learning model may process the audio data to generate an upscaled audio output (e.g., audio data of higher quality than the input audio data, etc.). As another example, the machine learning model may process the audio data to generate a predicted output.

[0078] In some implementations, the input to the machine learning model of the present disclosure may be latent coding data (e.g., a latent space representation of the input, etc.). The machine learning model may process the latent coding data to generate an output. As an example, the machine learning model may process the latent coding data to generate a recognition output. As another example, the machine learning model may process the latent coding data to generate a reconstruction output. As another example, the machine learning model may process the latent coding data to generate a search output. As another example, the machine learning model may process the latent coding data to generate a reclustered output. As another example, the machine learning model may process the latent coding data to generate a prediction output.

[0079] In some implementations, an input to a machine learning model of the present disclosure may be sensor data. The machine learning model may process the sensor data to generate an output. As an example, the machine learning model may process the sensor data to generate a recognition output. As another example, the machine learning model may process the sensor data to generate a prediction output. As another example, the machine learning model may process the sensor data to generate a segmentation output. As another example, the machine learning model may process the sensor data to generate a segmentation output. As another example, the machine learning model may process the sensor data to generate a visualization output. As another example, the machine learning model may process the sensor data to generate a detection output.

[0080] In some cases, the machine learning model may be configured to perform a task that includes encoding input data for reliable and / or efficient transmission or storage (and / or corresponding decoding). For example, the task may be an audio compression task. The input may include audio data, and the output may comprise compressed audio data. In another example, the input includes visual data (e.g., one or more images or videos), and the output comprises compressed visual data, and the task is a visual data compression task. In another example, the task may comprise generating an embedding for the input data (e.g., input audio or visual data).

[0081] In some cases, the input includes visual data and the task is a computer vision task. In some cases, the input includes pixel data for one or more images and the task is an image processing task. The image processing task may be object detection, where the image processing output identifies one or more regions of one or more images and, for each region, the likelihood that the region depicts an object of interest. As another example, the image processing task may be image segmentation, where the image processing output specifies, for each pixel in one or more images, a respective likelihood for each category in a predetermined set of categories. For example, the set of categories may be foreground and background. As another example, the set of categories may be object class. As another example, the image processing task may be depth estimation, where the image processing output specifies a respective depth value for each pixel in one or more images. As another example, the image processing task may be motion estimation, where the network input includes multiple images and the image processing output specifies, for each pixel in one of the input images, the motion of the scene depicted at the pixel between images in the network input.

[0082] 1A illustrates one exemplary computing system that may be used to implement the present disclosure. Other computing systems may also be used. For example, in some implementations, a user computing device 102 may include a model trainer 160 and a training dataset 162. In such implementations, the model 120 may be both trained and used locally on the user computing device 102. In some such implementations, the user computing device 102 may implement the model trainer 160 to personalize the model 120 based on user-specific data.

[0083] 1B illustrates a block diagram of an exemplary computing device 10 for executing, according to an exemplary embodiment of the present disclosure, the computing device 10 being a user computing device or a server computing device.

[0084] Computing device 10 includes several applications (e.g., applications 1-N). Each application includes its own machine learning library and machine learning model. For example, each application may include a machine learning model. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.

[0085] 1B , each application may communicate with several other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application may communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0086] 1C illustrates a block diagram of an exemplary computing device 50 for executing, according to an exemplary embodiment of the present disclosure, the computing device 50 may be a user computing device or a server computing device.

[0087] Computing device 50 includes several applications (e.g., applications 1-N). Each application is in communication with a central intelligence layer. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a common API across all applications).

[0088] The central intelligence layer includes several machine learning models. For example, as shown in FIG. 1C, a respective machine learning model (e.g., model) can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine learning model. For example, in some implementations, the central intelligence layer can provide a single model (e.g., a single model) to all of the applications. In some implementations, the central intelligence layer is included within or otherwise implemented by the operating system of the computing device 50.

[0089] The central intelligence layer can communicate with a central device data layer, which can be a centralized repository of data for computing device 50. As shown in FIG. 1C , the central device data layer can communicate with several other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0090] Exemplary Model Configuration 2 shows a block diagram of an exemplary augmentation model 200 being trained, according to an exemplary embodiment of the present disclosure. In some implementations, the augmentation model 200 is trained to receive a set of input data 202 describing one or more images depicting a scene having one or more occlusions, and provides output data 226 including an augmented predicted image as a result of receiving the input data 202. Thus, in some implementations, the augmentation model 200 may include an image augmentation model 210 operable to replace occlusions in an image with predicted pixels.

[0091] 2 includes training flows for both the augmented model 210 and the context attention model 220. The augmented model 210 and the context attention model 220 may be trained simultaneously or separately. In some implementations, the context attention model 220 may be trained before the augmented model 210.

[0092] Training the augmented model 210 may involve obtaining input data 202 and ground truth data 204. The input data 202 and the ground truth data 204 may each include one or more images. The ground truth images may depict a scene, and the input images may depict the scene with one or more occlusions added. The ground truth data 204 may be processed by a context attention model 220 to estimate context attention using machine learning. The output of the context attention model 220 may include a context attention output. The input data 202 may be processed by the augmented model 210 in conjunction with the inpainting model 212. The input data 202 may be processed by a predictive model 214 to generate predicted context attention data. The predicted context attention data and the context attention output may then be processed using a blending model 216 to randomly blend the data to generate blended data. The blended data and the input 202 data may then be processed to generate predicted data 226. The prediction data 226 may include a predicted image depicting the scene of the input image with one or more occlusions replaced with predicted pixels. The prediction data 226 and the ground truth data 204 may then be used to evaluate a loss function 228. One or more parameters of the augmented model 210 may be adjusted in response to the loss function 228 evaluation. The one or more parameters may be one or more parameters of at least one of the prediction model 214, the blending model 216, or the occlusion model 212. The trained augmented model may then be used for inpainting tasks with or without user guidance.

[0093] The context attention model 220 may be trained by processing the ground truth data 204 to generate a context attention output. The context attention output and the input data 202 may then be processed using the inpainting model 222 to apply the context attention map to an inpainting task to generate training augmented data. The training augmented data and the ground truth data 204 may be compared to evaluate a training context attention loss function 224. Based on the evaluation, one or more parameters of the context attention model 220 may be adjusted. In some implementations, the input data 202, the ground truth data 204, and the training augmented data may each include a respective image, where the training augmented image depicts the input image with one or more pixels replaced with predicted data based on the ground truth image 204.

[0094] 3 illustrates a block diagram of an exemplary augmented model 300, according to an exemplary embodiment of the present disclosure. In some implementations, the augmented model 300 is trained to receive a set of input data 302 describing an input RGB image having one or more objects that a user desires to replace, and provides output data 310, which includes an augmented RGB image in which the one or more objects have been replaced with predicted pixels as a result of receiving the input data 302. Thus, in some implementations, the augmented model 300 may include an inpainting augmented model 306 operable to complete the inpainting task.

[0095] In FIG. 3 , the augmented model 306 can perform inpainting tasks 308 based at least in part on the user-guided contextual attention data 304. For example, in this implementation, the augmented model 306 can receive input data 302 and one or more user inputs in the form of the user-guided contextual attention data 304 and can generate augmented data 310. The augmented model can be trained using ground truth training. In some implementations, the augmented data 310 can include predicted data generated based at least in part on the user-guided contextual attention data 304. In some implementations, the input data 302 can include RGB image data that can be augmented by replacing a plurality of pixels to generate the augmented data 310. The pixels being replaced can be manually selected or automatically recognized. In some implementations, the data replacing the plurality of pixels can include one or more pixels selected by a user.

[0096] 4 illustrates a block diagram of an exemplary extended model 400 according to an exemplary embodiment of the present disclosure. The extended model 400 is similar to the extended model 300 of FIG. 3, except that the extended model 400 further includes a predictive context attention model 414 and a blending model 416.

[0097] In Figure 4, the augmented model 410 is configured to receive input data 402 and user input data 404. The augmented model 410 may include one or more machine learning models trained to perform inpainting tasks. For example, the augmented model 410 of Figure 4 includes a predictive model 414 for generating predicted contextual attention data 414, a blending model 416 for blending the predicted contextual attention data and the user input data 404, and an occlusion model 412 for replacing occlusions in the input data 402 based on the blended data.

[0098] For example, the augmentation model 410 may obtain input data 402 and user input data 404. The input data 402 may be processed using the predictive model 414 to generate predicted contextual attention data. The predicted contextual attention data and the user input data 404 may then be blended with the blending model 416 to generate blended data. The blended data and the input data 402 may then be processed using the occlusion model 412 to replace one or more occlusions in the input data 402 to generate the augmented data 420.

[0099] 5 illustrates a block diagram of an exemplary context attention model 500, according to an exemplary embodiment of the present disclosure. In some implementations, the context attention model 500 is trained to receive a set of input data 502 describing an image, and provides output data 510 describing a context attention output as a result of receiving the input data 502. Accordingly, in some implementations, the context attention model 500 may include a context attention matrix 506 operable to transform the flattened input tensor 502.

[0100] The context attention model 500 may be used for context attention mapping. The context attention mapping may include receiving an input tensor 502. The input tensor 502 may then be flattened 504. The flattened input tensor 504 may be processed with a context attention matrix 506 to generate transformed data 508. The transformed data may then be unflattened to generate an output tensor 510.

[0101] 9 shows a diagram of an exemplary inpainting process 900 according to an exemplary embodiment of the present disclosure. A first image 902 may be an input image. The input image may be an RGB image. The input image may include one or more occlusions.

[0102] A second image 904 includes a recognition of a human 910 obstructing the scene and an airplane 912. A set of source pixels 908 can be identified for use in replacing the pixels that depict the airplane 912.

[0103] Finally, the third image 906 includes a representation of source pixels 914 selected to replace pixels depicting the human 910. The replacement may then be completed by blending the predictive contextual attention data with the selected / identified source pixels 912 and 914, and then generating an augmented image based on the blended data and the input image.

[0104] FIG. 10 shows a diagram of an exemplary repair result 1000 according to an exemplary embodiment of the present disclosure.

[0105] The top-left image 1002 shows an example input image, which can be processed with the inpainting model without user-guided input to generate the inpainting image in the top-right image 1004. The inpainting image includes replacement pixels in place of one or more occlusions (e.g., humans).

[0106] The bottom-left image 1006 depicts the input image with a depiction of user input 1010 superimposed on the input image. The user input 1010 may include a contextual attention input selection of a portion of the input image to use as a source pixel. The target pixel 1012 for replacement may be automatically recognized or manually selected. In some implementations, a predictive model may process the input image to generate predictive contextual attention data. The predictive contextual attention data and the contextual attention input may be processed with a blending model to generate blend data. The blending data and the input image may then be processed to generate the augmented image depicted in the bottom-right 1008. The augmented image based in part on the user input 1010 may provide a more accurate prediction than an inpainting image generated without user input.

[0107] FIG. 11 shows a diagram of an exemplary inpainting result 1100 according to an exemplary embodiment of the present disclosure. A first image 1102 may be an input image and may include one or more occlusions to be replaced. For example, the input image may include one or more people that a user wants removed from the scene. One or more machine learning models may recognize one or more objects in the image to retain and one or more objects to remove. The one or more objects 1110 to retain may be indicated via a representation superimposed on a portion of the input image in a first color. The one or more objects 1108 to remove may be indicated via a representation superimposed on a portion of the input image in a second color. A user may select an object 1112 to remove and may select a portion of the image to use as a source pixel 1114. The selection may be indicated via one or more representations superimposed on the image. A second image 1104 depicts a representation indicating the portion of the image to use as a source pixel 1114. The selected source pixels 1114 and the input image may be processed using the augmentation model to generate an output image 1106 in which the unwanted objects have been replaced with replacement pixels.

[0108] FIG. 12 shows a diagram of an exemplary inpainting result 1200, according to an exemplary embodiment of the present disclosure. In some implementations, an input image 1204 may be provided for display as part of a visual interface. The input image 1204 may include one or more occlusions that a user desires to remove. The occlusions may be manually selected or automatically recognized using one or more recognition techniques. The visual interface may provide an overlay interface 1202 that can indicate the recognized occlusions 1210 using one or more depictions superimposed on the input image. A user may interact with the visual interface to provide user input to select a portion of the image to use as a source pixel 1208. The selected source pixel 1208 may be indicated using one or more indicators.

[0109] The augmented system for image inpainting can process an input image 1204 with a predictive model to generate predictive contextual attention data. The predictive contextual attention data and the contextual attention data generated based on user input can be processed with a blending model to generate blended data. The blended data and the input image 1204 can then be processed with an occlusion model to generate an augmented image 1206 that includes replacement pixels in place of pixels used to depict one or more occlusions. Blending weights for the blending model can be selected manually or automatically determined during training of one or more models.

[0110] The systems and methods disclosed herein may include one or more occlusion models or one or more inpainting models for augmenting the data. The inpainting and / or occlusion models disclosed herein may include a conditional variational autoencoder for processing the input data and the blended data to generate the augmented data.

[0111] According to aspects of the present disclosure, the substitution data may be predicted by a machine learning model, such as, for example, a conditional variational autoencoder. The predicted substitution data may be based in large part on unoccluded data.

[0112] In particular, the proposed inpainting system may utilize machine learning techniques to better refine predicted replacement data that may be inpainted into an image. Training of the occlusion model and / or inpainting model may involve a ground truth image, an augmented image, and a mask. The ground truth image may be an image without garbage data. The input image may be a ground truth image with a portion of the image obscured by garbage data (e.g., the garbage data may be added to the ground truth image data to generate the augmented image data). The mask (e.g., a binary pixel mask) may indicate the location and / or size of the garbage data in the augmented image data. The mask may be generated automatically or may be generated at least in part based on one or more user inputs.

[0113]

[0013] As an example, the proposed repair system may utilize a machine learning autoencoder model to perform prediction of the replacement image data. The autoencoder model may be, for example, a conditional variational autoencoder. In some implementations, the autoencoder model may include an encoder model configured to encode input image data to generate encoded data, and a decoder model configured to predict the replacement data based on the encoded data generated by the encoder.

[0114] In some implementations, training of the occlusion model can begin by generating input image data from ground truth image data (e.g., by adding garbage data to the ground truth image data). A mask (e.g., a binary pixel mask) can indicate the location of the garbage data in the augmented image data.

[0115] The input image may then be input to an encoder together with the mask to generate encoded data, sometimes referred to as an embedding, as the encoder's output. In some implementations, during training, the ground truth image and the mask are also encoded by using a second, different encoder. The encoded ground truth image may be used to create a distribution value or feature vector to be used to aid the decoding prediction process to narrow prediction candidates. In some implementations, the distribution value may be multiplied by a random value to require the decoder to rely on both the embedding and the feature vector when decoding and predicting (e.g., to generalize the decoder model).

[0116] The encoded data created from the input image data using the mask and combined with the feature vector can be input to a decoder, which can decode the data to create a replacement image. In other words, the decoder can predict replacement data that will replace the junk data in an attempt to match the ground truth data.

[0117] In particular, after an image is decoded, the permuted image may be evaluated against the ground truth image using any number and / or combination of different loss functions. Three exemplary loss functions that may be used include an L1 loss function, a VGG loss function, and an adversarial loss function. After evaluation, a correction or update step may be performed to update parameters (e.g., of the encoder model and / or the decoder model) based on the loss function. Training may be repeated iteratively over several ground truth and input image training examples.

[0118] Once training is complete, the system can be operated to generate replacement data for portions of the input image identified by the mask as unwanted. Specifically, at inference time, a new input image having some unwanted data can be provided to the trained encoder model along with a mask that identifies the location and / or size of the unwanted data within the input image. The encoder can produce encoded data (e.g., sometimes referred to as "embedding") based on the input image and the mask.

[0119] Furthermore, in some implementations, a conditioning vector (which may be, for example, a zero vector in some cases) may be included with (e.g., concatenated with) the embedding data generated by the encoder from the input data and the mask. Because the system may be trained using randomized feature vectors, a well-trained system can produce reasonable results. The system may use the trained parameters to create replacement data in place of the junk data.

[0120] Thus, at inference time, image data having unwanted image data may be input to an encoder along with a mask that identifies the unwanted image data. The encoded image data may then be input to a decoder along with a conditioning vector. The decoder may output a replacement image in which the unwanted image data has been removed and replaced with replacement data (e.g., depicting a portion of the scene previously obstructed by the unwanted image data).

[0121] A method for inpainting using machine learning to accurately replace unwanted data with predicted replacement data enables the removal of unwanted objects and people from pictures or videos. The same need may be found in other forms of media, including audio waveform data (e.g., it may be desirable to remove unwanted noises such as clicks, hisses, etc., or to isolate a single speaker by removing audio data corresponding to other speakers or background noise). The unwanted data may be replaced with predicted data based on characteristics of the remaining data. Thus, while the systems and methods are described herein with respect to visual image data such as scenes, they may also be applied to other types or modalities of data (e.g., audio data, audio data / sound images, text data, text images, etc.) in which replacement data is predicted to replace the unwanted data. Furthermore, the image data may include two-dimensional image data (e.g., photographs) or three-dimensional image data (e.g., mesh models or point clouds, such as lidar point clouds). For example, in the case of a point cloud, a mask may indicate which points in the point cloud are unwanted. More generally, for various other modalities, a mask may indicate which portions of the data are unwanted.

[0122] Because repair and data replacement can be non-deterministic, the predictions required for creating replacement data can be difficult. Machine learning can be one method for training a system to more accurately predict correct replacement data. The trained prediction system can then be utilized to create the most accurate replacement data. Training using ground truth data and input data can allow the system to evaluate and modify system parameters to more accurately predict what is being disturbed by the junk data. The use of ground truth data for training means that the training process is not non-deterministic.

[0123] The process of removing unwanted image data from image data may be referred to as inpainting. Machine learning models may be implemented in the system or process to provide increasingly accurate and efficient results for automated inpainting. For example, in some implementations, inpainting may be achieved through the use of a conditional variational autoencoder.

[0124] In some implementations, the system or method may utilize a conditional variational autoencoder for dense prediction in conjunction with a discriminator component, which separates the entire image data into two areas: a present area and a missing area. The conditional variational autoencoder may use ground truth information in addition to the embedding feature vectors of the variational autoencoder. The conditional variational autoencoder may use image pixels outside of the unwanted image data to aid in prediction.

[0125] In some implementations, the ground truth image data may be utilized to train an occlusion model and / or an inpainting model. In some implementations, the training includes taking input image data using an encoder along with a mask indicating the size and location of the unwanted image data, outputting embedded data, taking the ground truth image along with the mask by another encoder, outputting a feature vector, randomizing the feature vector, inputting the embedded data and the randomized feature vector to a decoder, outputting replacement image data, evaluating the replacement image data against the ground truth image, and modifying parameters of an operation based on the evaluation of the replacement image data against the ground truth image data.

[0126] In some implementations, the ground truth data may be data that does not include junk data, may be an ideal outcome of the replacement data created by the system, or may be a useful data set for determining the accuracy of an inpainting method or system.

[0127] In some implementations, the augmented data may include junk data. In some implementations, the junk data may be data that obscures the ground truth data. For example, junk data left after an image undergoes a denoising process, or a person obscuring the scenery in a picture.

[0128] In some implementations, the augmented data may be a created data set. The augmented data may be created by adding garbage data into a set of ground truth data. For example, some pixels of the set of ground truth image data may be disturbed by adding color blemishes or other objects. The color blemishes may be considered garbage data, and therefore, an inpainting system may be used to remove and replace the color blemishes or other objects.

[0129] In some implementations, a repair system or method may include a mask. The mask may be an indicator of the size and location of the garbage data. The mask may indicate what needs to be replaced, and may be used to isolate what data is part of the desired dataset. In some implementations, an inverse mask may be created for discriminant training.

[0130] In some implementations, the mask may be created manually by a user, in some implementations, the mask may be created automatically, in some implementations, the automated creation of the mask may be performed by a system trained to create masks using a machine learning model (e.g., a segmentation model).

[0131] In some implementations, a machine learning model may be utilized to train and provide rules for the repair system. One example of a machine learning model that may be trained and implemented may be a conditional variational autoencoder. For example, the system may have an upper variational encoder pipeline and a lower encoder pipeline. For example, the upper pipeline may include an encoder for encoding the augmented data and mask to create embedding data, and the lower pipeline may include an encoder for encoding ground truth data to create feature vectors. In some implementations, the upper and lower pipelines may converge. The embedding data with the guide of the feature vectors may produce replacement data when input to a decoder.

[0132] In some implementations, the feature vector may include a distribution value. The distribution value may be a standard deviation value and a mean value. In some implementations, the distribution value may be randomized to ensure that the decoder does not rely solely on the feature vector to predict the substitution data.

[0133] In some implementations, the conditioning vector may be a zero vector, which may provide reasonable predictive data due to training with the categorized feature vectors.

[0134] In some implementations, the evaluation of the permutation data against the ground truth data may be quantified by a loss function. The loss functions may be used individually or in any combination. For example, an L1 loss function, a VGG loss function, and / or an adversarial loss function may be used in combination to evaluate the model's predictions. Evaluation may also be completed using any of the three loss functions individually. In some implementations, a KL divergence loss function can assist in evaluating the training. For example, the KL divergence loss function may cause the second term to trend toward zero. A trend toward zero may indicate system improvement and that the system is getting closer to being optimized.

[0135] In some implementations, the discriminator method or system involves two levels: a semantic level and a texture level. The semantic level can relate to understanding the data as a whole. The texture level can relate to finer details of the predicted data, including the sharpness of the replacement data.

[0136] In some implementations, inpainting systems and methods may be applied to 3D point cloud editing. One or more points of the point cloud may be junk data and may need to be removed or replaced. In some implementations, an inpainting system or method for a 3D point cloud may be trained using a ground truth 3D point cloud, an input 3D point cloud, and a mask. The input 3D point cloud may be a ground truth 3D point cloud with the addition of junk data. The junk data may be out-of-place points, unwanted points, or some other form of distracting data. The mask may be an indicator of the location of the junk data.

[0137] In some implementations, repair systems and methods may be applied in speech recognition to fill in areas of the received speech segment that have poor audio quality. The filled speech segment may then be provided as input to the speech recognition system. Improving the audio quality of the speech segment provided to the speech recognition system may lead to a higher accuracy of the speech recognition process and / or may allow speech recognition to be used with speech segments that have poor audio quality.

[0138] In some implementations, restoration systems and methods may be applied to colorize black-and-white photographs. For example, in some implementations, a set of black-and-white photographs may be colorized manually or with computer assistance. For example, to train the colorization system, a set of manually colorized black-and-white photographs may be input to the restoration system as ground truth data, and the original black-and-white photographs may be input instead of the augmented data. The colorization system may use this data sample for training. Once training is complete, the system may produce colorized images from old black-and-white photographs that were not previously colorized.

[0139] In some implementations, the inpainting system may replace or be used in conjunction with a denoising variational autoencoder. For example, a denoising variational autoencoder may remove noise from a set of data, and the inpainting system may remove and replace junk data left after the denoising process.

[0140] In some implementations, the repair system may be a web application. In some implementations, the system may be an offline desktop application. Additionally, the system may be a mobile application. In some implementations, the system may be an add-on or extension for another application. The system may be a built-in feature of a larger application. In another example, the system may be provided as a service (e.g., as a service layer and / or by a server computing device). In some implementations, automated mask creation may be built into the same application as the repair system.

[0141] In some implementations, a conditional variational autoencoder may include two autoencoders. Only the lower encoder may be utilized in training by generating feature vectors by encoding ground truth image data. The feature vectors may be penalized by a KL divergence loss function to require the conditional variational autoencoder not to rely solely on the feature vectors. Randomization of the feature vectors may still provide useful information for decoder prediction.

[0142] The upper encoder may encode input image data and a mask. The input image data may be ground truth image data cluttered with objects or other unwanted image data. The encoded input image data and mask may be added to the feature vector from the lower encoder. The added data may then be decoded to generate replacement image data. The operation may be implemented as concatenation.

[0143] In some implementations, the inference process may include a conditional variational autoencoder that has an upper encoder but no lower encoder. The lower encoder may be replaced with a conditioning vector, such as a zero vector, which may produce reasonable image data due to randomized feature vector training.

[0144] In some implementations, the system or method may include large-scale information because the upper encoder is the primary source for prediction data. In some implementations, the upper encoder and decoder may have skip connections in a convolutional neural network.

[0145] In some implementations that use an adversarial loss during training, the discriminator model used to generate the adversarial loss may be separated into two levels: texture and semantic. The discriminator may have two separated layers. The discriminator may assist in distinguishing real image data from replacement image data generated by the decoder. In some implementations, the input image resolution may be changed. For example, the input image resolution may be changed from 256 pixels by 256 pixels to 16 pixels by 16 pixels. The 16x16 image may be a receptive field to address the texture of the replacement area. The image may be isolated with the aid of a segmented image mask. The semantic component of the model may view the image data as a whole. Thus, the predicted replacement data may be aided by both the texture and semantic components of the discriminator.

[0146] In some implementations, the discriminator model may include two texture level networks and a semantic level network. The first texture level network may process a portion of the ground truth image data at a location indicated by the mask and output a first texture discriminator output. The second texture level network may process a portion of the predicted image data at a location indicated by the mask and output a second texture discriminator output. The semantic level network may include a sharing network. In some implementations, the sharing network may process the ground truth image data with garbage data removed therefrom to generate the semantic discriminator output. In some implementations, the semantic level network may utilize an inverse mask for discriminator processing. The semantic level network may generate the discriminator output based on the first texture discriminator output, the second texture discriminator output, and the semantic discriminator output.

[0147] In some implementations, the inpainting problem may be addressed by using a variable encoder pipeline, dual-coding discriminative training, and / or human perceptual loss. A system or method may implement these functions individually or in any combination. The variable encoding pipeline may include ground truth image data being used as input to train an inpainting model. The model may include a noise-varying model to address potential issues with the noise magnitude of the predicted replacement data. The dual-coding discriminative training may first address the texture level of the isolated unwanted image data, and then address semantic-level data to discriminate between ground truth data and predicted data when training. In some implementations, the inpainting model may further include a texture synthesis step to address any extreme values ​​generated by the prediction step.

[0148] Exemplary Methods 6 shows a flowchart diagram of an exemplary method for performing, according to an exemplary embodiment of the present disclosure. While FIG. 6 shows steps performed in a particular order for purposes of illustration and explanation, the method of the present disclosure is not particularly limited to the order or arrangement shown. Various steps of method 600 can be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0149] At 602, a computing system may receive an input image and a ground truth image. The input image may be generated by adding one or more occlusions to the ground truth image. In some implementations, the ground truth image may depict a scene, and the input image may depict a scene with one or more occlusions. The one or more occlusions may include a human, animal, or object obstructing a portion of the scene.

[0150] At 604, the computing system may process the ground truth images with a context attention model to generate a context attention output. The context attention model may include a convolutional neural network and one or more context attention blocks. Training the context attention model may include processing one or more training images with the context attention model to generate a training context attention output. The training context attention output and the input image may then be processed with an inpainting model to generate training augmented images. A training loss function may then be evaluated based on differences between the training augmented images and the ground truth images. One or more context attention parameters may then be adjusted based at least in part on the training loss function.

[0151] At 606, the computing system may process the input image and the context attention output using the augmented model to generate a predicted image. The predicted image may include content of the input image with one or more occlusions replaced with predicted pixels. The augmented model may include multiple sub-blocks including a prediction model and a blending model. The prediction model may process the input image to generate predicted context attention data. The blending model may process the predicted context attention data and the context attention output to generate blending data. The blending model may be trained to randomly blend the predicted context attention data and the context attention output. In some implementations, the augmented model may include an occlusion model for processing the blending data and the input image to generate the predicted image.

[0152] At 608, the computing system can evaluate a loss function that evaluates the difference between the predicted image and the ground truth image. The loss function can include a perceptual loss, a GAN loss, a VGG loss, and / or a KL divergence loss.

[0153] At 610, the computing system may adjust one or more parameters of the augmented model based at least in part on the loss function. In some implementations, the one or more parameters are at least one of a prediction model, a blending model, or an occlusion model.

[0154] 7 shows a flowchart diagram of an exemplary method for performing, according to an exemplary embodiment of the present disclosure. While FIG. 7 shows steps performed in a particular order for purposes of illustration and explanation, the methods of the present disclosure are not particularly limited to the order or arrangement shown. Various steps of method 700 can be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0155] At 702, a computing system may obtain an input image. The input image may include one or more objects for replacement. The objects may be depicted using multiple pixels in the input image and may be recognized by object recognition techniques.

[0156] At 704, the computing system may obtain one or more user inputs. The one or more user inputs may include one or more contextual attention inputs. The one or more user inputs may include one or more selections of portions of the input image. The one or more user inputs may include lasso gesture inputs, brush gestures, and / or tap gestures. In some implementations, the user inputs may include a first set of user inputs for selecting portions of the input image to replace and a second set of user inputs for selecting portions of the image to clone for image augmentation.

[0157] At 706, the computing system can process the input image and one or more user inputs using an augmented model to generate an augmented image. The augmented model can be trained using ground truth training. The augmented image can include a scene of the input image with one or more objects replaced with predicted pixels. The augmented model can include a blending model trained to blend predictive contextual attention data and user-input contextual attention data, where the predictive contextual attention data can be generated based at least in part on the input image and the user-input contextual attention data can be based at least in part on one or more user inputs. Blending weights for the blending model can be manually selected by a user or automatically determined.

[0158] At 708, the computing system can provide the augmented image to the user. The augmented image can be sent to the user computing device and / or provided for display on a visual display of the user computing device. The augmented image can be provided for display as part of the visual interface. In some implementations, the visual interface can provide an input image for display and receive user input via the user interface (e.g., a selection on a touchscreen display while providing a visual display). In some implementations, a representation of one or more user inputs can be provided as part of the visual interface, which representation can be overlaid on the input image.

[0159] 8 shows a flowchart diagram of an exemplary method for performing, according to an exemplary embodiment of the present disclosure. While FIG. 8 shows steps performed in a particular order for purposes of illustration and explanation, the methods of the present disclosure are not particularly limited to the order or arrangement shown. Various steps of method 800 can be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0160] At 802, the computing system may receive input data from a user. The input data may include one or more images. In some implementations, the input data may include audio data. In some implementations, the one or more images may be frames of a video.

[0161] At 804, the computing system can obtain contextual attention data. The contextual attention data can include selected pixel data. The contextual attention data can include a flattened input tensor. In some implementations, the contextual attention data can be generated based at least in part on a lasso gesture input by a user.

[0162] At 806, the computing system can process the input data with the predictive model to generate predictive contextual attention data.

[0163] At 808, the computing system can process the contextual attention data and the predicted contextual attention data with the blending model to generate blended data.

[0164] At 810, the computing system can process the input data and the blended data with an occlusion model to remove one or more objects from one or more images. In some implementations, the output of the occlusion model can include an augmented video output. In some implementations, the input data and the blended data can be processed to remove occlusion from audio data. In some implementations, an occlusion model can be trained using the occlusion model.

[0165] The computing system may add one or more pixels back into one or more images to generate a rendered image.

[0166] Additional Disclosures The technology described herein refers to servers, databases, software applications, and other computer-based systems, as well as actions taken on and information sent to such systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functionality among components. For example, the processes described herein may be performed using a single device or component or multiple devices or components acting in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate serially or in parallel.

[0167] While the present subject matter has been described in detail with reference to various specific exemplary embodiments thereof, each example is provided by way of explanation, not limitation, of the present disclosure. Those skilled in the art, once they arrive at the above understanding, will be able to readily create modifications, variations, and equivalents of such embodiments. Accordingly, the present disclosure does not exclude the inclusion of such modifications, variations, and / or additions to the present subject matter as would be readily apparent to one skilled in the art. For example, features illustrated or described as part of one embodiment may be used with another embodiment to yield yet a further embodiment. Accordingly, it is intended that the present disclosure cover such modifications, variations, and equivalents. [Explanation of symbols]

[0168] 10. Computing Devices 50 computing devices 100 Computing Systems 102 User Computing Devices 112 processors 114 memory 116 Data 118 Command 120 Extended Models, Machine Learning Models 122 User Input Components 130 Server Computing System 132 processors 134 memory 136 Data 138 Command 140 Extended Model, Machine Learning Extended Model 150 Training Computing System 152 processors 154 memory 156 Data 158 Command 160 Model Trainer 162 training data, training dataset 180 Network 200 Expansion Model 202 Input Data 204 Ground truth data, ground truth images 210 Image Augmentation Model 212 Inpainting model, occlusion model 214 Predictive Model 216 Blend Model 220 Context Attention Model 222 Repair Model 224 Training Contextual Attention Loss Function 226 Output data, forecast data 228 Loss Function 300 Expansion Model 302 Input Data 304 Contextual Attention Data 306 Extended Model, Repair Extended Model 308 Repair Tasks 310 Output Data, Extended Data 400 Expansion Model 402 Input Data 404 User-entered data 410 Extended Model 412 Occlusion Model 414 Predictive Context Attention Model, Predictive Context Attention Data, Predictive Model 416 Blend Model 420 Extended Data 500 Contextual Attention Model 502 input data, flattened input tensor, input tensor 504 Flattening Input Tensors 506 Contextual Attention Matrix 508 Transformed Data 510 Output data, output tensor 900 Repair Process 902 First Image 904 Second Image 906 Third Image 908 source pixels 910 Humans 912 Plane, Source Pixel 914 source pixels 1000 Repair Results 1010 User Input 1012 target pixels 1100 Repair results 1106 Output Image 1108 Objects 1110 Objects 1112 Objects 1114 source pixels 1200 Repair results 1202 Overlay Interface 1204 input images 1206 Extended Images 1208 source pixels 1210 Occlusion

Claims

1. 1. A computer-implemented method for training an inpainting model, comprising: receiving, by a computing system comprising one or more processors, an input image and a ground truth image, wherein the ground truth image depicts a scene and the input image depicts the scene having one or more occlusions; processing, by the computing system, the ground truth images with a context attention model to generate a context attention output; processing, by the computing system, the input image and the context attention output using an augmented model to generate a predicted image; evaluating, by the computing system, a loss function that evaluates the difference between the predicted image and the ground truth image; adjusting, by the computing system, one or more parameters of the augmented model based at least in part on the loss function; 1. A computer-implemented method comprising:

2. the augmented model comprises a prediction model, a blending model, and an occlusion model, and processing the input image and the context attention output using the augmented model includes: processing, by the computing system, the input image using the predictive model to generate predictive contextual attention data; processing, by the computing system, the predicted contextual attention data and the contextual attention output using a blending model to generate blended data; processing, by the computing system, the blended data and the input image to generate the predicted image.

10. The computer-implemented method of claim 1.

3. 3. The computer-implemented method of claim 2, wherein the blending model is trained to randomly blend the predicted contextual attention data and the contextual attention output.

4. 4. The computer-implemented method of claim 1, wherein the input image is generated by adding one or more occlusions to the ground truth image.

5. 5. The computer-implemented method of claim 1, wherein the context attention model comprises a convolutional neural network and one or more context attention blocks.

6. processing, by the computing system, one or more training images with the context attention model to generate a training context attention output; processing, by the computing system, the training context attention output using an inpainting model to generate training augmented images; evaluating, by the computing system, a training loss function that evaluates the difference between the training augmented images and the ground truth images; and adjusting, by the computing system, one or more context attention parameters of the context attention model based at least in part on the training loss function; The context attention model is trained.

6. A computer-implemented method according to any one of claims 1 to 5.

7. receiving, by the computing system, one or more inputs describing a selection of a portion of the input image; the predicted image is generated based at least in part on the one or more inputs.

7. A computer-implemented method according to any one of claims 1 to 6.

8. A computing system configured to perform the computer-implemented method of any one of claims 1 to 7.

9. A computer program configured to cause a computing system to perform the computer-implemented method of any one of claims 1 to 7.

10. 1. A computing system for repair, the computing system comprising: one or more processors; and one or more computer-readable storage media collectively storing instructions that, when executed by the one or more processors, cause the computing system to perform operations, including: obtaining an input image, the input image comprising one or more objects for replacement; obtaining one or more user inputs, the one or more user inputs comprising one or more contextual attention inputs; processing the input image and the one or more user inputs with an augmentation model to generate an augmented image, the augmentation model being trained to replace the one or more objects; providing the augmented image to a user; The augmented model is trained by a computer-implemented method for training an augmented model, the method comprising: receiving, by a computing system comprising one or more processors, an input image and a ground truth image, wherein the ground truth image depicts a scene and the input image depicts the scene having one or more occlusions; processing, by the computing system, the ground truth images with a context attention model to generate a context attention output; processing, by the computing system, the input image and the context attention output using an augmented model to generate a predicted image; evaluating, by the computing system, a loss function that evaluates the difference between the predicted image and the ground truth image; and adjusting, by the computing system, one or more parameters of the augmented model based at least in part on the loss function. Computing system.

11. The operation is obtaining one or more second user inputs, the one or more second user inputs describing a selection of a portion of the input image; the augmented image is generated based at least in part on the one or more second user inputs.

11. The computing system of claim 10.

12. 12. The computing system of claim 10 or 11, wherein the augmented model comprises a blending model trained to blend predictive contextual attention data and user-inputted contextual attention data, the predictive contextual attention data being generated based at least in part on the input image, and the user-inputted contextual attention data being based at least in part on the one or more user inputs.

13. 13. The computing system of claim 12, wherein the predictive contextual attention data and the user-entered contextual attention data are blended based at least in part on blending weights, the blending weights being manually controlled by a user.

14. 13. The computing system of claim 12, wherein the predicted contextual attention data and the user-entered contextual attention data are blended based at least in part on blending weights, the blending weights being adjusted during ground truth training.

15. The operation is providing the input image for display to provide a visual interface to the user. A computing system according to any one of claims 10 to 14.

16. 16. The computing system of claim 10, wherein the one or more user inputs are obtained using a touchscreen display.

17. The operation is providing a representation of the one or more user inputs superimposed on the input image.

17. A computing system according to any one of claims 10 to 16.

Images