Machine Learning for High-Quality Image Processing

Abstract

A system or method for image restoration can be developed using machine learning and real-value data training. Using real-value image data to train machine-learned restoration models can increase efficiency and accuracy in image restoration. Furthermore, machine-learned restoration models can assist with non-deterministic predictions of various data types and are applicable to the removal and / or replacement of various data types. Due to calibration parameters adjusted through training, the trained model is able to make predictions even without the guarantee of real-value data.

Description

Technical Field

[0001] The present disclosure relates generally to processing image data and more particularly to a machine learning model for high-quality image restoration that can be trained with real-valued data. Background Art

[0002] Images (e.g., photographs) and other forms of data often include unwanted data. As one example, unwanted data may correspond to artifacts resulting from processing an image to reduce noise in the image. As another example, unwanted data may correspond to humans in the foreground of a landscape or unknown people in the background of a family photo. As another example, unwanted data may correspond to unsightly objects in an otherwise original background.

[0003] Thus, the unwanted data can correspond to an object that occludes or obscures other portions of the image, such as the depicted scene. However, replacing the unwanted data with replacement data (e.g., replacement image data depicting the occluded portion of the image occluded by the unwanted data, a process also known as "inpainting") is a challenging problem that is inherently non-deterministic. In other words, multiple possible solutions can be determined from the same image, leading to a difficult problem. Summary of the Invention

[0004] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the description which follows, or can be learned from the description, or can be learned through practice of the embodiments.

[0005] The present disclosure provides systems and methods for replacing unwanted data with replacement data based on data characteristics and true value training techniques. A computing system according to the present disclosure can be configured to receive a given set of augmented data, a mask, and a set of true value data; encode the augmented data and the mask; encode the true value data; compile the data received from both encodings; decode the encoding; compare the output with the true value data; and modify system parameters. By using true value encoding, the computing system can be further assisted in replacing data. In this way, embodiments of the present disclosure can be used to create replacement data to replace unwanted data.

[0006] One example aspect of the present disclosure is directed to a computer-implemented method for training a machine learning image restoration model. The method can include a conditional variational autoencoder. The method can include obtaining a training sample, the training sample comprising: true-value image data, enhanced image data derived from adding unwanted image data to the true-value image data, and a mask that can indicate one or more locations of the unwanted image data within the enhanced image data. The method can further include processing the enhanced data and the mask using a first encoder model of the conditional variational autoencoder to generate an embedding for the image data, and can include processing the true-value image data and the mask using a second encoder model to generate one or more distribution values. In addition, the method can include processing the embedding and the one or more distribution values ​​using a decoder model of the conditional variational autoencoder to generate predicted image data, the predicted image data can include replacement image data at the one or more locations indicated by the mask, wherein the replacement image data can replace the unwanted image data. Additionally, the method can include evaluating one or more loss functions based on a comparison of the predicted image data with the true-valued image data, and then modifying one or more parameter values ​​of the conditional variational autoencoder based at least in part on the one or more loss functions.

[0007] Another example aspect of the present disclosure is directed to a computing system comprising at least one processor, a machine-learned image inpainting model, and at least one tangible, non-transitory computer-readable medium storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations. The system can include an encoder configured to encode image data and a decoder configured to decode the image data. The machine-learned image inpainting model can be trained to input image data and a mask into the encoder, wherein the image data can include unwanted image data and wherein the mask can indicate the location and size of the unwanted image data; further, the machine-learned image inpainting model can be trained to receive an embedding from the encoder, wherein the embedding can include the encoded image data. The machine-learned image inpainting model can be trained to input the embedding and a conditional vector into the decoder. The machine-learned image inpainting model can further be trained to receive predicted image data as output from the decoder, wherein the predicted image data can replace the unwanted image data with predicted replacement data based at least in part on the image data and the conditional vector.

[0008] Another example aspect of the present disclosure is directed to one or more non-transitory computer-readable media that can collectively store instructions that cause one or more computing devices to perform operations. The operations can include a conditional variational autoencoder. The operations can include obtaining a training sample that includes true-valued data, augmented data derived from adding unwanted image data to the true-valued image data, and a mask that can indicate one or more locations of the unwanted data within the augmented data. The operations can further include processing the augmented data and the mask using a first encoder model of the conditional variational autoencoder to generate an embedding for the data, and can include processing the true-valued data and the mask using a second encoder model to generate one or more distribution values. In addition, the operations can include processing the embedding and the one or more distribution values ​​using a decoder model of the conditional variational autoencoder to generate predicted data, the predicted data can include replacement data at the one or more locations indicated by the mask, wherein the replacement image data can replace the unwanted data. Additionally, the operations can further include evaluating one or more loss functions based on the comparison of the predicted data with the true-valued data, and then modifying one or more parameter values ​​of the conditional variational autoencoder based at least in part on the one or more loss functions. The operations can further include evaluating one or more loss functions based on the comparison of the predicted image data with the true-valued data. Furthermore, the operations can include modifying one or more parameter values ​​of the conditional variational autoencoder based at least in part on the one or more loss functions.

[0009] Other aspects of the present disclosure are directed to various systems, devices, non-transitory computer-readable media, user interfaces, and electronic devices. These and other features, aspects, and advantages of various embodiments of the present disclosure will be better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] A detailed discussion of embodiments for those of ordinary skill in the art is set forth in the specification with reference to the accompanying drawings, in which:

[0011] Figure 1A A block diagram of an example computing system is depicted, according to an example embodiment of the present disclosure.

[0012] Figure 1B A block diagram of an example computing device is depicted according to an example embodiment of the present disclosure.

[0013] Figure 1C A block diagram of an example computing device is depicted according to an example embodiment of the present disclosure.

[0014] Figure 2 Depicted is a block diagram of an example training process for training a model for machine learning, according to an example embodiment of the present disclosure.

[0015] Figure 3 Depicted is a block diagram of an example inference process for inferring replacement data using a machine learning model, according to an example embodiment of the present disclosure.

[0016] Figure 4 A flowchart depicts an example method of training a model for machine learning according to an example embodiment of the present disclosure.

[0017] Figure 5 A flowchart depicts an example method of using a machine learning model to infer replacement data according to an example embodiment of the present disclosure.

[0018] Figure 6 Depicted is a block diagram of an example dual-coded discriminative training method according to an example embodiment of the present disclosure.

[0019] Reference numerals that are repeated across multiple drawings are intended to identify like features in the various embodiments. DETAILED DESCRIPTION

[0020] Generally, the present disclosure relates to systems and methods for using machine learning to perform inpainting, which can refer to replacing unwanted data with replacement data. As an example, in the context of image data, inpainting can include removing humans or other undesirable objects from an image and filling in the image with the replacement data in the place of the removed data. According to aspects of the present disclosure, the replacement data can be predicted by a machine learning model, such as, for example, a conditional variational autoencoder. The predicted replacement data can be largely based on unoccluded data.

[0021] In particular, the proposed restoration system can utilize machine learning techniques to better refine the predicted replacement data that can be restored into the image. The training of the machine learning model can involve a true value image, an enhanced image, and a mask. The true value image can be an image without unwanted data. The enhanced image can be a true value image in which a portion of the image is obscured by unwanted data (e.g., unwanted data can be added to the true value image data to generate the enhanced image data). The mask (e.g., a binary pixel mask) can indicate the location and / or size of the unwanted data within the enhanced image data.

[0022] As an example, the proposed restoration system can utilize a machine-learned autoencoder model to perform prediction of replacement image data. The autoencoder model can be, for example, a conditional variational autoencoder. In some embodiments, the autoencoder model can include an encoder model configured to encode input image data to generate encoded data and a decoder model configured to predict replacement data based on the encoded data generated by the encoder.

[0023] In some embodiments, training of a machine learning model can begin by generating augmented image data from the real-valued image data (e.g., by adding unwanted data to the real-valued image data). A mask (e.g., a binary pixel mask) can indicate the location of the unwanted data within the augmented image data.

[0024] Next, the augmented image can be input into an encoder with a mask to generate encoded data, which can also be called an embedding, as the output of the encoder. In some embodiments, during training, the real-valued image and the mask are also encoded using a second different encoder. The encoded real-valued image can be used to create a distribution value or feature vector to help the prediction process of decoding to narrow down the prediction possibilities. In some embodiments, the distribution value can be multiplied by a random value to require the decoder to rely on both the embedding and the feature vector in decoding and prediction (e.g., to generalize the decoder model).

[0025] The encoded data generated from the masked augmented image data and combined with the feature vector can be input into a decoder. The decoder can decode the data to create a replacement image. In other words, the decoder can predict replacement data that replaces the unwanted data in an attempt to match the true value data.

[0026] Specifically, after the image is decoded, the replacement image can be evaluated against the true value image using any number of different loss functions and / or combinations of different loss functions. Three example loss functions that can be used include: L1 loss function, VGG loss function, and adversarial loss function. After evaluation, a modification or update step can be performed to update the parameters (e.g., of the encoder and / or decoder model) based on the loss function. Training can be iteratively repeated on multiple true value and augmented image training examples.

[0027] Once training is complete, the system can be run to generate replacement data for the unwanted portions of the input image identified by the mask. Specifically, at inference time, a new input image with some unwanted data can be provided to the trained encoder model along with a mask identifying the location and / or size of the unwanted data within the input image. The encoder can generate encoded data (e.g., which can also be referred to as an "embedding") based on the input image and the mask.

[0028] Furthermore, in some embodiments, a conditional vector (e.g., which in some instances can be a zero vector) can be included (e.g., concatenated) with the embedded data generated by the encoder from the input data and the mask. Since the system is trained using randomized feature vectors, a well-trained system can produce reasonable results. The system can use the trained parameters to create replacement data that replaces unwanted data.

[0029] Thus, at inference time, image data having unwanted image data can be input into an encoder along with a mask identifying the unwanted image data. The encoded image data can then be input into a decoder along with a conditional vector. The decoder can output a replacement image in which the unwanted image data has been removed and replaced with replacement data (e.g., depicting a portion of a scene previously obscured by the unwanted image data).

[0030] A method for inpainting using machine learning that accurately replaces unwanted data with predicted replacement data allows for the removal of unwanted objects and people from a picture or video. Similar needs can be found in other forms of media including audio waveform data (e.g., it may be desirable to remove unwanted noises such as clicks, hisses, etc., or it may be desirable to isolate a single speaker by removing audio data corresponding to other speakers or background noise). The predicted data can be used to replace unwanted data based on attributes in the remaining data. Thus, although the systems and methods are described herein with reference to visual image data such as a scene, they can also be applied to other types or modalities of data (e.g., audio data, audio data / sound images, text data, text images, etc.), where the replacement data is predicted to replace the unwanted data (e.g., as identified via a mask). Furthermore, the image data can include two-dimensional image data (e.g., a photograph) or three-dimensional image data (e.g., a mesh model or point cloud, such as, for example, a LiDAR point cloud). For example, for a point cloud, a mask can indicate which points in the point cloud are unwanted. More generally, for various other modalities, the mask may indicate which parts of the data are unwanted.

[0031] Because image restoration and data replacement can be non-deterministic, the predictions required to create replacement data are difficult. Machine learning can be one method for training the system to more accurately predict the correct replacement data. The trained prediction system can then be used to create the most accurate replacement data. Training with both ground truth and augmented data allows the system to evaluate and modify its parameters to more accurately predict what is being occluded by the unwanted data. Training with ground truth data means the training process is non-deterministic.

[0032] The process of removing unwanted image data from image data can be referred to as inpainting. Machine learning models can be implemented as systems or processes to provide increasingly accurate and efficient results for automated inpainting. For example, in some embodiments, inpainting can be accomplished by utilizing conditional variational autoencoders.

[0033] In some embodiments, the system or method can utilize a conditional variational autoencoder in conjunction with a discriminative component for dense prediction, wherein the discriminative component divides the entire image data into two regions, an existing region and a missing region. In addition to the feature vector embedded by the variational autoencoder, the conditional variational autoencoder can also use real-valued information. The conditional variational autoencoder can use image pixels other than the unwanted image data to assist in prediction.

[0034] In some embodiments, the ground truth image data can be used for machine learning training. In some embodiments, the training includes: ingesting augmented image data by an encoder having a mask indicating the size and location of unwanted image data, outputting embedded data, ingesting the masked ground truth image by another encoder, outputting a feature vector, randomizing the feature vector, inputting the embedded data and the randomized feature vector into a decoder, outputting replacement image data, evaluating the replacement image data relative to the ground truth image, and modifying parameters of an operation based on the evaluation of the replacement image data compared to the ground truth image data.

[0035] In some embodiments, true value data can be data that does not include unwanted data. True value data can be the ideal result of the replacement data created by the system. True value data can be a useful data set for determining the accuracy of a repair method or system.

[0036] In some embodiments, the augmented data may include unwanted data. In some embodiments, the unwanted data may be data that obscures the true value data. For example, unwanted data may be left after an image has been subjected to a denoising process, or a person in the image may be obscured by the scenery.

[0037] In some embodiments, the augmented data can be a created dataset. The augmented data can be generated by adding unwanted data to the true-value dataset. For example, several pixels of the true-value image dataset can be obscured by adding color blobs or other objects. The color blobs can be considered unwanted data, and therefore, the inpainting system can be used to remove and replace the color blobs or other objects.

[0038] In some embodiments, a mask can be included in a repair system or method. The mask can be an indicator of the size and location of unwanted data. The mask can be used to separate what needs to be replaced and what data is part of the desired data set. In some embodiments, an inverse mask can be created for use in discriminative training.

[0039] In some embodiments, the mask can be manually created by a user. In some embodiments, the mask can be automatically created. In some embodiments, the automatic creation of the mask can be performed by a system trained to create masks using a machine learning model (e.g., a segmentation model).

[0040] In some embodiments, a machine learning model can be used to train and provide rules for repairing the system. An example of a machine learning model that can be trained and implemented can be a conditional variational autoencoder. For example, the system can have an upper variational encoder pipeline and a lower encoder pipeline. For example, the upper pipeline can include an encoder for encoding enhanced data and masks to create embedded data, and the lower pipeline can include an encoder for encoding real-valued data to create feature vectors. In some embodiments, the upper and lower pipelines can converge. When the embedded data with the guidance of the feature vector is input into the decoder, it can generate replacement data.

[0041] In some embodiments, the feature vector may include a distribution value. The distribution value may be a standard deviation value and a mean value. In some embodiments, the distribution value may be randomized to ensure that the decoder does not rely solely on the feature vector to predict the replacement data.

[0042] In some embodiments, the conditional vector may be a zero vector. In some embodiments, the zero vector may provide reasonable prediction data due to the use of a variety of feature vectors for training.

[0043] In some embodiments, the evaluation of the replacement data relative to the true value data can be quantified by a loss function. The loss function can be used alone or in any combination. For example, the L1 loss function, the VGG loss function, and / or the adversarial loss function can be used in combination to evaluate the predictions of the model. The evaluation can also be completed using any of the three loss functions alone. In some embodiments, the KL divergence loss function can help evaluate the training. For example, the KL divergence loss function can have a second term that tends to zero. Tendency to zero can indicate improvement in the system and that the system is getting closer to being optimized.

[0044] In some embodiments, the discriminator method or system involves two levels: semantic level and texture level. The semantic level can be related to the understanding of the data as a whole. The texture level can be related to the finer parts of the predicted data including the sharpness of the replacement data.

[0045] In some embodiments, the repair system and method can be applied to 3D point cloud editing. One or more points in the point cloud may be unwanted data and may need to be removed or replaced. In some embodiments, a repair system or method for a 3D point cloud can be trained using a true-value 3D point cloud, an enhanced 3D point cloud, and a mask. The enhanced 3D point cloud can be a true-value 3D point cloud with unwanted data added. The unwanted data may be points that are out of place, unwanted points, or some other form of occluded data. The mask can be an indicator of the location of the unwanted data. In some embodiments, the repair system and method can be applied to speech recognition to fill in areas with low audio quality in a received speech segment. The filled speech segment can then be provided as input to a speech recognition system. Improving the audio quality of the speech segment provided to the speech recognition system may result in higher accuracy in the speech recognition process and / or allow speech recognition to be used with speech segments of low audio quality.

[0046] In some embodiments, the restoration systems and methods can be applied to colorize black-and-white photographs. For example, in some embodiments, a collection of black-and-white photographs can be colorized manually or with computer assistance. For example, to train a colorization system, a collection of manually colorized black-and-white photographs can be input as ground truth data into the restoration system, and the original black-and-white photographs can be input instead of the augmented data. The colorization system can be trained using this data sample. Once trained, the system can generate colorized images from older, previously uncolorized black-and-white photographs.

[0047] In some embodiments, the repair system can replace or be used in conjunction with a denoising variational autoencoder. For example, the denoising variational autoencoder can remove noise from a data set, and the repair system can remove and replace the unwanted data left after the denoising process.

[0048] In some embodiments, the image restoration system can be a web application. In some embodiments, the system can be an offline desktop application. Additionally, the system can be a mobile application. In some embodiments, the system can be an add-on or extension to another application. The system can be a built-in feature of a larger application. In another example, the system can be provided as a service (e.g., as a service layer and / or by a server computing device). In some embodiments, automated mask creation can be built into the same application as the restoration system.

[0049] In some embodiments, the conditional variational autoencoder can include two autoencoders. The lower encoder can be utilized in training by encoding only the real-valued image data to generate a feature vector. The feature vector can be penalized by a KL divergence loss function to require that the conditional variational autoencoder does not rely solely on the feature vector. Randomization of the feature vector can still provide useful information for decoder prediction.

[0050] The upper encoder can encode augmented image data and a mask. The augmented image data can be real-valued image data that is obscured by an object or other unwanted image data. The encoded augmented image data and mask can be added to the feature vector from the lower encoder. The added data can then be decoded to generate replacement image data. The operation can be implemented as a concatenation.

[0051] In some embodiments, the inference process may include a conditional variational autoencoder with an upper encoder, but without a lower encoder. The lower encoder may be replaced with a conditional vector, such as a zero vector. Due to the randomized feature vector training, the zero vector may produce reasonable image data.

[0052] In some embodiments, the system or method can include large information from the upper encoder as the primary source of prediction data. In some embodiments, the upper encoder and decoder can have skip connections within a convolutional neural network.

[0053] In some embodiments using an adversarial loss during training, the discriminator model used to generate the adversarial loss can be split into two levels: texture and semantics. The discriminator can have two separate layers. The discriminator can help distinguish real image data from replacement image data generated by the decoder. In some embodiments, the input image resolution can be changed. For example, the input image resolution can be changed from 256 pixels × 256 pixels to 16 pixels × 16 pixels. The 16 × 16 image can be the receptive field to resolve the texture of the replacement area. The image can be isolated with the help of a segmented image mask. The semantic component of the model can treat the image data as a whole. Therefore, the predicted replacement data can be helped by both the texture and semantic components of the discriminator.

[0054] In some embodiments, the discriminator model may include two texture-level networks and one semantic-level network. The first texture-level network may process a portion of the true-value image data at the location indicated by the mask and may output a first texture discriminator output. The second texture-level network may process a portion of the predicted image data at the location indicated by the mask and may output a second texture discriminator output. The semantic-level network may include a shared network. In some embodiments, the shared network may process the true-value image data from which unwanted data has been removed to generate a semantic discriminator output. In some embodiments, the semantic-level network may use an inverse mask for discriminator processing. The semantic-level network may generate a discriminator output based on the first texture discriminator output, the second texture discriminator output, and the semantic discriminator output.

[0055] In some embodiments, the image restoration problem can be solved by using a variable encoder pipeline, dual-encoding discriminative training and / or human perception loss. The system or method can implement these features individually or in any combination. The variable encoding pipeline can include real-valued image data used as input to train the restoration model. The model can include a noise change model to address potential issues with noise amplitude in the predicted replacement data. Dual-encoding discriminative training can first address the texture level of isolated unwanted image data, and then can address semantic level data to discriminate between real-valued data and predicted data in training. In some embodiments, the image restoration model can further include a texture synthesis step to address any extreme values ​​generated by the prediction step.

[0056] The systems and methods of the present disclosure provide a number of technical effects and benefits. As an example, the inpainting machine learning system can help computing performance by refining the parameters of the predictions made to create replacement data. As a result, the image inpainting performed can be of higher quality (e.g., more accurate) than previous techniques, which represents an improvement in the performance of the computing system. In addition, the proposed method can eliminate the need to create such a large range of predictions to be evaluated, as required by certain existing techniques. Eliminating the need to create a large number of different predictions can result in saving computing resources, such as processor usage, memory usage, and / or network bandwidth usage. The use of true value data also removes some confusion from training and makes training more efficient, thereby saving computing resources. The trained system can reduce the amount of computing resources utilized compared to previous systems.

[0057] Because the implementation of machine learning also eliminates the need to manually edit every occurrence of unwanted data in an image, further efficiency can be achieved. The system can also eliminate the need for coders to write long, extended code, run it, refine it, and continuously monitor performance.

[0058] Furthermore, the systems and methods described herein can be used in any process where images are used as input to a system to provide higher quality input images to the system. Non-limiting examples of possible applications include: medical images, such as x-rays or scans of patients; monitoring the condition of a piece of machinery, where images of a piece of machinery are acquired periodically and used to determine when the part may need repair or replacement; and autonomous vehicles, which make decisions about their route and speed based on acquired images of their surroundings.

[0059] Referring now to the accompanying drawings, example embodiments of the present disclosure will be discussed in greater detail.

[0060] Example devices and systems

[0061] Figure 1A A block diagram of an example computing system 100 that performs repair according to an example embodiment of the present disclosure is depicted. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 communicatively coupled via a network 180.

[0062] The user computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet computer), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0063] The user computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and can be a single processor or multiple processors operatively connected. The memory 114 can include one or more temporary or non-temporary computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 can store data 116 and instructions 118 that can be executed by the processor 112 to cause the user computing device 102 to perform operations.

[0064] In some embodiments, the user computing device 102 can store or include one or more repair models 120. For example, the repair model 120 can be or can otherwise include various machine learning models, such as neural networks (e.g., deep neural networks) or other types of machine learning models, including nonlinear models and / or linear models. The neural network can include a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks. Figure 2 and Figure 3An example repair model 120 is discussed.

[0065] In some implementations, the one or more image inpainting models 120 can be received from the server computing system 130 via the network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing device 102 can implement multiple parallel instances of a single inpainting model 120 (e.g., to perform parallel generation of predicted replacement data across multiple instances of unwanted data in a data set).

[0066] More specifically, the inpainting model can have a training module that uses a training data set to train the model's parameters to optimize the generation of predicted data. The training module can rely on ground truth data to increase the efficiency and accuracy of the training module. Training can include creating augmented data from ground truth data by adding unwanted data to the ground truth data. Masks can also be used during training to provide labels for the size and location of unwanted data.

[0067] The repair model can obtain machine learning data from the training module to assist the inference module. The inference module can ingest user data, where the user data includes unwanted data. The inference module can then generate replacement data based on the user data and the mask, where the replacement data includes predicted data that replaces the unwanted data. The server can contain the machine learning data to assist in generating the predicted data.

[0068] Additionally or alternatively, one or more image restoration models 140 can be included in or otherwise stored and implemented by a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the restoration models 140 can be implemented by the server computing system 140 as part of a web service (e.g., an image editing service). Thus, one or more models 120 can be stored and implemented at the user computing device 102, and / or one or more models 140 can be stored and implemented at the server computing system 130.

[0069] The user computing device 102 can also include one or more user input components 122 for receiving user input. For example, the user input component 122 can be a touch-sensitive component (e.g., a touch-sensitive display or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component can be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0070] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and can be a single processor or multiple processors operatively connected. The memory 134 can include one or more temporary or non-temporary computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 can store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0071] In some implementations, server computing system 130 includes or is otherwise implemented by one or more server computing devices. Where server computing system 130 includes multiple server computing devices, such server computing devices can operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0072] As described above, the server computing system 130 can store or otherwise include one or more machine-learned repair models 140. For example, the model 140 can be or can otherwise include various machine-learned models. Example machine-learned models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Figure 2 and Figure 3 Discuss example model 140.

[0073] User computing device 102 and / or server computing system 130 can train models 120 and / or 140 via interaction with training computing system 150 communicatively coupled via network 180. Training computing system 150 can be separate from server computing system 130 or can be part of server computing system 130.

[0074] The training computing system 150 includes one or more processors 152 and a memory 154. The one or more processors 152 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and can be a single processor or multiple processors operatively connected. The memory 154 can include one or more temporary or non-temporary computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158 that can be executed by the processor 152 to cause the training computing system 150 to perform operations. In some embodiments, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0075] The training computing system 150 can include a model trainer 160 that trains the machine-learned models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130 using various training or learning techniques, such as, for example, backpropagation of errors. For example, a loss function can be backpropagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions can be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques can be used to iteratively update the parameters over multiple training iterations.

[0076] In some implementations, performing backpropagation of the error can include performing truncated backpropagation through time.The model trainer 160 can perform a number of generalization techniques (eg, weight decay, backoff, etc.) to improve the generalization capabilities of the model being trained.

[0077] In particular, the model trainer 160 can train the repair model 120 and / or 140 based on the training data set 162. The training data 162 can include, for example, a set of true value data, a set of augmented data, and a set of masks to indicate the size and location of unwanted data added to the corresponding true value data to create the corresponding augmented data.

[0078] In some implementations, if the user has provided consent, training examples can be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 can be trained by the training computing system 150 on user-specific data received from the user computing device 102. In some cases, this process can be referred to as personalizing the model.

[0079] The model trainer 160 includes computer logic for providing the desired functionality. The model trainer 160 can be implemented in hardware, firmware, and / or software that controls a general-purpose processor. For example, in some embodiments, the model trainer 160 includes a program file stored on a storage device, loaded into memory, and executed by one or more processors. In other embodiments, the model trainer 160 includes one or more sets of computer-executable instructions stored in a tangible computer-readable storage medium such as a RAM hard disk or optical or magnetic media.

[0080] The network 180 can be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and can include any number of wired or wireless links. In general, communications over the network 180 can be carried via any type of wired and / or wireless connection using a variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0081] Figure 1A The diagram illustrates an example computing system that can be used to implement the present disclosure. Other computing systems can also be used. For example, in some embodiments, the user computing device 102 can include a model trainer 160 and a training data set 162. In such embodiments, the model 120 can be trained and used locally at the user computing device 102. In some such embodiments, the user computing device 102 can implement the model trainer 160 to personalize the model 120 based on user-specific data.

[0082] Figure 1B Depicted is a block diagram of an example computing device 10 performing in accordance with an example embodiment of the present disclosure. Computing device 10 can be a user computing device or a server computing device.

[0083] Computing device 10 includes multiple applications (e.g., applications 1 to N). Each application contains its own machine learning library and machine learning model. For example, each application can include a machine learning model. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc.

[0084] like Figure 1B As shown, each application can communicate with multiple other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some embodiments, each application can communicate with each device component using an API (e.g., a public API). In some embodiments, the API used by each application is specific to that application.

[0085] Figure 1C Depicted is a block diagram of an example computing device 50 performing in accordance with an example embodiment of the present disclosure. Computing device 50 can be a user computing device or a server computing device.

[0086] The computing device 50 includes a plurality of applications (e.g., applications 1 through N). Each application communicates with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, and the like. In some embodiments, each application is capable of communicating with the central intelligence layer (and the models stored therein) using an API (e.g., a public API across all applications).

[0087] The central intelligence layer includes multiple machine learning models. For example, Figure 1C As shown, a corresponding machine-learned model (e.g., model) can be provided for each application and managed by the central intelligence layer. In other embodiments, two or more applications can share a single machine-learned model. For example, in some embodiments, the central intelligence layer can provide a single model (e.g., a single model) for all applications. In some embodiments, the central intelligence layer is included in or otherwise implemented by the operating system of the computing device 50.

[0088] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized data repository for computing devices 50. Figure 1C As shown, the central device data layer can communicate with multiple other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some embodiments, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0089] Example model layout

[0090] Figure 2 A block diagram depicts an example technique for training an example image inpainting model 200 according to an example embodiment of the present disclosure. In some embodiments, the inpainting model 200 is trained to receive a set of input data 202 describing augmented image data and, as a result of receiving the input data 202, provide output data 216, which can be replacement image data. Thus, in some embodiments, the inpainting model 200 can be or include a conditional variational autoencoder model trained to replace unwanted data with predicted replacement data.

[0091] In some embodiments, the training process for the restoration model can have an upper pipeline and a lower pipeline. The upper pipeline can receive the enhanced image data 202 and the mask 204 as input. The enhanced image data 202 can include unwanted data (e.g., that has been added to the ground truth image 206). In the example shown, the unwanted data is illustrated using a circle in the enhanced image data 202. The mask 204 indicates the location of the unwanted data within the enhanced image data 202. The upper pipeline can include an encoder 208 to encode the enhanced image data 202 and the mask 204 to create embedded image data.

[0092] The lower pipeline may include the true-valued image data 206 , the mask 204 , and an encoder 208 to encode the true-valued image data 206 and the mask 204 to create a feature vector that may be randomized 210 .

[0093] In some embodiments, the embedded image data and the randomized feature vector 210 can be compiled 212 (e.g., concatenated). The compiled data can be decoded by a decoder 214 to create predicted replacement image data 216. The predicted replacement image data 216 can then be evaluated against the ground truth image data 206 based on various loss functions 218, either individually or in combination. Three example loss functions 218 that can be used include: an L1 loss function, a VGG loss function, and / or an adversarial loss function. Figure 6 An example discriminator model that can be used to generate an adversarial loss is shown in .

[0094] In some embodiments, in addition to the above loss functions, a KL divergence loss function 220 may be used to evaluate the randomized feature vector 210. For example, the KL divergence loss function 220 may take the following form:

[0095]

[0096] In some embodiments, the KL loss 220 may be equal to the component in X The sum of all KL divergences between μ and the standard normal. In some embodiments, when μ i =0,σ i =1, the KL loss 220 can be minimized.

[0097] One or more parameters of the model 200 can be modified based on the evaluation data (e.g., based on the loss function 218 and / or 220). For example, the loss function can be back-propagated through the model, and the parameters of the model can be changed based on the gradient of the loss function. In some embodiments, this process can be done iteratively to train the model on multiple different training examples.

[0098] Figure 3A block diagram of an example repair model 300 performing inference according to an example embodiment of the present disclosure is depicted. The repair model 300 is similar to the repair model 300 except that the repair model 300 involves an inference process of the model. Figure 2 The inference process can occur after the system has undergone a round of training using real-value training techniques.

[0099] The inference process can start with a dataset 302 including unwanted data and a mask 304, and the model 300 can output a replacement dataset 316 in which the unwanted data is replaced with predicted data. In some embodiments, the inference process can replace the downstream pipeline of the training process with a conditional vector 310. The conditional vector 310 can be a zero vector.

[0100] In some embodiments, the inference process may involve encoding a set of data 302 and a mask 304 by an encoder 306 to create embedded data. The data set 302 may include unwanted data, and the mask 304 may indicate the size and location of the unwanted data. Furthermore, the condition vector 310 and the embedded data may be compiled 312 (e.g., concatenated). The compiled data 312 may be decoded by a decoder 314. The decoding by the decoder 314 may create a set of replacement data 316 based on the system's prediction.

[0101] Figure 6 A block diagram of an example dual-coded discriminative training method according to an example embodiment of the present disclosure is depicted. More specifically, in some embodiments, evaluating a repair model (e.g., Figure 2 The loss of the model 200) can include evaluating an adversarial loss generated based on a discriminator output generated by the discriminator model based on predicted image data and true-valued image data. Figure 6 An example discriminator model 600 is shown in FIG.

[0102] like Figure 6 As shown, the discriminator model 600 includes a texture level 602 and a semantic level 604. The texture level 602 includes a first texture level network 606 that processes a portion 610 of the true value image data 612 at one or more locations identified by a mask 614 to generate a first texture discriminator output 616. The texture level 602 also includes a second texture level network 608 that processes a portion 618 of the predicted image data 620 at one or more locations identified by the mask 614 to generate a second texture discriminator output 622.

[0103] The semantic level 604 includes a shared network 624 that processes the ground truth image data 612 from which unwanted data has been removed (e.g., shown at 626 and generated based on an inverse mask 628) to generate a semantic discriminator output 630. The discriminator model 600 generates a discriminator output 632 based on the first texture discriminator output 616, the second texture discriminator output 622, and the semantic discriminator output 630.

[0104] Figure 6 The proposed discriminator model 600 shown in takes full advantage of the fact that the task of inpainting can be divided into two levels: texture and semantics. The discriminator 600 has those two separate layers at 602 and 604.

[0105] In some embodiments, the first texture level network 606 and the second texture level network 608 can share the same weights. Similarly, the parts shown as triangles in level 604 can also share weights with each other (these can be referred to as semantic level networks).

[0106] In some embodiments, the first texture level network 606 and the second texture level network 608 can change the input image resolution from a first resolution (e.g., 256×256) to a smaller second resolution (e.g., 16×16), which means that the size of the pixel (e.g., 16×16) is the receptive field.

[0107] As designed, the texture level 602 can focus more on texture than semantics because most of the image area is masked, so there is less information about semantics. On the other hand, the shared network 624 can also focus on texture information, but should try to give semantic information to pass to the semantic level network. The semantic level network can receive the outputs 616, 622 and 630 and make a final discriminant decision to provide a discriminator output 632 (real or generated). An important part of the architecture is the shared network 624 that is able to focus on semantic meaning due to its surrounding connections. The final discriminator output 632 can be used (e.g., via backpropagation) to train an image restoration model.

[0108] Example Method

[0109] Figure 4 Depicted is a flow chart of an example method performed according to an example embodiment of the present disclosure. Figure 4 The steps of method 400 are described in a specific order for the purpose of illustration and discussion, but the method of the present disclosure is not limited to the specific order or arrangement shown. The steps of method 400 can be omitted, rearranged, combined and / or adjusted in various ways without departing from the scope of the present disclosure.

[0110] At 402, the computing system may include obtaining training examples. The training examples may be obtained using one or more computing devices. The training examples may include ground truth image data, enhanced image data, and a mask. The enhanced image data may be derived by adding unwanted image data to the ground truth image data, and the mask may indicate the location and size of the unwanted image data within the enhanced image data.

[0111] At 404, the computing system may include processing the enhanced image data, the true value image data, and the mask. In some embodiments, the processing may be performed by one or more computing devices. The enhanced image data and the mask may be processed by a first encoder model of the conditional variational autoencoder to generate an embedding. The true value image data and the mask may be processed by a second encoder model of the conditional variational autoencoder to generate a feature vector. The feature vector may include a distribution value, and the distribution value may be a mean and a standard deviation value. The feature vector may be randomized after generation. The embedding and randomized feature vector may be compiled and processed by a decoder model. The decoder model of the conditional variational autoencoder may generate predicted image data. The predicted image data may include replacement image data for the area indicated by the mask. Unwanted data may be replaced by the replacement image data.

[0112] At 406, the computing system may include evaluating the generated predicted image data relative to the ground truth image data. The evaluation may be performed by one or more computing devices. The evaluation may be based on one or more loss functions. The L1 loss function, the VGG loss function, and the adversarial loss function may be used alone or in combination.

[0113] At 408, the computing system may include modifying one or more parameters. The modification may be performed by one or more computing devices. The modification may be performed in response to the evaluation data. The parameters of the conditional variational autoencoder may be modified.

[0114] Figure 5 Depicted is a flow chart of an example method performed according to an example embodiment of the present disclosure. Figure 5 The steps of method 500 are described in a specific order for the purpose of illustration and discussion, but the method of the present disclosure is not limited to the specific order or arrangement shown. The steps of method 500 can be omitted, rearranged, combined and / or adjusted in various ways without departing from the scope of the present disclosure.

[0115] At 502, the computing system may include inputting the image data and the mask into an encoder. In some embodiments, the encoder may be an encoder for a conditional variational autoencoder. The image data may include unwanted image data, and the mask may provide the location and size of the unwanted image data. The unwanted image data may be a person in the foreground of a landscape photo, a person in the background of a family photo, or another occluding object.

[0116] At 504, the computing system may include receiving an embedding from an encoder. The embedding may include encoded image data. The embedding may be supplemented by a conditional vector. The conditional vector may be a zero vector.

[0117] At 506, the computing system may include inputting the embedding and the conditional vector into a decoder. In some implementations, the decoder may be a decoder for a conditional variational autoencoder.

[0118] At 508, the computing system may include receiving predicted image data from the decoder. The predicted image data may include replacement image data. The replacement image data may replace the unwanted image data. The replacement image data may be the exact size and exact location of the unwanted data indicated by the mask.

[0119] Additional public content

[0120] The techniques discussed herein refer to servers, databases, software applications, and other computer-based systems, as well as the actions taken and information sent to and from these systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functionality between and among components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0121] Although the present invention has been described in detail with reference to various specific example embodiments of the present invention, each example is provided by way of explanation rather than limitation of the present invention. Those skilled in the art, after obtaining an understanding of the foregoing, can easily produce changes, variations and equivalents to these embodiments. Therefore, the present invention does not exclude such modifications, variations and / or additions to the present invention, which will be apparent to those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment can be used together with another embodiment to produce yet further embodiments. Therefore, the present invention is intended to cover these changes, variations and equivalents.

Claims

1. A computer-implemented method for training a conditional variational autoencoder to perform image processing, the method comprising: obtaining, by one or more computing devices, training examples comprising ground-truth image data, augmented image data derived from adding unwanted image data to the ground-truth image data, and a mask indicating one or more locations of the unwanted image data within the augmented image data; Processing, by the one or more computing devices, the augmented image data and the mask using a first encoder model of the conditional variational autoencoder to generate an embedding for the image data; processing, by the one or more computing devices, the real-valued image data and the mask using a second encoder model to generate one or more distribution values; processing, by the one or more computing devices, the embedding and the one or more distribution values ​​using a decoder model of the conditional variational autoencoder to generate predicted image data, the predicted image data comprising replacement image data at the one or more locations indicated by the mask, wherein the replacement image data replaces the undesired image data; evaluating, by the one or more computing devices, one or more loss functions based on a comparison of the predicted image data with the true-valued image data; and One or more parameter values ​​of the conditional variational autoencoder are modified, by the one or more computing devices, based at least in part on the one or more loss functions.

2. The computer-implemented method of claim 1 , wherein: The ground truth image data includes a two-dimensional photograph.

3. The computer-implemented method of claim 1 , wherein: The true-valued image data includes a true-valued three-dimensional point cloud, the augmented image data includes one or more unwanted points added to the true-valued three-dimensional point cloud, and the mask identifies the one or more unwanted points.

4. The computer-implemented method of claim 1 , wherein: Evaluating the one or more loss functions includes evaluating an L1 loss between the predicted image data and the true-valued image data.

5. The computer-implemented method of claim 1 , wherein: Evaluating the one or more loss functions includes evaluating an adversarial loss generated based on a discriminator output, the discriminator output generated by a discriminator model based on the predicted image data and the true-valued image data.

6. The computer-implemented method of claim 5, wherein: The discriminator model includes: a first texture-level network that processes portions of the real-valued image data at the one or more locations identified by the mask to generate a first texture discriminator output; a second texture-level network that processes portions of the predicted image data at the one or more locations identified by the mask to generate a second texture discriminator output; and a semantic-level network comprising a shared network that processes the ground-truth image data from which the unwanted data has been removed to generate a semantic discriminator output; The semantic level network generates the discriminator output based on the first texture discriminator output, the second texture discriminator output and the semantic discriminator output.

7. The computer-implemented method of claim 1 , wherein: The distribution value includes a mean value and a standard deviation value.

8. The computer-implemented method of claim 7, wherein: The distribution value is penalized by the KL divergence loss function.

9. The computer-implemented method of claim 1 , further comprising: The distribution value is multiplied by a random value to generate a modified distribution value, wherein a discriminator model processes the modified distribution value to generate the predicted image data.

10. The computer-implemented method of claim 1 , wherein: Obtaining the enhanced image data includes: Identify collections of unwanted data; determining locations where the unwanted data occludes the true-value image data; A portion of the true-value image data at the location is replaced with the set of unwanted data.

11. The computer-implemented method of any one of claims 1 to 10, wherein: The real-valued image data depicts a scene; The unwanted image data includes an occluding object; and The replacement image data depicts one or more portions of the scene occluded by the occluding object.

12. The computer-implemented method of claim 11, wherein: The occluding object includes a human being.

13. The computer-implemented method of claim 11 , wherein: The occluding objects include artifacts from a denoising process applied to the image.

14. A computing system comprising: one or more processors; One or more non-transitory computer-readable images that collectively store: Conditional variational autoencoder models for machine learning, including: an encoder, wherein the encoder is configured to encode image data; a decoder, wherein the decoder is configured to decode the encoded image data; and instructions that, when executed by the one or more processors, cause the computing system to perform operations comprising: inputting image data and a mask into the encoder, wherein the image data includes unwanted image data, and wherein the mask indicates a location and size of the unwanted image data; receiving an embedding from the encoder, wherein the embedding comprises the encoded image data; inputting the embedding and the conditional vector into the decoder; and Predicted image data is received as an output of the decoder, wherein the predicted image data is based at least in part on the image data and the conditional vector, and the unwanted image data is replaced with predicted replacement data.

15. The computing system of claim 14, wherein: The conditional vector includes a zero vector that replaces a set of randomized feature vectors used during training of the machine-learned conditional variational autoencoder model.

16. The computing system of claim 14, wherein: The machine-learned conditional variational autoencoder model has been trained based on a loss function that compares true-valued training image data with predicted training image data generated by the machine-learned conditional variational autoencoder model based on augmented training image data, where the augmented training image data is created by inserting unwanted image data into the true-valued training image data.

17. The computing system of claim 14, wherein: The input image data includes a two-dimensional photograph.

18. The computing system of claim 14, wherein: The input image data includes a three-dimensional point cloud.

19. The computing system of any one of claims 14 to 18, wherein: The unwanted image data includes an occluding object; and The predicted replacement image data depicts one or more portions of the scene occluded by the occluding object.

20. The computing system of claim 19, wherein: The occluding object includes a human being.

21. One or more non-transitory computer-readable media collectively storing instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations comprising: obtaining, by one or more computing devices, training examples comprising ground-truth data, augmented data derived from adding unwanted data to the ground-truth data, and a mask indicating one or more locations of the unwanted data within the augmented data; Processing the augmented data and the mask using a first encoder model of a conditional variational autoencoder by the one or more computing devices to generate an embedding for the data; processing, by the one or more computing devices, the real-valued data and the mask using a second encoder model to generate one or more distribution values; processing, by the one or more computing devices, the embedding and the one or more distribution values ​​using a decoder model of the conditional variational autoencoder to generate predicted data, the predicted data comprising replacement data at the one or more locations indicated by the mask, wherein the replacement data replaces the unwanted data; evaluating, by the one or more computing devices, one or more loss functions based on a comparison of the predicted data with the true value data; and One or more parameter values ​​of the conditional variational autoencoder are modified, by the one or more computing devices, based at least in part on the one or more loss functions.

22. The one or more non-transitory computer-readable media of claim 21, wherein: The real-value data includes real-value audio waveform data, the enhanced data includes enhanced audio waveform data, the replacement data includes replacement audio waveform data, and The unwanted data includes unwanted audio waveform data.

Images