Image segmentation upscaling

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for upscaling segmentation masks for image processing. One of the methods includes generating, using a trained machine learning model, a semantic mask of a first image, wherein the semantic mask includes an image classification and a confidence value for pixels of the first image across N image classes; generating, using the image classification and confidence values for the pixels of the first image, a semantic mask subset that identifies a subset of classes for each of the pixels of the first image; generating an upscaled version of the semantic mask subset; generating a second semantic mask subset based on the upscaled version of the semantic mask subset; and processing an upscaled version of the first image using the second semantic mask subset to obtain a processed output image.

Description

[0001] IMAGE SEGMENTATION UPSCALING

[0002] BACKGROUND

[0003] Image adjustments can be used to improve the quality of images taken, e.g., by a camera of a smartphone or other mobile device. Raw data from image sensors of such cameras can appear unnatural or have other artifacts (e.g., blooming, chromatic aberration, aliasing, moire, noise). Adjusting images to, e.g., remove or reduce the presence of such artifacts in the captured image, can include classifying portions of an image using a segmentation mask and then adjusting the portions based on the classification, e.g., adjusting values corresponding to pixels or groups of pixels of the captured image.

[0004] SUMMARY

[0005] In general, one aspect of the subject matter described in this specification can be embodied in methods that include the actions of generating, using a trained machine learning model, a semantic mask of a first image, wherein the semantic mask includes an image classification and a confidence value for pixels of the first image across N image classes; generating, using the image classification and confidence values for the pixels of the first image, a semantic mask subset that identifies a subset of classes for each of the pixels of the first image; generating an upscaled version of the semantic mask subset; generating a second semantic mask subset based on the upscaled version of the semantic mask subset; and processing an upscaled version of the first image using the second semantic mask subset to obtain a processed output image.

[0006] Other implementations of this aspect include corresponding computer systems, apparatus, computer program products, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods. A system of one or more computers can be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination of them installed on the system that in operation causes or cause the system to perform the actions. One or more computer programs can be configured to perform particular operations or actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions.

[0007] The foregoing and other implementations can each optionally include one or more of the following features, alone or in combination. Feature 1 : Generating the semantic mask subset comprises: determining, for one or more pixels in the first image, a classification for the pixel with a highest confidence value across the N image classes; and generating a value in the semantic mask subset for the one or more pixels in the first image, wherein each value corresponds to a pixel and represents the determined classification for that pixel. Feature 2: Generating the semantic mask subset comprises: adjusting, based on one or more classifications for pixels that neighbor a first pixel in the semantic mask of the first image, a classification of the first pixel. Feature 3: The actions include generating, using a raw image, the first image by downscaling the raw image. Feature 4: Generating the second semantic mask subset comprises: adjusting, based on one or more classifications for pixels that neighbor a first pixel in the upscaled version of the semantic mask subset, a classification of the first pixel. Feature 5: Generating the second semantic mask subset comprises: determining, for a first pixel and one or more pixels neighboring the first pixel in an upscaled version of the first image, (i) a degree of spatial similarity between the first pixel and the one or more pixels neighboring the first pixel and (ii) a degree of color similarity between the first pixel and the one or more pixels neighboring the first pixel; and adjusting, using the degree of spatial similarity between the first pixel and the one or more pixels neighboring the first pixel and (ii) the degree of color similarity between the first pixel and the one or more pixels neighboring the first pixel, a value of the upscaled version of the semantic mask subset. Feature 6: Adjusting the value of the upscaled version of the semantic mask subset comprises: changing a classification value that represents a semantic classification for a pixel of the upscaled version of the first image. Feature 7: Generating the upscaled version of the semantic mask subset comprises: sampling the semantic mask subset bilinearly; and combining the sampled elements of the semantic mask subset. Feature 8: The actions include training the machine learning model, wherein the training comprises: providing, to the machine learning model, training data that includes images of a first size; comparing output of the machine learning model to ground truth data representing known locations of objects or regions within images of the training data; and updating one or more values of the machine learning model based on the comparison. Feature 9: The first size is less than one million pixels.

[0008] This specification uses the term “configured to” in connection with systems, apparatus, and computer program components. That a system of one or more computers is configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform those operations or actions. That one or more computer programs is configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform those operations or actions. That special-purpose logic circuitry is configured to perform particular operations or actions means that the circuitry has electronic logic that performs those operations or actions.

[0009] The subject matter described in this specification can be implemented in various implementations and may result in one or more of the following advantages. For example, techniques can include identifying a subset of semantic classifications used for segmentation and upscaling only that subset instead of upscaling each element of a fully generated semantic mask, which may include confidence values, per semantic class, per pixel of an input image. The techniques described can identify a subset of classes and store the one or more subsets of class identifiers for each pixel of an input image. This resulting subset can be upscaled using techniques described in this document — e.g., adjusting pixels- based classification or confidence values of neighboring pixels. The upscaling of a generated subset of the initially generated mask allow s a processor to require less memory for storing masks and less computation cycles for processing elements of the generated subsets. Savings in memory can be a factor of 10 and savings in processing cycles can be a factor of 20, as described in this document (e.g., as described with reference to FIG. 2). The techniques can allow for the use of more efficient machine learning models — e.g., models that are trained using low resolution images and training data can be more efficient than models trained using higher resolution images and training data. These more efficient machine learning models can be paired with the upscaling techniques described in this document to help improve image processing for higher resolution images — e.g., on mobile devices or processor restrictive scenarios.

[0010] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

[0011] BRIEF DESCRIPTION OF THE DRAWINGS

[0012] FIG. 1 shows an example image system for image segmentation upscaling.

[0013] FIG. 2 shows an example segmentation upscaling system.

[0014] FIG. 3 is a flowchart of an example process for image segmentation upscaling.

[0015] FIG. 4 is a diagram illustrating an example of a computing system used for upscaling. Like reference numbers and designations in the various drawings indicate like elements.

[0016] DETAILED DESCRIPTION

[0017] FIG. 1 shows an example image system 100 for image segmentation upscaling. The system 100 includes an imaging engine 102, an image processing engine 106, and a display engine 110. In some implementations, the system 100 includes a mobile device 120, such as a smartphone. For example, the imaging engine 102 can include a camera of a device, such as a mobile device. Operations of the image processing engine 106 can be performed by one or more processors of the mobile device 120 or processors of a server or other processor. The display engine 110 can include a screen of a device, such as the mobile device 120.

[0018] In general, the image system 100 can process an image to adjust, correct, or improve colors within an image — e.g., to make images look more life-like or appealing. In particular, the imaging engine 102 generates a raw image 104. For example, the imaging engine 102 can include at least one or more of: a camera, lens, image sensor, or other elements (e.g., for obtaining and generating a raw digital image). The raw image 104 can include one or more values indicating values of pixels of the raw image. The imaging engine 102 provides the raw image 104 to the image processing engine 106. For example, the image processing engine 106 can perform image processing using one or more segmentation masks, which can include semantic masks, generated using techniques described in this document. A segmentation mask can include indications of regions or objects within a captured image. Indications can include labels for one or more pixels. A semantic mask can indicate regions or objects of a particular set of one or more classes within a given captured image — e.g., which pixels in an image correspond to a human, tree, sky, or a combination of these among other objects or regions. By using initial masks that can be later upscaled, the system 100 can use a memory and processing efficient mask model. A memory and processing efficient mask model can be a model that is trained using downscaled or lower resolution images (e.g., 500 pixels by 500 pixels, where lower resolution can refer to images that are lower in resolution than an image that is captured by a device or an image that is used for display). By using downscaled or lower resolution images, a model can be trained with less memory' and less processing time, where memory can be reduced by storing lower resolution images compared to higher resolution images as training data and processing can be reduced by processing fewer pixels corresponding to fewer bytes of training data because the training data is of a lower resolution — e.g., less than one million pixels per image.

[0019] Techniques described in this application achieve processing requirements and time improvements while also allowing processing of higher resolution images. Because processing a high-resolution image (e.g., 1080 pixels by 1080 pixels) by directly using a lower resolution segmentation mask can decrease the quality of the processed output — e.g., by introducing noise along boundaries of segmentation — the techniques described include processes for upscaling a generated lower resolution segmentation mask. The upscaled segmentation mask can then be used to process higher resolution images — e.g., high- definition images captured in digital photography, such as smartphones.

[0020] The segments of the segmentation mask can correspond to identified semantic classes. The generated upscaled mask can then be used to process a corresponding higher resolution image. The techniques improve upon other techniques of upscaling by reducing the memory and processing requirements for one or more processors configured to perform the upscaling actions — e.g., by generating a subset mask to decrease the number of class segmentation masks to be upscaled. Through such memory and processing reductions, an elapsed time of processing can also be reduced. The techniques described can generate upscaled masks with smooth boundaries, instead of jagged boundaries which can typically result from using upscaling approaches, such as bilinear upsampling.

[0021] The image processing engine 106 can include processes of an image signal processor configured to process data from the imaging engine 102. The image processing engine 106 can include processes of a display processor — e.g., configured to obtain image data and adjust the data before displaying the data using one or more displays. The image data can be obtained from the imaging engine 102. The image data can be obtained from an image signal processor, application, decoder, among others. The image processing engine 106 generates a processed image 108. For example, the processed image 108 can be a processed version of the raw image 104 where processes can include adj ustments based on a segmentation mask that has been generated and upscaled using techniques described in this document — e.g., adjusting a first portion, classified as part of a segment corresponding to a '‘sky” semantic class, of the raw image 104 using adjustments specified for the “sky” class. Any suitable number or types of classes can be used, where classes can generally include different types of visual elements, such as faces, landscapes, people, animals, food, sky. or the like. The system 100 can include a trained segmentation mask model that generates a segmentation mask using the raw image 104 as input. The segmentation mask model can be apart of the image processing engine 106. In some cases, the segmentation mask model performs semantic mask generation. The segmentation mask model can be trained using one or more images as training data and specified segments within the images as ground truth data. The memory and processing requirements of the segmentation mask model can be reduced by using low er resolution images as described in this document.

[0022] The image processing engine 106 provides the processed image 108 to the display engine 1 10. For example, the image processing engine 106 can include one or more processors that can provide pixel values for display using the display engine 110. The display engine 110 can provide data to a screen for viewing the processed image 108. In some cases, the image processing engine 106 can provide the processed image 108 for one or more other actions — e.g., encoding or compression for streaming or transporting data of the processed image 108 to another device or to a memory storage.

[0023] FIG. 2 shows an example segmentation upscaling system 200. The system 200 can generate an upscaled semantic mask 234 using an iterative method of joint voting. The upscaled semantic mask 234 can then be used to generate a processed output image 240. The processed output image 240 can be an example of the processed image 108 shown and described in reference to FIG. 1. In some cases, actions described in reference to FIG. 2 can be performed by the system 100, e g., the imaging processing engine 106. In some cases, the system 200 includes hardware or software configured to perform the described actions. For example, display hardware can include one or more circuit elements configured to perform operations described in this document. The display hardware can be used to process or enhance images that are then show n on a display of the display hardware or are sent for storage or to another device.

[0024] A semantic mask engine 206 of the system 200 obtains an input image 202. For example, the semantic mask engine 206 can obtain an input image 202 that is of lower resolution, such as 500 pixels by 500 pixels. The input image 202 can be actively downscaled by the system 200, e.g., after being captured by a camera of a device, or be captured natively by a camera of a device as a lower resolution image. In some cases, the system 200 can actively downscale an image by performing one or more downscaling processes, such as bicubic or bilinear downsampling. The input image 202 used by the semantic mask engine 206 to generate the semantic mask 208 can be of lower resolution compared to a high-resolution image 238 used to generate the processed output image 240. The semantic mask engine 206 generates a semantic mask 208. For example, the semantic mask engine 206 can identify each pixel of the input image 202 as either belonging, or not belonging to, one or more semantic classes. The identification can include confidence values for each of the one or more semantic classes, e.g., N semantic classes where N represents any number and where each confidence value for a pixel indicates a confidence or probability that the pixel belongs to a particular semantic class. In some cases, the confidence values for each semantic class, corresponding to each pixel of the input image 202, can be stored as separate sets of confidence values. For example, element 210 of FIG. 2 is a visual representation of sets of confidence values including a set 212. Each set 212 corresponds to a particular semantic class, such that, if there are N semantic classes, there would be N sets 212. Moreover, each set 212 can include a confidence value, for each pixel of the input image 202, indicating whether the given pixel is in a particular class corresponding to the given set 212, e.g., "sky", “person”, or other suitable class. Confidence values can range from a first value to second value, where a first value indicates a pixel likely represents a particular semantic class element and a second value indicates a pixel likely does not represent the particular semantic class element. Values ranges can be, e g., 0-1, 0-100, or the like. In some cases, higher values indicate a greater likelihood than smaller values.

[0025] In some implementations, the semantic mask engine 206 includes one or more trained machine learning models. For example, the semantic mask engine 206 can include a model trained to generate semantic masks using training data representing various elements and ground truth data representing labels of the various elements with locations of where the elements are located in the training data images. The system 200 can train one or more models using training data and ground truth data to minimize an error term or by using one or more machine learning training techniques. By using lower resolution images for training data, the one or more models can be trained more quickly as the lower resolution images include fewer data bytes to be processed by the model. Furthermore, data storage required for the training data can be reduced using the reduced resolution images. In addition, during runtime, the models trained using the lower resolution images can generate masks for similar images more rapidly (and by consuming fewer computer processing resources) compared to models trained using higher resolution images which require higher resolution input during runtime (and by extension, relatively more computer processing resources). The semantic mask engine 206 can provide the semantic mask 208 to a class selection engine 214. The class selection engine 214 can generate a semantic mask subset 216. The semantic mask subset 216 can include at least one data structure representing semantic classifications for pixels of the input image 202.

[0026] In some implementations, the semantic mask subset 216 includes a most likely classification for each pixel of the input image 202. For example, the class selection engine 214 can compare confidence values across N semantic classes for each pixel classified in the semantic mask. Based on the comparison, the class selection engine 214 can identify a most likely classification — e.g., which class corresponds to a highest confidence value or, more generally, is closest to a first value in a range of confidence values that is associated with a greatest degree of confidence in the classification, such as 1 on a scale of 0-1. The class selection engine 214 can store the most likely classification for each pixel in a data structure, e.g., shown visually in item 218 as semantic mask subset element 220.

[0027] As indicated in item 218, in some cases, the class selection engine 214 can generate a semantic mask subset 216 that includes multiple classifications per pixel. For example, instead of identifying the most likely classification from the N classes represented in the semantic mask 208, the class selection engine 214 can identify a top n classes, where n represents any number less than or equal to N. With n equal to 2, the class selection engine 214 can identify the tw o most likely classifications for each pixel of the input image 202. For example, the class selection engine 214 can perform sorting or use functions such as argmax or the like to determine confidence values that indicate higher confidence values for a given confidence value range. The class selection engine 214 can sort all confidence values for each pixel using data of the semantic mask 208, and then select the top n for each pixel for inclusion in the semantic mask subset 216. The top n can include the highest n values, lowest n values, or other values depending on the range chosen for the confidence values of the semantic mask 208.

[0028] In some implementations, the class selection engine 214 uses a joint voting filter engine 226 to generate the semantic mask subset 216. For example, the joint voting filter engine 226 can be used to smooth the boundaries of the semantic mask subset 216 before being processed by an iterative upscaling engine 222. The processes of the joint voting filter engine 226 are described in reference to the iterative upscaling engine 222.

[0029] The class selection engine 214 can provide the semantic mask subset 216 to the iterative upscaling engine 222. The iterative upscaling engine 222 includes an upscaling engine 224 and the joint voting filter engine 226. The upscaling engine 224 can perform one or more upscaling operations — e.g., to increase the size of at least a portion of the semantic mask subset 216. In some cases, the upscaling engine 224 can perform bilinear upsampling or other suitable upscaling approaches. The upscaling engine 224 can provide an upscaled version of the semantic mask subset 216 to the joint voting filter engine 226.

[0030] The joint voting filter engine 226 can adjust one or more elements of the semantic mask subset 216. For example, the joint voting filter engine 226 can adjust a pixel 230 of a portion 228 of the semantic mask subset 216. The portion 228 can be from an original version of the semantic mask subset 216 generated by the class selection engine 214 or from an upscaled version generated by the upscaling engine 224. The pixel 230 can represent an element of the semantic mask subset 216 that represents a classification of a pixel in the input image 202 or a classification for a portion of a pixel of the input image 202 in a case of an upscaled version — e.g., where each upscaled pixel represents a subset of an original, non-upscaled, pixel.

[0031] The joint voting filter engine 226 can adjust the pixel 230 using neighboring pixels as input. For example, the joint voting filter engine 226 can obtain classification categories for pixels neighboring the pixel 230. Based on the categories, e.g., semantic class classifications, the joint voting filter engine 226 can generate one or more scores. In some cases, the joint voting filter engine 226 generates a score for each class that is included in the pixels that neighbor the pixel 230. The score can be a summation, such as a weighted summation, based on values corresponding to neighboring pixels of the pixel 230. An example summation function can be: where p represents the pixel 230 and q represents a given neighboring pixel. Gs(p, q) represents a function that generates a result based on spatial distance between p and q, Gr(Ip>Iq) represents a function that generates a result based on similarity of the classes of pixel p and q — e.g., color similarity of the classes , id(qn) represents the semantic class identity' of pixel q. i represents a particular semantic class identity, and S represents a neighboring window of pixels that are used in determining each wt.

[0032] In some cases, the joint voting filter engine 226 determines color similarity, e.g., Gr(JP> Iq using an upscaled or at least partially upscaled version of the input image 202. For example, the upscaling engine 224 can upscale the input image 202 using bilinear sampling or other upscaling techniques. The upscaling engine 224 can upscale one or more semantic masks corresponding to the input image 202 such that the upscaled version and the upscaled one or more semantic masks include a same number of pixels — e.g., where each pixel of each of the upscaled version of the input image 202 and the upscaled one or more semantic masks can be referenced as pixel p or q. For example, in determining Gr(Ipilq), the joint voting filter engine 226 can determine color similarity by referencing the pixels in an upscaled version of the input image 202 that correspond to the pixel location of p and q. The joint voting filter engine 226 can determine color similarity by comparing one or more values representing the pixels of the upscaled version of the input image 202 — e.g., comparing red, green, and blue pixel values for each pixel or lightness / value, hue, and saturation values for each pixel or other values used to represent pixels. In some cases, the upscaling engine 224 upscales both the input image 202 and the one or more semantic masks, e.g., of the semantic mask subset 216. In some cases, the upscaling engine 224 performs iterations of upscaling before or after iterations of processes performed by the joint voting filter engine 226 until one or more thresholds or criteria are satisfied. In response to one or more thresholds or criteria are satisfied, the iterative upscaling engine 222 can output the upscaled semantic mask 234.

[0033] In an example case where a neighboring window of pixels around the pixel 230 include three different semantic classes, the joint voting filter engine 226 can generate three scores for the pixel 230 corresponding to the three different semantic classes — e.g., 1.05 for the first class. 0.45 for the second class, and 0.03 for the third class, where value ranges can be adjusted to suit a given implementation. In some cases, the neighboring window is a 5 by 5 grid of pixels with the pixel 230 in the center. In some case, other windows can be used — e g., circular shaped windows, different sized windows, or the like.

[0034] In some implementations, the joint voting filter engine 226 determines an adjustment to the pixel 230 using one or more generated scores. For example, the joint voting filter engine 226 can use the three scores generated in the previous example to determine an adjustment to the pixel 230. The adjustment can include a null adjustment — e.g., making no changes. An adjustment can include changing a classification for the pixel 230. In some cases, the joint voting filter engine 226 can change a classification of the pixel 230 to match the classification associated with the highest generated score. For example, in a case where the joint voting filter engine 226 generates a score of 1.05 for the first class, 0.45 for the second class, and 0.03 for the third class, the joint voting filter engine 226 can identify the highest score, 1.05, and adjust the pixel 230 to match the associated classification, the first class. The first class can include any suitable semantic class, e.g., “sky’', “person", “trees”, or the like. In some cases, an argmax function, or the like, can be used where the input to the function includes the generated scores for each classification according to the identified neighboring window. In some cases, the joint voting filter engine 226 uses an argmax process that obtains one or more scores and generates the maximum of the obtained scores. In some cases, the joint voting filter engine 226 generates one or more confidence values for use in the upscaling system 200. Fe, the joint voting filter engine 226 can generate one or more confidence values corresponding to scores of each of one or more semantic classes. These confidence values can be used to determine, e.g., adjustments to the pixel 230.

[0035] In some implementations, the iterative upscaling engine 222 performs iterations of upscaling and joint voting. For example, the iterative upscaling engine 222 can perform joint voting adjustments by the joint voting filter engine 226 following upscaling by the upscaling engine 224. The iterative upscaling engine 222 can perform upscaling by the upscaling engine 224 after joint voting adjustments by the joint voting filter engine 226. The iterations can stop after a predetennined number of iterations or after a size of an upscaled version satisfies a size threshold. For example, the system 200 can detect when a threshold number of iterations have been performed and, in response, prevent any further iterations and provide the upscaled semantic mask 234 as output. The system 200 can determine a resolution of an upscaled version satisfies a threshold resolution and. in response, similarly prevent further iterations and provide the upscaled semantic mask 234 as output.

[0036] The iterative upscaling engine 222 can provide the upscaled semantic mask 234 to the mask processing engine 236. The mask processing engine 236 can obtain the upscaled semantic mask 234 and a high-resolution image 238. The high-resolution image 238 can be a version of the input image 202 with more defined pixels per unit of area in a captured view. In some cases, the high-resolution image 238 is used by a downscaling process performed by the system 200 to generate the input image 202. In some cases, the high- resolution image 238 is generated using an image upscaling process using the input image 202.

[0037] The mask processing engine 236 can perform one or more adjustments on the high- resolution image 238 to generate the processed output image 240. For example, the adjustments can be based on semantic classes of portions of the high-resolution image 238 identified in the upscaled semantic mask 234. For example, the upscaled semantic mask 234 can identify a class for each pixel of the high-resolution image 238. Using the identified classes of pixels, the mask processing engine 236 can make adjustments to one or more pixels of the high-resolution image 238. The adjustments can be class based — e.g., brightening a face of a person more than brightening a background with a semantic class of sky. The mask processing engine 236 can generate the processed output image 240 which can then be displayed or sent for storage or further processing. In some cases, the processed output image 240 can be an example of the processed image 108 and be processed for display using the display engine 110.

[0038] The techniques described in reference to FIG. 2 can help reduce processing and memory requirements for generating an upscaled version of a segmentation mask. For example, in a case where an input image is 500 pixels by 500 pixels and there are 20 different semantic classes, processes that upscale each mask individually require 500x500x20=5 megabytes of memory where each pixel is represented by 20 sets of 500 by 500 pixels and where each element in the set can represent a confidence value corresponding to the class where N=20. In contrast, the proposed techniques can require only 500x500x(n+n) bytes where n represents the number of subset classes to identify as described in this document and two sets can be used with one representing the confidence values and one representing the corresponding semantic classification per pixel. For the case where n=l, the number of bytes equal 500x500x2=0.5 megabytes representing a compression factor of 10 compared to processes that upscale each mask individually. Similarly, processing can be reduced by the proposed techniques. Processing can include using a filter of k by k (such as a filter of neighboring filters used in the joint voting filter engine 226 of FIG. 2). Thus, processing energy7for one or more upscaling processors can be proportional to, using values from the previous example for an input image and class size, 500x500x20xkxk for processes upscaling with each separate mask and 500x500xlxkxk for processes upscaling with the proposed techniques which can reduce processing time or energy of one or more processors by a factor of 20.

[0039] FIG. 3 is a flowchart of an example process 300 for image segmentation upscaling. For convenience, the process 300 will be described as being performed by a system of one or more computers, located in one or more locations, and programmed appropriately in accordance with this specification. For example, the system 200 of FIG. 2 or the system 100 of FIG. 1, appropriately programmed, can perform the process 300.

[0040] The process 300 includes generating, using a trained machine learning model, a semantic mask of a first image (302). In some cases, the semantic mask includes an image classification and a confidence value for pixels of the first image across N image semantic classes (as described with reference to FIGS. 1-2). For example, the semantic mask engine 206 can generate the semantic mask 208 using the input image 202 as input. The first image can be obtained as a low-resolution image or be generated from an initially captured image using one or more downscaling operations. In some cases, a device captures a low resolution and high-resolution image when capturing an image. The lower resolution can be used with a trained machine learning model for generate a semantic mask. By using a lower resolution image, processing requirements and time for training a model can be reduced — e.g., training data for the model can be stored in fewer bytes of memory thus requiring less physical storage mediums and less processing to obtain and process the training data in one or more training operations. The runtime operation of a machine learning model trained using lower resolution images can also be reduced compared to models operating on higher resolution images.

[0041] The process 300 includes generating a semantic mask subset that identifies a subset of the N image classes, for each of the pixels of the first image, based on the image classification and confidence values for the pixels across the N image classes (304), as described with reference to FIGS. 1-2. For example, the class selection engine 214 can generate the semantic mask subset 216 using data from the semantic mask 208.

[0042] The process 300 includes generating an upscaled version of the semantic mask subset (306). In some implementations, the upscaling engine 224 of the iterative upscaling engine 222 can upscale at least a portion of the semantic mask subset 216, as described with reference to FIGS. 1-2. For example, the upscaling engine 224 can use bilinear sampling or other upscaling techniques to increase a number of elements represent each pixel of the input image 202.

[0043] The process 300 includes generating a second semantic mask subset based on the upscaled version of the semantic mask subset (308). For example, the joint voting filter engine 226 of the iterative upscaling engine 222 can generate a version of the semantic mask subset 216 where values of elements are adjusted — e.g., based on values of neighboring elements. After one or more iterations, the joint voting filter engine 226 can generate the upscaled semantic mask 234 which can be an upscaled version of the semantic mask subset 216.

[0044] The process 300 includes processing a version of the first image using the second semantic mask subset to obtain a processed output image (310). For example, the mask processing engine 236 can process the high-resolution image 238 using the upscaled semantic mask 234 to generate the processed output image 240. The order of operations in the process 300 described above is illustrative only, and can be performed in different orders in some cases. In some implementations, the process 300 can include additional operations, fewer operations, or some of the operations can be divided into multiple operations. For example, processing a version of the first image can include processing an upscaled version of the first image where the first image is upscaled using one or more processes performed, e.g., in parallel or before one or more operations included in the process 300.

[0045] FIG. 4 is a diagram illustrating an example of a computing system used for upscaling. The computing system includes computing device 400 and a mobile computing device 450 that can be used to implement the techniques described herein. For example, one or more components of the system 100 or 200 could be an example of the computing device 400 or the mobile computing device 450.

[0046] The computing device 400 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The mobile computing device 450 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart-phones, mobile embedded radio systems, radio diagnostic computing devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to be limiting.

[0047] The computing device 400 includes a processor 402, a memory 404, a storage device 406, a high-speed interface 408 connecting to the memory 404 and multiple highspeed expansion ports 410, and a low-speed interface 412 connecting to a low-speed expansion port 414 and the storage device 406. Each of the processor 402. the memory 404, the storage device 406, the high-speed interface 408, the high-speed expansion ports 410, and the low-speed interface 412, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor 402 can process instructions for execution within the computing device 400. including instructions stored in the memory 404 or on the storage device 406 to display graphical information for a GUI on an external input / output device, such as a display 416 coupled to the high-speed interface 408. In other implementations, multiple processors and / or multiple buses may be used, as appropriate, along with multiple memories and types of memory. In addition, multiple computing devices may be connected, with each device providing portions of the operations (e.g., as a server bank, a group of blade servers, or a multi-processor system). In some implementations, the processor 402 is a single threaded processor. In some implementations, the processor 402 is a multi -threaded processor. In some implementations, the processor 402 is a quantum computer.

[0048] The memory 404 stores information within the computing device 400. In some implementations, the memory 404 is a volatile memory unit or units. In some implementations, the memory 404 is a non-volatile memory unit or units. The memory 404 may also be another form of computer-readable medium, such as a magnetic or optical disk.

[0049] The storage device 406 is capable of providing mass storage for the computing device 400. In some implementations, the storage device 406 may be or include a computer- readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configurations. Instructions can be stored in an information carrier. The instructions, when executed by one or more processing devices (for example, processor 402), perform one or more methods, such as those described above. The instructions can also be stored by one or more storage devices such as computer- or machine-readable mediums (for example, the memory 404, the storage device 406, or memory' on the processor 402). The high-speed interface 408 manages bandwidth-intensive operations for the computing device 400, while the low- speed interface 412 manages lower bandwidth-intensive operations. Such allocation of functions is an example only. In some implementations, the high-speed interface 408 is coupled to the memory 404, the display 416 (e.g., through a graphics processor or accelerator), and to the high-speed expansion ports 410, which may accept various expansion cards (not shown). In the implementation, the low-speed interface 412 is coupled to the storage device 406 and the low-speed expansion port 414. The low-speed expansion port 414, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) may' be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

[0050] The computing device 400 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server 420, or multiple times in a group of such servers. In addition, it may be implemented in a personal computer such as a laptop computer 422. It may also be implemented as part of a rack server system 424. Alternatively, components from the computing device 400 may be combined with other components in a mobile device, such as a mobile computing device 450. Each of such devices may include one or more of the computing device 400 and the mobile computing device 450, and an entire system may be made up of multiple computing devices communicating with each other.

[0051] The mobile computing device 450 includes a processor 452, a memory 464, an input / output device such as a display 454, a communication interface 466, and a transceiver 468, among other components. The mobile computing device 450 may also be provided with a storage device, such as a micro-drive or other device, to provide additional storage. Each of the processor 452, the memory 464, the display 454, the communication interface 466, and the transceiver 468, are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other manners as appropriate.

[0052] The processor 452 can execute instructions within the mobile computing device 450, including instructions stored in the memory 464. The processor 452 may be implemented as a chipset of chips that include separate and multiple analog and digital processors. The processor 452 may provide, for example, for coordination of the other components of the mobile computing device 450, such as control of user interfaces, applications run by the mobile computing device 450, and wireless communication by the mobile computing device 450.

[0053] The processor 452 may communicate with a user through a control interface 458 and a display interface 456 coupled to the display 454. The display 454 may be, for example, a TFT (Thin-Film-Transistor Liquid Crystal Display) display or an OLED (Organic Light Emitting Diode) display, or other appropriate display technology. The display interface 456 may include appropriate circuitry for driving the display 454 to present graphical and other information to a user. The control interface 458 may receive commands from a user and convert them for submission to the processor 452. In addition, an external interface 462 may provide communication with the processor 452, so as to enable near area communication of the mobile computing device 450 with other devices. The external interface 462 may provide, for example, for wired communication in some implementations, or for wireless communication in other implementations, and multiple interfaces may also be used.

[0054] The memory 464 stores information within the mobile computing device 450. The memory' 464 can be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. An expansion memory 474 may also be provided and connected to the mobile computing device 450 through an expansion interface 472, which may include, for example, a SIMM (Single In Line Memory Module) card interface. The expansion memory 474 may provide extra storage space for the mobile computing device 450, or may also store applications or other information for the mobile computing device 450. Specifically, the expansion memory 474 may include instructions to cany7out or supplement the processes described above, and may include secure information also. Thus, for example, the expansion memory 474 may be provide as a security module for the mobile computing device 450, and may be programmed with instructions that permit secure use of the mobile computing device 450. In addition, secure applications may be provided via the SIMM cards, along with additional infonnation, such as placing identifying infonnation on the SIMM card in a non-hackable manner.

[0055] The memory may include, for example, flash memory and / or NVRAM memory (nonvolatile random access memory), as discussed below. In some implementations, instructions are stored in an information carrier such that the instructions, when executed by one or more processing devices (for example, processor 452). perform one or more methods, such as those described above. The instructions can also be stored by one or more storage devices, such as one or more computer- or machine-readable mediums (for example, the memory7464, the expansion memory' 474, or memory' on the processor 452). In some implementations, the instructions can be received in a propagated signal, for example, over the transceiver 468 or the external interface 462.

[0056] The mobile computing device 450 may communicate wirelessly through the communication interface 466, which may include digital signal processing circuitry' in some cases. The communication interface 466 may provide for communications under various modes or protocols, such as GSM voice calls (Global System for Mobile communications), SMS (Short Message Service), EMS (Enhanced Messaging Service), or MMS messaging (Multimedia Messaging Service). CDMA (code division multiple access), TDMA (time division multiple access), PDC (Personal Digital Cellular), WCDMA (Wideband Code Division Multiple Access), CDMA2000, or GPRS (General Packet Radio Service), LTE, 4G / 6G cellular, among others. Such communication may occur, for example, through the transceiver 468 using a radio frequency. In addition, short-range communication may occur, such as using a Bluetooth, Wi-Fi, or other such transceiver (not shown). In addition, a GPS (Global Positioning System) receiver module 470 may provide additional navigation- and location-related wireless data to the mobile computing device 450, which may be used as appropriate by applications running on the mobile computing device 450.

[0057] The mobile computing device 450 may also communicate audibly using an audio codec 460, which may receive spoken information from a user and convert it to usable digital information. The audio codec 460 may likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of the mobile computing device 450. Such sound may include sound from voice telephone calls, may include recorded sound (e.g., voice messages, music files, among others) and may also include sound generated by applications operating on the mobile computing device 450.

[0058] The mobile computing device 450 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a cellular telephone 480. It may also be implemented as part of a smart-phone 482, personal digital assistant, or other similar mobile device.

[0059] In general, use of “or” can refer to “and / or.” When providing a list of two or more items, the conjunction “or” can indicate any one of the items, any combination of a subset of the items, or all items in combination.

[0060] In this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0061] The subject matter and the actions and operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. The subject matter and the actions and operations described in this specification can be implemented as or in one or more computer programs, e.g., one or more modules of computer program instructions, encoded on a computer program carrier, for execution by, or to control the operation of, data processing apparatus. The carrier can be a tangible non-transitory computer storage medium. Alternatively or in addition, the carrier can be an artificially- generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be or be part of a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. A computer storage medium is not a propagated signal.

[0062] The term “data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. Data processing apparatus can include special-purpose logic circuitry, e.g.. an FPGA (field programmable gate array), an ASIC (application-specific integrated circuit), or a GPU (graphics processing unit). The apparatus can also include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0063] A computer program can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program, e.g., as an app, or as a module, component, engine, subroutine, or other unit suitable for executing in a computing environment, which environment may include one or more computers interconnected by a data communication network in one or more locations.

[0064] A computer program may, but need not, correspond to a file in a file system. A computer program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code.

[0065] The processes and logic flows described in this specification can be performed by one or more computers executing one or more computer programs to perform operations by operating on input data and generating output. The processes and logic flows can also be performed by special-purpose logic circuitry, e.g., an FPGA, an ASIC, or a GPU, or by a combination of special-purpose logic circuitry and one or more programmed computers.

[0066] Computers suitable for the execution of a computer program can be based on general or special-purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a readonly memory or a random access memory or both. The essential elements of a computer are a central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special-purpose logic circuitry.

[0067] Generally, a computer will also include, or be operatively coupled to, one or more mass storage devices, and be configured to receive data from or transfer data to the mass storage devices. The mass storage devices can be, for example, magnetic, magneto-optical, or optical disks, or solid state drives. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0068] To provide for interaction with a user, the subject matter described in this specification can be implemented on one or more computers having, or configured to communicate with, a display device, e.g., a LCD (liquid crystal display) monitor, or a virtual-reality (VR) or augmented-reality (AR) display, for displaying information to the user, and an input device by which the user can provide input to the computer, e.g., a keyboard and a pointing device, e.g., a mouse, a trackball or touchpad. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback and responses provided to the user can be any form of sensory feedback, e.g., visual, auditory', speech, or tactile feedback or responses; and input from the user can be received in any form, including acoustic, speech, tactile, or eye tracking input, including touch motion or gestures, or kinetic motion or gestures or orientation motion or gestures. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser, or by interacting with an app running on a user device, e.g., a smartphone or electronic tablet. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0069] This specification uses the term “configured to” in connection with systems, apparatus, and computer program components. That a system of one or more computers is configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. That one or more computer programs is configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions. That special-purpose logic circuitry is configured to perform particular operations or actions means that the circuitry has electronic logic that performs the operations or actions.

[0070] The subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0071] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some implementations, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0072] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of what is being claimed, which is defined by the claims themselves, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claim may be directed to a subcombination or variation of a subcombination. Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this by itself should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

CLAIMS1. A method comprising: generating, using a trained machine learning model, a semantic mask of a first image, wherein the semantic mask includes an image classification and a confidence value for pixels of the first image across N image classes; generating, using the image classification and confidence values for the pixels of the first image, a semantic mask subset that identifies a subset of classes for each of the pixels of the first image; generating an upscaled version of the semantic mask subset; generating a second semantic mask subset based on the upscaled version of the semantic mask subset; and processing an upscaled version of the first image using the second semantic mask subset to obtain a processed output image.

2. The method of claim 1, wherein generating the semantic mask subset comprises: determining, for one or more pixels in the first image, a classification for the pixel with a highest confidence value across the N image classes; and generating a value in the semantic mask subset for the one or more pixels in the first image, wherein each value corresponds to a pixel and represents the determined classification for that pixel.

3. The method of claim 1. wherein generating the semantic mask subset comprises: adjusting, based on one or more classifications for pixels that neighbor a first pixel in the semantic mask of the first image, a classification of the first pixel.

4. The method of claim 1, comprising: generating, using a raw image, the first image by downscaling the raw image.

5. The method of claim 1, wherein generating the second semantic mask subset comprises:adjusting, based on one or more classifications for pixels that neighbor a first pixel in the upscaled version of the semantic mask subset, a classification of the first pixel.

6. The method of claim 1, wherein generating the second semantic mask subset comprises: determining, for a first pixel and one or more pixels neighboring the first pixel in an upscaled version of the first image, (i) a degree of spatial similarity between the first pixel and the one or more pixels neighboring the first pixel and (ii) a degree of color similarity between the first pixel and the one or more pixels neighboring the first pixel; and adjusting, using the degree of spatial similarity between the first pixel and the one or more pixels neighboring the first pixel and (ii) the degree of color similarity between the first pixel and the one or more pixels neighboring the first pixel, a value of the upscaled version of the semantic mask subset.

7. The method of claim 6, wherein adjusting the value of the upscaled version of the semantic mask subset comprises: changing a classification value that represents a semantic classification for a pixel of the upscaled version of the first image.

8. The method of claim 1, wherein generating the upscaled version of the semantic mask subset comprises: sampling the semantic mask subset bilinearly; and combining the sampled elements of the semantic mask subset.

9. The method of claim 1, comprising training the machine learning model, wherein the training comprises: providing, to the machine learning model, training data that includes images of a first size; comparing output of the machine learning model to ground truth data representing known locations of objects or regions within images of the training data; and updating one or more values of the machine learning model based on the comparison.

10. The method of claim 9, wherein the first size is less than one million pixels.

11. One or more computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising: generating, using a trained machine learning model, a semantic mask of a first image, wherein the semantic mask includes an image classification and a confidence value for pixels of the first image across N image classes; generating, using the image classification and confidence values for the pixels of the first image, a semantic mask subset that identifies a subset of classes for each of the pixels of the first image; generating an upscaled version of the semantic mask subset; generating a second semantic mask subset based on the upscaled version of the semantic mask subset; and processing an upscaled version of the first image using the second semantic mask subset to obtain a processed output image.

12. The media of claim 11 , wherein generating the semantic mask subset comprises: determining, for one or more pixels in the first image, a classification for the pixel with a highest confidence value across the N image classes; and generating a value in the semantic mask subset for the one or more pixels in the first image, wherein each value corresponds to a pixel and represents the determined classification for that pixel.

13. The media of claim 11 , wherein generating the semantic mask subset comprises: adjusting, based on one or more classifications for pixels that neighbor a first pixel in the semantic mask of the first image, a classification of the first pixel.

14. The media of claim 11, wherein the operations comprise: generating, using a raw image, the first image by downscaling the raw image.

15. The media of claim 11, wherein generating the second semantic mask subset comprises: adjusting, based on one or more classifications for pixels that neighbor a first pixel in the upscaled version of the semantic mask subset, a classification of the first pixel.

16. The media of claim 11, wherein generating the second semantic mask subset comprises: determining, for a first pixel and one or more pixels neighboring the first pixel in an upscaled version of the first image, (i) a degree of spatial similarity' between the first pixel and the one or more pixels neighboring the first pixel and (ii) a degree of color similarity between the first pixel and the one or more pixels neighboring the first pixel; and adjusting, using the degree of spatial similarity between the first pixel and the one or more pixels neighboring the first pixel and (ii) the degree of color similarity between the first pixel and the one or more pixels neighboring the first pixel, a value of the upscaled version of the semantic mask subset.

17. The media of claim 16, wherein adjusting the value of the upscaled version of the semantic mask subset comprises: changing a classification value that represents a semantic classification for a pixel of the upscaled version of the first image.

18. The media of claim 11 , wherein generating the upscaled version of the semantic mask subset comprises: sampling the semantic mask subset bilinearly; and combining the sampled elements of the semantic mask subset.

19. The media of claim 11 , wherein the operations comprise training the machine learning model, wherein the training comprises: providing, to the machine learning model, training data that includes images of a first size; comparing output of the machine learning model to ground truth data representing known locations of objects or regions within images of the training data; andupdating one or more values of the machine learning model based on the comparison.

20. A system comprising: one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perfomr operations comprising: generating, using a trained machine learning model, a semantic mask of a first image, wherein the semantic mask includes an image classification and a confidence value for pixels of the first image across N image classes; generating, using the image classification and confidence values for the pixels of the first image, a semantic mask subset that identifies a subset of classes for each of the pixels of the first image; generating an upscaled version of the semantic mask subset; generating a second semantic mask subset based on the upscaled version of the semantic mask subset; and processing an upscaled version of the first image using the second semantic mask subset to obtain a processed output image.

Images