Efficient image-to-image deep architecture

Abstract

Machine learning models that operate on images can exhibit significant increases in their cost to execute as the size (e.g., resolution, number of pixels) of the images increase. While it is possible to downsample input images and perform some or all of the model image processing in a lower-resolution space, followed by upsampling, the results of such operation have previously been poor. Embodiments are provided that overcome these limitations, resulting in decreased computational cost without decreasing output image quality. These benefits are obtained, in part, by combining pixels of an input image (e.g., by concatenation) into an effectively lower-resolution image space, performing computations thereon, and then de-concatenating or otherwise separating the combined pixels to generate an output image. TTiis can allow individual processor elements of a TPU to efficiently implement a machine learning model, thereby improving output image quality while limiting computational costs to those available even on resource-constrained platforms.

Description

EFFICIENT IMAGE-TO-IMAGE DEEP ARCHITECTURE CROSS-REFERENCE TO RELATED APPLICATION

[0001] The present application is a non-provisional patent application claiming priority to U.S. Provisional Patent Application No. 63 / 669,343, filed July 10, 2024, the contents of which are hereby incorporated by reference. BACKGROUND

[0002] Convolutional neural networks or other trained machine learning models commonly operate by applying one or more inputs to an input layer, whose output(s) are applied to subsequent layers, with processing proceeding from layer to layer until a final output is generated (e.g., an output label or set of class probabilities for a classifier, an output image or other type of output for a generative model). For inputs that are organized in a one, two, or more-dimensional array or other structure (e.g., a one-dimensional array of values representing a sound, a two-dimensional array representing pixels of an image), the model may be organized in a manner that corresponds to the structure of the input. For example, when the input is a two- dimensional array of pixels of an image (e.g., an X-byY-by-3 array of pixels, each pixel represented by three color channels), layers of the model could be organized as two- dimensional arrays of (optionally identical) units in each layer to perform computation on the input (or an intermediate-layer representation of the informational content thereof) in a manner that recapitulates the two-dimensional organization of the input.

[0003] Such model structures can include convolutional neural networks or other structured organizations of parameters and computations. However, as the size of the input (e.g., the resolution of an input image) increases, the number of units to compute in each layer may increase, increasing the computational cost (e.g., processor cycles, memory budget, parameter storage, communications bandwidth), and accordingly power consumption, to execute the model. In some examples, downsampling or other resolution-reduction techniques can be applied to reduce the number of units at one or more layers of the model (e.g., at the innermost layers) in order to reduce the computational cost of model execution; however, such methods may result in reduced accuracy or quality in the outputs. In some examples, one or more final layers of the model may be executed at the full ‘output’ resolution of the model in order to improve the quality of the output, however, this results in increased model execution cost.SUMMARY

[0004] In a first aspect, a method is provided that includes: (i) obtaining a model input that at least partially represents an input image, wherein the model input comprises a two- dimensional array of elements at a first resolution; (ii) applying a first set of one or more layers of a machine learning model to the model input to generate a first intermediate model output, wherein the first intermediate model output comprises a two-dimensional array of elements at the first resolution; (iii) upsampling the first intermediate model output to generate a second intermediate model output that comprises a two-dimensional array of elements at a second resolution, wherein the second resolution is greater than the first resolution; (iv) applying a second set of one or more layers of the machine learning model to the second intermediate model output to generate a third intermediate model output, wherein the third intermediate model output comprises a two-dimensional array of elements at the second resolution; and (v) partitioning the third intermediate model output to generate a terminal model output that comprises a two-dimensional array of elements at a third resolution, wherein the third resolution is greater than the second resolution, and wherein partitioning the third intermediate model output to generate the terminal model output comprises partitioning each element of the third intermediate model output into a respective block of elements of the terminal model output.

[0005] In a second aspect, a method is provided that includes: (i) obtaining a model input that at least partially represents an input image, wherein the model input comprises a two- dimensional array of elements at a first resolution; (ii) applying a first set of one or more layers of a machine learning model to the model input to generate a first intermediate model output, wherein the first intermediate model output comprises a two-dimensional array of elements at the first resolution, and wherein at least two layers of the first set of one or more layers are residual-in-residual dense blocks; and (iii) partitioning the first intermediate model output to generate a terminal model output that comprises a two-dimensional array of elements at a second resolution, wherein the second resolution is greater than the first resolution, and wherein partitioning the first intermediate model output to generate the terminal model output comprises partitioning each element of the first intermediate model output into a respective block of elements of the terminal model output.

[0006] In a third aspect, a method is provided that includes: (i) obtaining a model input from an input image, wherein the input image comprises a plurality of pixels at a first resolutionand the model input comprises a two-dimensional array of elements at a second resolution that is less than the first resolution, wherein obtaining the model input comprises combining sets of pixels of the input image to generate respective elements of the model input such that each element of the model input represents a respective block of pixels of the input image; (ii) applying a first set of one or more layers of a machine learning model to the model input to generate a first intermediate model output, wherein the first intermediate model output comprises a two-dimensional array of elements at the second resolution; and (iii) partitioning the first intermediate model output to generate a terminal model output that comprises a two- dimensional array of elements at a third resolution, wherein the third resolution is greater than the second resolution, and wherein partitioning the first intermediate model output to generate the terminal model output comprises partitioning each element of the first intermediate model output into a respective block of elements of the terminal model output.

[0007] In another aspect, a non-transitory computer readable medium is provided having stored thereon program instructions executable by at least one processor to cause the at least one processor to perform the above methods.

[0008] In another aspect a system is provided that includes: (i) at least one processor; and (ii) a non-transitory computer-readable medium, having stored therein instructions executable by the at least one processor to cause the system to perform the above methods.

[0009] These as well as other aspects, advantages, and alternatives will become apparent to those of ordinary skill in the art by reading the following detailed description with reference where appropriate to the accompanying drawings. Further, it should be understood that the description provided in this summary section and elsewhere in this document is intended to illustrate the claimed subject matter by way of example and not by way of limitation. BRIEF DESCRIPTION OF THE FIGURES

[0010] Figure 1 illustrates aspects of an image processing method, according to an example embodiment.

[0011] Figure 2A illustrates aspects of an image processing method, according to an example embodiment.

[0012] Figure 2B illustrates aspects of an image processing method, according to an example embodiment.

[0013] Figure 3A depicts examples of outputs generated from an image processingmethod as described herein, according to an example embodiment.

[0014] Figure 3b depicts examples of outputs generated from an image processing method as described herein, according to an example embodiment.

[0015] Figure 3C depicts examples of outputs generated from an image processing method as described herein, according to an example embodiment.

[0016] Figure 3D depicts examples of outputs generated from an image processing method as described herein, according to an example embodiment.

[0017] Figure 4A depicts experimental results.

[0018] Figure 4B depicts experimental results.

[0019] Figure 5 is a diagram illustrating training and inference phases of a machine learning model, in accordance with example embodiments.

[0020] Figure 6 is a simplified block diagram showing some of the components of an example computing system.

[0021] Figure 7 is a flowchart of an example method.

[0022] Figure 8 is a flowchart of an example method.

[0023] Figure 9 is a flowchart of an example method. DETAILED DESCRIPTION

[0024] Examples of methods and systems are described herein. It should be understood that the words "exemplary," “example,” and “illustrative,” are used herein to mean "serving as an example, instance, or illustration." Any embodiment or feature described herein as "exemplary," “example,” or “illustrative,” is not necessarily to be construed as preferred or advantageous over other embodiments or features. Further, the exemplary embodiments described herein are not meant to be limiting. It will be readily understood that certain aspects of the disclosed systems and methods can be arranged and combined in a wide variety of different configurations. I. Overview

[0025] For many applications, the input to a machine learning model is organized as a 1-, 2-, 3-, or more-dimensional array of elements. For example, an audio signal or other 1- dimensional input could be organized as a one-dimensional array of inputs (e.g., of scalars for a monaural audio signal, of 2-vectors for a stereo audio signal). In another example, an image or other 2-dimensional input could be organized as a two-dimensional array of pixels of input (e.g., of scalars for a monochromatic image, of 3-vectors for a three-color image, of vectors ofsize greater than three for a hyperspectral image). A machine learning model (e.g., a convolutional neural network) trained to receive such inputs could be configured to have one or more layers composed of a correspondingly-organized array of units. For example, layers of an ML model configured to upsample (e.g., to effect a synthetic zoom function), deblur, sharpen, remove compression or other artifacts from, add details and texture, or otherwise operate on an input image or perform some other image-to-image task could include two- dimensional arrays of (optionally identical) units configured to output a two-dimensional array of intermediate (or terminal) outputs, and the pattern of interconnection between units of different layers could reflect the underlying two-dimensional organization of the input, output, and layers of the model.

[0026] For such models, increasing the resolution (or size) of the input can result in a corresponding increase in the processor cycles, memory, storage space, interconnect bandwidth, or other computational costs to execute the model. In general, it is desirable to decrease such costs. Indeed, some resource-limited systems (e.g., cellphones, tablets, or other embedded systems) may lack sufficient memory or other computational resources to load and execute models having large numbers of units in one or more layers of the model. While it is possible to perform such computations at lower resolution (e.g., by downsampling or otherwise reducing a size of an input image, performing computations at reduced resolution, and then upsampling or otherwise increasing the resolution of an output thereof) or in an otherwise reduced fashion (e.g., converting RGB or other color image data to grayscale), such methods often result in reduced accuracy or quality in the output. Alternatively, the quality of the output can be increased by performing some model computation (e.g., one or more terminal model layers) at full or near-full resolution, however, this results in increased computational cost.

[0027] The embodiments described herein provide improvements with respect to computational cost and output accuracy and quality for various image-to-image machine learning model architectures. These improvements are obtained by combining multiple pixels of an input into ‘combined’ pixels (such an operation may be referred to as a ‘space2depth’ transformation, or tensor shuffling). This can include concatenating the information content of the source pixels together or otherwise combining the pixels (e.g., via concatenation or other lossless processes) before processing them. Then, after the processing, the combined output pixels can be separated (e.g., via an ‘un-shuffling,’ ‘de-concatenation,’ ‘depth2space,’ or other partitioning process that mirrors the original concatenation or other combining process) to render a terminal output of the model at full resolution / size. Such a terminal output can includedirect representations of the pixels of an output image, or may be post-processed (e.g., via truncation, mapping to another color space, color correction, cropping, or some other processing step(s)) to generate an image output. To generate such a terminal output of the model, a machine learning model can process the combined pixels, which are fewer but include more elements individually, using relatively fewer units per layer of the model. This can allow the computational cost (e.g., memory footprint, processor cycles) of executing the model to be reduced or maintained relative to alternative model architectures, or adapted to the specific computational resources of a resource-limited system (e.g., to the available compute units of a tensor processing unit of a cellphone or other mobile system) while increasing output accuracy and / or quality. The concatenation or other ‘shuffling’ processes to combine and separate the pixels of the input / output can be lossless and may exhibit very low computational cost to implement (e.g., compared to applying lowpass or other types of filters, interpolation, or other processes in order to upsample and / or downsample outputs and / or inputs of a model).

[0028] Fig 1 depicts an example of a machine learning model-based image-to-image processing task 100a (e.g., synthetic zoom, image upsampling, super-resolution, blur or other artifact removal). The method of Fig. 1 includes obtaining an input image 101 at a first resolution and then subjecting it to an input processing step 110 that results in an effectively lower-resolution ‘image’ at a second resolution; a first stage 120a of image processing is performed at the second, lower ‘resolution.’ The input processing step 110 could be a downsampling step (e.g., applying linear, quadratic, biquadratic, cubic, or some other variety of downsampling) to render a lower-resolution image, at the second resolution, from the input image 101 at the first resolution. Alternatively, the input processing step 110 could include a concatenation or other pixel-combining step (e.g., a ‘space2depth’ process) to generate ‘composite’ pixels, at the second resolution, by combining sets of pixels (e.g., non-overlapping sets of pixels) of the image 101 at the first resolution. Part of the first stage 120a of image processing includes upsampling 125 (e.g., linear, quadratic, biquadratic, cubic, or some other variety of upsampling) the results of the image processing at the second resolution to a third, higher resolution and then performing a second stage 130a of image processing at the third resolution. Part of the second stage 130a of image processing includes upsampling 135 (e.g., linear, quadratic, biquadratic, cubic, or some other variety of upsampling) the results of the image processing at the third resolution to a fourth, highest resolution that is the target resolution of the output image 103a of the overall image processing process. A final, third stage 140a of image processing is then performed at the fourth resolution to generate the outputimage 103a.

[0029] In the example process 100a, the second stage 130a of image processing includes the execution of two convolutional layers 131, 133 at the third resolution followed by an upsampling step 135 and the third stage 140a of image processing includes the execution of two convolutional layers 141, 143 at the fourth resolution followed by an upsampling step 147; however, this is intended only as a non-limiting example embodiment, and alternative arrangements are possible. In the experimental results reported herein, such an example process may be referred to as an “RRDB” image-to-image process.

[0030] As noted above, such a process 100a, which includes extensive image processing at the highest “target” resolution of the output image as well as processing as one or more intermediate resolutions, including the computationally expensive process of upsampling from lower resolutions to such higher resolutions (e.g., 125, 135), can incur significant computational costs (e.g., with respect to processor cycles, memory use, network interconnect bandwidth use, power). Such costs may be amplified when the process 100a is implemented on a resource-constrained system (e.g., a cellphone) which may include a TPU or other processor specialized for the execution of machine learning models but which may include significantly fewer tensor processing units or other elements than the number of pixels (or other elements) at the terminal or other high-resolution image processing stages (e.g., 130a, 140a). In such examples, the methods described herein can be employed to adapt the image- to-image processing task to the available computational resources which generate output images that exhibit reduced hallucinations or other artifacts or that otherwise exhibit increased quality (e.g., relative to an implementation of the process 100a using similar amounts of computational resources).

[0031] Fig. 2A illustrates aspects of an example of such an improved process 100b. Fig.1 depicts an example of a machine learning model-based image-to-image processing task 100b (e.g., synthetic zoom, image upsampling, super-resolution, blur or other artifact removal). The method of Fig. 2A includes obtaining an input image 101 at a first resolution and then subjecting it to pixel combination step 110 that results in an effectively lower-resolution array of image elements (e.g., each element a concatenation of information from a corresponding set of pixels at the first resolution) at a second resolution; a first stage 120b of image processing is performed at the second, lower resolution. The pixel combination step 110 could include a concatenation or other pixel-combining step (e.g., a ‘space2depth’ process) to generate composite pixels or other types of image elements, at the second resolution, by combining setsof pixels (e.g., non-overlapping sets of pixels) of the image 101 at the first resolution. Part of the first stage 120b of image processing includes upsampling 125 (e.g., linear, quadratic, biquadratic, cubic, or some other variety of upsampling) the results of the image processing at the second resolution to a third, higher resolution and then performing a second stage 130b of image processing at the third resolution. Finally, a pixel expansion process 130 (e.g., a ‘depth2space’ process) is applied to partition each pixel or other element of the output from the second stage 130b into a respective block of pixels or other elements of the terminal model output image 103b. This can include a pixel shuffling or de-concatenation process, e.g., for a given element of the output of the second stage 130b, a first quarter of the given element could be partitioned into a first pixel (e.g., upper right pixel) of a corresponding block of pixels of the output image 103b, a second quarter of the given element could be partitioned into a second pixel (e.g., upper left pixel) of a corresponding block of pixels of the output image 103b, etc.

[0032] In the example process 100b, the second stage 130b of image processing includes the execution of two convolutional layers 132, 134 at the third resolution; however, this is intended only as a non-limiting example embodiment, and alternative arrangements are possible. In the experimental results reported herein, such an example process may be referred to as a “UBolt” image-to-image process.

[0033] Such a process 100b could be further improved (e.g., to adapt to available computational resources on a cellphone) while providing similar levels of output image quality by, e.g., performing substantially all image processing at a single reduced effective resolution, followed by an element-partitioning step to efficiently render an array of pixels or other elements at the target resolution. Fig.2B illustrates aspects of an example of such an improved process 100c. Fig.2B depicts an example of a machine learning model-based image-to-image processing task 100c (e.g., synthetic zoom, image upsampling, super-resolution, blur or other artifact removal). The method of Fig. 2C includes obtaining an input image 101 at a first resolution and then subjecting it to pixel combination step 110 that results in an effectively lower-resolution ‘image’ at a second resolution; a first stage 120c of image processing is performed at the second, lower ‘resolution.’ The pixel combination step 110 could include a concatenation or other pixel-combining step (e.g., a ‘space2depth’ process) to generate ‘composite’ pixels, at the second resolution, by combining sets of pixels (e.g., non-overlapping sets of pixels) of the image 101 at the first resolution. Finally, a pixel expansion process 130 (e.g., a ‘depth2space’ process) is applied to partition each pixel or other element of the output from the first stage 120c into a respective block of pixels or other elements of the terminalmodel output image 103c. This can include a pixel shuffling or de-concatenation process, e.g., for a given element of the output of the first stage 130c, a first quarter of the given element could be partitioned into a first pixel (e.g., upper right pixel) of a corresponding block of pixels of the output image 103c, a second quarter of the given element could be partitioned into a second pixel (e.g., upper left pixel) of a corresponding block of pixels of the output image 103c, etc.

[0034] The example process 100c depicts a variety of model architecture specifics (e.g., the inclusion of a final convolution layer 127 following the feedforward combination of the input of the first stage 120c with the output of the penultimate convolutional layer 123 of the first stage 120c); however, this is intended only as a non-limiting example embodiment, and alternative arrangements are possible. In the experimental results reported herein, such an example process may be referred to as a “TinyUBolt” image-to-image process.

[0035] As shown by way of non-limiting examples in Figs. 1 and 2A-B, processes 100a-c include first stages of image processing, at a lowest resolution, that include an initial at least one convolutional layer 121 followed by a number of residual-in-residual dense block layers 122, 124, 126, followed by another at least one convolutional layer 123. The output of at least one convolutional layer 123 is then combined, via a feedforward connection, with the input to the first stage to generate an intermediate first stage output. This intermediate first stage output is then, for the first 100a and second 100b example processes, subjected to upsampling 125 to a third resolution for further processing by a second stage 130a, 130b; for the third process 100c, this intermediate first stage output is subjected to a final at least one convolutional layer 127 before being subjected to partitioning 130 to render a final model output image 103c. Such configurations of the initial image processing stage, at the lowest resolution, are intended as non-limiting example embodiments; more or fewer of each type of layer, different types of layers, different arrangements of layers, more or fewer feedforward connections, or other modifications are possible within the scope of the present disclosure.

[0036] Further, in some applications, the image-to-image processing task may be broken down into a number of tiles, e.g., with each tile representing a respective sub-portion of an input image. In such examples, each tile may be processed as described herein (e.g., using one or more of the processes 100a-c) to generate respective output tiles, which are then combined (e.g., with overlap, and the overlapping regions filtered or otherwise combined to reduce tile-edge artifacts) to generate a single composite output image. For example, for a synthetic zoom, super-resolution, or other image-to-image processing task with a set targetresolution, an input image could be broken up into a number of tiles to reduce the latency (e.g., below a set maximum latency) and / or overall computation cost of the overall image processing task (e.g., by selecting individual image tile sizes to comport with a number of tensor processing units of a TPU or some other constraint of a local processing resource). This could include determining the number and / or size of such tiles based on the target image resolution and / or source image resolution (e.g., the resolution of a portion of a camera image to be synthetically zoomed in order to, e.g., simulate a greater zoom than is optically available in a camera).

[0037] The embodiments described herein allow the degree of combination / separation of pixels of the input / output, as well as the number of parameters, computational cost, and accuracy / quality of the resulting model, to be tailored to a desired application. For example, for super-resolution, un-blurring, artifact removal, or other image-to-image tasks, the embodiments described herein were able, using an architecture that includes a stack of residual- in-residual dense blocks (RRDB) operating at less than the target output resolution, to execute over 40% faster than conventional RRDB models (which use multiple upsampling stages and that include layers of processing at both full target resolution and intermediate resolution(s)) while exhibiting similar output accuracy and similar total number of model parameters.

[0038] A variety of model architectures can be improved using the embodiments described herein, combining (e.g., concatenating) pixels or other elements of an input image to allow the model to operate at an effectively reduced resolution (but potentially increased size of the representation of each composite pixel or other type of image element) and then, to generate the model output, de-concatenating, partitioning, or otherwise un-combining the pixels of the model output to generate pixels of an output image at a target resolution. In some examples, (e.g., super-resolution or other upsampling image-to-image tasks), the initial combining step could be omitted (e.g., the input could be at the lower, native resolution of the model), and only an end un-combining step applied to render, from the outputs of the model at the lower, native resolution, pixels or other elements of an output image at a higher target resolution.

[0039] The degree of combination of pixels of the input could be throttled or otherwise controlled in order to control a degree of upsampling applied to the input while maintaining the native resolution of the machine learning model at a constant level, thereby setting substantially constant the computational cost to upsample, to a single target resolution, input images at a range of different resolutions. For example, to upsample an input image 4 times,no combination of pixels of an input image could be applied (e.g., each pixel or other type of image element of the combined input to the model could be a corresponding pixel of the input image replicated sixteen times). In another example, the same model could be used only to de- blur or otherwise improve an input image without upsampling by combining four-pixel-wide blocks of pixels to generate each element of input at the native resolution of the model (e.g., by concatenating blocks of sixteen pixels of the input image to form each pixel of input to the model). Additionally or alternatively, as described above, an input image may be separated into a plurality of overlapping image tiles which are separately subjected to synthetic zoom, super- resolution, deblurring, artifact removal, image sharpening, or other image-to-image processing task(s) and the output tiles then combined (e.g., with some degree of filtering to reduce or remove tile edge effects) to generate a single composite output image. The number and / or size of such tiles could be determined in order to reduce a computational cost of the image processing task, to reduce a latency of the image processing task (e.g., to less than a specified latency), and / or to satisfy some other constraint or consideration with respect to a particular set of available computational resources, e.g., based on a source and / or target image resolution.

[0040] The specifics of the model architecture between the input combination step (or lack thereof, e.g., for upsampling tasks) and final un-combination step can vary according to an application. For example, the model could include a series of RRDB layers, convolution layers, or other types of layers. Such layers could be connected in series, in parallel, or in some other manner. For example, an output of a series of RRDB or other layers (e.g., a series of RRDB layers flanked by one or more convolutional layers) could be added to or otherwise combined with a feedforward copy or other version of the model input before being un- combined (optionally following an additional convolutional layer) to generate the target- resolution pixels of an output image (e.g., as depicted in the example of Figure 2B). In some examples, some of the computation of the model (e.g., one or more convolutional layers) could be performed at an intermediate resolution (i.e., between the lower ‘native’ resolution of the RRDB or other initial layers of the model and the higher ‘target’ resolution of the model output) in order to provide a trade-off between increased model accuracy and increased model execution cost. For example, an output of the model at a lower resolution (e.g., a version of the model input added or otherwise combined with the output of a stack of RRDB layers, optionally flanked by convolutional layers) could be upsampled (e.g., via interpolation) in order to provide an intermediate model output at an intermediate resolution. This intermediate output could then be further processed (e.g., by two or some other number of convolutional layers) at theintermediate resolution before the processed elements thereof are partitioned or otherwise un- combined to generate the separate pixels of the model output at the higher ‘target’ resolution. Figure 2A depicts an example of such an embodiment.

[0041] The embodiments described herein were implemented and experimentally assessed in order to evaluate their efficacy with respect to various image processing tasks (e.g., with respect to their ability to generate high-quality output images that exhibit reduced hallucinations or otherwise improved quality relative to alternative methods) and with respect to the computational costs thereof (e.g., with respect to latency, processor cycles, memory use, etc.). Specifically, an embodiment as in Fig.1 and an embodiment as in Fig.2B were compared with respect to image-to-image processing of a 1024x1024 pixel image and evaluated with respect to number of model parameters, computational cost in processor cycles, memory use, and latency as the number of RRDB blocks thereof was varied from 1 to 15. The embodiments described herein exhibited significant reductions with respect to all of the measured outcomes across all evaluated conditions.

[0042] The embodiments described herein also exhibit improvements with respect to the image quality of images generated thereby. Figs. 3A-D depict the relative quality of de- blurred versions of input images (“Input”) deblurred via the methods described herein (“TinyUBolt”) and via two alternative methods (“PSDeblur” and “RRDB 1x”). As shown, the embodiments described herein render higher quality, subjectively improved images. Indeed, the methods described herein achieved a DxOMARK zoom score of 155, the highest score and surpassing the next-highest score of only 142. A long-range zoom score of 69 was obtained by implementing the methods described herein, increasing from a score of 55 without these methods. The methods described herein were also able, for a specified image quality, to reduce the number of RRDB layers from 23 to 7, resulting in an overall 40% faster performance.

[0043] The methods described herein were also able to significantly increase the range of synthetic zoom that is possible within a specified maximum latency and image quality, from 25-30x to the greater range of 13-30x. Fig. 4A depicts the performance of the methods described herein (“TinyUBolt”) relative to prior methods (“RRDB”) for a synthetic zoom task (increasing the resolution of a portion of a camera image by a specified factor, to a target image resolution of 12MP, thereby implementing various synthetic zoom levels). As shown, for a specified maximum latency of 3000 milliseconds, the methods described herein were able to be employed across a range of zooms from 30x down to 13.6x, while the compared previous method was only able to be employed down to 22.6x.

[0044] As noted above, the methods described herein can be applied separately to multiple different tiles of a single input image, with the resulting output tiles composited together (e.g., with filtering or other post-processing steps applied to remove or reduce tile- edge effects) to generate a single overall output image for the input image. This can be done to, e.g., optimize a particular image-to-image processing task for the specific computational resources of a target platform (e.g., to increase output image quality or improve some other metric while maintaining latency below a specified maximum or satisfying some other constraint(s)). The number, size, or other aspects of such image tiles could be determined based on the resolution of the input and / or target output images in order to satisfy the specified constraint(s) for a given platform. Fig. 4B depicts the latency of a tile-based version of the image-to-image processing methods described herein across a variety of tile sizes (e.g., 96x96, 128x96… up to 384x384) for a particular platform for a variety of different input and output image resolutions (1.4MP to 5.6MP, 2.2MP to 8.8MP, and 3.1MP to 12.4MP). As shown, the latency of the task generally increases with the resolution of the input / output images and with the size of the individual tiles. Thus, for a given specified maximum latency, a different tile size may be selected for different input and / or output image resolutions.

[0045] Such significant reductions in latency and / or computational cost, as facilitated by the embodiments described herein, enable a variety of new applications on various resource- limited platforms (e.g., cellphones). For example, the improved super-resolution and related image-to-image processing tasks (e.g., deblur, removal of motion or other image artifacts) can allow for the implementation of synthetic zoom across a wider range of magnification factors (e.g., 13-30x). This can allow a cellphone or other camera-including and computational resource-limited platform to obtain a desired range of functional zooms (using the synthetic zoom functionality) while employing a camera with a narrower range of optical zooms, thus allowing such a camera to be smaller, lighter, less complex, or otherwise improved relative to an alternative camera that is able to implement a correspondingly greater range of optical zooms. In another example, such a resource-limited platform could employ the methods described herein to locally implement synthetic zoom, deblurring, image sharpening, motion blur reduction, or other image-to-image processing tasks on the many image frames of video clips (e.g., short-form video), thereby avoiding the bandwidth, power, or other costs that could be incurred by instead uploading representations of such video clips (potentially non- compressed) for remote processing, followed by reception of the processed versions of such video clips from the remote processor.II. Example Machine Learning Models and Training Thereof

[0046] A machine learning model or layer or other aspect thereof as described herein may include, but is not limited to: an artificial neural network (e.g., Transformers, layered models wherein each layer includes two or more sub-layers one or more of which could include artificial neural networks, convolutional neural networks, a recurrent neural network, a Bayesian network, a hidden Markov model, a Markov decision process, a logistic regression function, a support vector machine, a suitable statistical machine learning algorithm, and / or a heuristic machine learning system), a support vector machine, a regression tree, an ensemble of regression trees (also referred to as a regression forest), a decision tree, an ensemble of decision trees (also referred to as a decision forest), or some other machine learning model architecture or combination of architectures.

[0047] An artificial neural network (ANN) could be configured in a variety of ways. For example, the ANN could include two or more layers, could include units having linear, logarithmic, or otherwise-specified output functions, could include fully or otherwise- connected neurons, could include recurrent and / or feed-forward connections between neurons in different layers, could include filters or other elements to process input information and / or information passing between layers, or could be configured in some other way to facilitate the processing of input images, sets or subsets (e.g., tiles) of pixels of input images, elements representing concatenated or otherwise combined blocks or sets of pixels of input images, downstream representations of pixels or combinations of pixels determined by the operation of one or more layers or sublayers of a multi-layer model, and / or individual vectors (e.g., vectors representing intensity, color, or other information about one or more pixels or other elements of an input, output, or intermediate image).

[0048] An ANN could include one or more filters that could be applied to the input and the outputs of such filters could then be applied to the inputs of one or more neurons of the ANN. For example, such an ANN could be or could include a convolutional neural network (CNN). Convolutional neural networks are a variety of ANNs that are configured to facilitate ANN-based classification, filtering, generation, or other processing based on images or other large-dimensional inputs whose elements are organized within two or more dimensions. The organization of the ANN along these dimensions may be related to some structure in the input structure (e.g., as relative location within the two-dimensional space of an array of vectors can be related to locations of respective pixels of an input image).

[0049] In example embodiments, a CNN includes at least one two-dimensional (orhigher-dimensional) filter that is applied to an input; the filtered input is then applied to neurons of the CNN (e.g., of a convolutional layer of the CNN). The convolution of such a filter and an input could represent the color values of a pixel or a group of pixels from the input, in embodiments where the input is an image. A set of neurons of a CNN could receive respective inputs that are determined by applying the same filter to an input. Additionally or alternatively, a set of neurons of a CNN could be associated with respective different filters and could receive respective inputs that are determined by applying the respective filter to the input. Such filters could be trained during training of the CNN or could be pre-specified. For example, such filters could represent wavelet filters, center-surround filters, biologically-inspired filter kernels (e.g., from studies of animal visual processing receptive fields), or some other pre-specified filter patterns.

[0050] A CNN or other variety of ANN could include multiple convolutional layers (e.g., corresponding to respective different filters and / or features), pooling layers, rectification layers, fully connected layers, or other types of layers. Convolutional layers of a CNN represent convolution of an input image, or of some other input (e.g., of a filtered, downsampled, or otherwise-processed version of an input image), with a filter. Pooling layers of a CNN apply non-linear downsampling to higher layers of the CNN, e.g., by applying a maximum, average, L2-norm, or other pooling function to a subset of neurons, outputs, or other features of the higher layer(s) of the CNN. Rectification layers of a CNN apply a rectifying nonlinear function (e.g., a non-saturating activation function, a sigmoid function) to outputs of a higher layer. Fully connected layers of a CNN receive inputs from many or all of the neurons in one or more higher layers of the CNN. The outputs of neurons of one or more fully connected layers (e.g., a final layer of an ANN or CNN) could be used to determine information about areas of an input image (e.g., for each of the pixels of an input image) or for the image as a whole.

[0051] FIG.5 shows diagram 500 illustrating a training phase 502 and an inference phase 504 of trained machine learning model(s) 532, in accordance with example embodiments. Some machine learning techniques involve training one or more machine learning algorithms, on an input set of training data to recognize patterns in the training data and provide output inferences and / or predictions about (patterns in the) training data. Such output could take the form of filtered or otherwise modified versions of the input, e.g., an input an image at a first resolution could be super-resolved or otherwise modified by the machine learning model into an output image that represents the input image at a higher resolution, at a higher quality (e.g., with blur, motion artifacts, or other imperfections reduced or removed), or otherwise modified. Theresulting trained machine learning algorithm can be termed as a trained machine learning model. For example, FIG. 5 shows training phase 502 where one or more machine learning algorithms 520 are being trained on training data 510 to become trained machine learning model 532. Then, during inference phase 504, trained machine learning model 532 can receive input data 530 and one or more inference / prediction requests 540 (perhaps as part of input data 530) and responsively provide as an output one or more inferences and / or predictions 550.

[0052] As such, trained machine learning model(s) 532 can include one or more models of one or more machine learning algorithms 520. Machine learning algorithm(s) 520 may include, but are not limited to: an artificial neural network (e.g., a herein-described convolutional neural networks, a recurrent neural network, a Bayesian network, a hidden Markov model, a Markov decision process, a logistic regression function, a support vector machine, a suitable statistical machine learning algorithm, and / or a heuristic machine learning system), a Transformer, a support vector machine, a regression tree, an ensemble of regression trees (also referred to as a regression forest), a decision tree, an ensemble of decision trees (also referred to as a decision forest), or some other machine learning model architecture or combination of architectures. For example, the trained machine learning model(s) 532 could include a plurality of artificial neural networks and other elements related to such networks (e.g., mixing or weighting matrices, attention heads or other attentional mechanisms, sums, products, feedforward connections) arranged according to the multi-layer and sublayer architecture of a Transformer or similar model architecture designed to process input sequences. Machine learning algorithm(s) 520 may be supervised or unsupervised, and may implement any suitable combination of online and offline learning.

[0053] In some examples, machine learning algorithm(s) 520 and / or trained machine learning model(s) 532 can be accelerated using on-device coprocessors, such as graphic processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), and / or application specific integrated circuits (ASICs). Such on-device coprocessors can be used to speed up machine learning algorithm(s) 520 and / or trained machine learning model(s) 532. In some examples, trained machine learning model(s) 532 can be trained, reside and execute to provide inferences on a particular computing device, and / or otherwise can make inferences for the particular computing device.

[0054] During training phase 502, machine learning algorithm(s) 520 can be trained by providing at least training data 510 as training input using unsupervised, supervised, semi- supervised, and / or reinforcement learning techniques. Unsupervised learning involvesproviding a portion (or all) of training data 510 to machine learning algorithm(s) 520 and machine learning algorithm(s) 520 determining one or more output inferences based on the provided portion (or all) of training data 510. Supervised learning involves providing a portion of training data 510 to machine learning algorithm(s) 520, with machine learning algorithm(s) 520 determining one or more output inferences based on the provided portion of training data 510, and the output inference(s) are either accepted or corrected based on correct results associated with training data 510. In some examples, supervised learning of machine learning algorithm(s) 520 can be governed by a set of rules and / or a set of labels for the training input, and the set of rules and / or set of labels may be used to correct inferences of machine learning algorithm(s) 520.

[0055] Semi-supervised learning involves having correct results for part, but not all, of training data 510. During semi-supervised learning, supervised learning is used for a portion of training data 510 having correct results, and unsupervised learning is used for a portion of training data 510 not having correct results. Reinforcement learning involves machine learning algorithm(s) 520 receiving a reward signal regarding a prior inference, where the reward signal can be a numerical value. During reinforcement learning, machine learning algorithm(s) 520 can output an inference and receive a reward signal in response, where machine learning algorithm(s) 520 are configured to try to maximize the numerical value of the reward signal. In some examples, reinforcement learning also utilizes a value function that provides a numerical value representing an expected total of the numerical values provided by the reward signal over time. In some examples, machine learning algorithm(s) 520 and / or trained machine learning model(s) 532 can be trained using other machine learning techniques, including but not limited to, incremental learning and curriculum learning.

[0056] In some examples, machine learning algorithm(s) 520 and / or trained machine learning model(s) 532 can use transfer learning techniques. For example, transfer learning techniques can involve trained machine learning model(s) 532 being pre-trained on one set of data and additionally trained using training data 510. More particularly, machine learning algorithm(s) 520 can be pre-trained on data from one or more computing devices and a resulting trained machine learning model provided to computing device CD1, where CD1 is intended to execute the trained machine learning model during inference phase 504. Then, during training phase 502, the pre-trained machine learning model can be additionally trained using training data 510, where training data 510 can be derived from kernel and non-kernel data of computing device CD1. This further training of the machine learning algorithm(s) 520 and / or the pre-trained machine learning model using training data 510 of CD1’s data can be performed using either supervised or unsupervised learning. Once machine learning algorithm(s) 520 and / or the pre-trained machine learning model has been trained on at least training data 510, training phase 502 can be completed. The trained resulting machine learning model can be utilized as at least one of trained machine learning model(s) 532.

[0057] In particular, once training phase 502 has been completed, trained machine learning model(s) 532 can be provided to a computing device, if not already on the computing device. Inference phase 504 can begin after trained machine learning model(s) 532 are provided to computing device CD1.

[0058] During inference phase 504, trained machine learning model(s) 532 can receive input data 530 and generate and output one or more corresponding inferences and / or predictions 550 about input data 530. As such, input data 530 can be used as an input to trained machine learning model(s) 532 for providing corresponding inference(s) and / or prediction(s) 550 to kernel components and non-kernel components. For example, trained machine learning model(s) 532 can generate inference(s) and / or prediction(s) 550 in response to one or more inference / prediction requests 540. In some examples, trained machine learning model(s) 532 can be executed by a portion of other software. For example, trained machine learning model(s) 532 can be executed by an inference or prediction daemon to be readily available to provide inferences and / or predictions upon request. Input data 530 can include data from computing device CD1 executing trained machine learning model(s) 532 and / or input data from one or more computing devices other than CD1.

[0059] Input data 530 can include a collection of images provided by one or more sources. The collection of images can include photographs, simulated images (e.g., computer-generated images based on a model of a real or artificial environment), cartoon or other human-generated images, images modified to artificially include blur, motion artifacts, or other artifacts (e.g., based on a model of a particular camera or other specific image generation apparatus), and / or other images. Other types of input data are possible as well.

[0060] Inference(s) and / or prediction(s) 550 can include output images at the same or different resolution as input images, e.g., super-resolved, zoomed, or otherwise higher-resolution images than input images and / or other output data produced by trained machine learning model(s) 532 operating on input data 530 (and training data 510). In some examples, trained machine learning model(s) 532 can use output inference(s) and / or prediction(s) 550 as input feedback 560. Trained machine learning model(s) 532 can also rely on past inferences as inputs forgenerating new inferences. III. Example Systems

[0061] Figure 6 illustrates an example computing device 600 that may be used to implement the methods described herein. By way of example and without limitation, computing device 600 may be a cellular mobile telephone (e.g., a smartphone), a computer (such as a desktop, notebook, tablet, or handheld computer, a server), elements of a cloud computing system, a robot, a drone, an autonomous vehicle, or some other type of device. It should be understood that computing device 600 may represent a physical computing device such as a server, a particular physical hardware platform on which a machine learning application operates in software, or other combinations of hardware and software that are configured to carry out machine learning functions as described herein.

[0062] As shown in Figure 6, computing device 600 may include a communication interface 602, a user interface 604, a controller 606 (which may include one or more processors), a camera 607, and data storage 608, all of which may be communicatively linked together by a system bus, network, or other connection mechanism 610.

[0063] Communication interface 602 may function to allow computing device 600 to communicate, using analog or digital modulation of electric, magnetic, electromagnetic, optical, or other signals, with other devices, access networks, and / or transport networks. Thus, communication interface 602 may facilitate circuit-switched and / or packet-switched communication, such as plain old telephone service (POTS) communication and / or Internet protocol (IP) or other packetized communication. For instance, communication interface 602 may include a chipset and antenna arranged for wireless communication with a radio access network or an access point. Also, communication interface 602 may take the form of or include a wireline interface, such as an Ethernet, Universal Serial Bus (USB), or High-Definition Multimedia Interface (HDMI) port. Communication interface 602 may also take the form of or include a wireless interface, such as a Wi-Fi, BLUETOOTH®, global positioning system (GPS), or wide-area wireless interface (e.g., WiMAX or 3GPP Long-Term Evolution (LTE)). However, other forms of physical layer interfaces and other types of standard or proprietary communication protocols may be used over communication interface 602. Furthermore, communication interface 602 may comprise multiple physical communication interfaces (e.g., a Wi-Fi interface, a BLUETOOTH® interface, and a wide-area wireless interface).

[0064] In some embodiments, communication interface 602 may function to allowcomputing device 600 to communicate, with other devices, remote servers, access networks, and / or transport networks. For example, the communication interface 602 may function to access one or more machine learning models and / or input(s) therefor via communication with a remote server or other remote device or system in order to allow the computing device 600 to use the machine learning model(s) to generate outputs (e.g., super-resolved, artificially zoomed, or otherwise improved versions of input images) based on input data. For example, the computing system 600 could be an inference server and the remote system could be a smartphone that generated an image (e.g., an image from a camera) to be applied to a machine learning model in order to determine information about the contents of the image, e.g., to determine a zoomed in, super-resolved, or otherwise improved version of the image.

[0065] User interface 604 may function to allow computing device 600 to interact with a user, for example to receive input from and / or to provide output to the user. Thus, user interface 604 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, and so on. User interface 604 may also include one or more output components such as a display screen which, for example, may be combined with a presence-sensitive panel. The display screen may be based on CRT, LCD, and / or LED technologies, or other technologies now known or later developed. User interface 604 may also be configured to generate audible output(s), via a speaker, speaker jack, audio output port, audio output device, earphones, and / or other similar devices.

[0066] Controller 606 may comprise one or more general purpose processors – e.g., microprocessors – and / or one or more special purpose processors – e.g., digital signal processors (DSPs), graphics processing units (GPUs), floating point units (FPUs), network processors, tensor processing units (TPUs), or application-specific integrated circuits (ASICs). In some instances, special purpose processors may be capable of image processing (e.g., application of CNN kernels or other filters to images via, e.g., convolution), machine learning model training, execution, and / or inference, among other applications or functions. Data storage 608 may include one or more volatile and / or non-volatile storage components, such as magnetic, optical, flash, or organic storage, and may be integrated in whole or in part with controller 606. For example, a portion of the data storage 608 may be implemented as cache or other on-chip memory of a graphics processing unit or tensor processing unit integrated circuit and / or as RAM or some other variety of storage that is collocated with a GPU or TPU, e.g., on a graphics card, tensor acceleration card, or other semi-discrete subsystem of the overall system600. Such storage could be used to store parameters that define a machine learning model (e.g., weights or other parameters of units of a multi-layer neural network or other multi-unit machine learning model). Data storage 608 may include removable and / or non-removable components.

[0067] Controller 606 may be capable of executing program instructions 618 (e.g., compiled or non-compiled program logic and / or machine code) stored in data storage 608 to carry out the various functions described herein. Therefore, data storage 608 may include a non-transitory computer-readable medium, having stored thereon program instructions that, upon execution by computing device 600, cause computing device 600 to carry out any of the methods, processes, or functions disclosed in this specification and / or the accompanying drawings. The execution of program instructions 618 by controller 606 may result in controller 606 using data 612.

[0068] By way of example, program instructions 618 may include an operating system 622 (e.g., an operating system kernel, device driver(s), and / or other modules) and one or more application programs 620 (e.g., functions for executing trained machine learning models and / or training such models) installed on computing device 600. Data 612 may include stored training data 614 (e.g., images and zoomed, deblurred, upsampled, super-resolved, artifact-removed, or otherwise improved versions of the images) that could be used to train (e.g., to pre-train, to fine-tune) one or more machine learning models 616.

[0069] Application programs 620 may communicate with operating system 622 through one or more application programming interfaces (APIs). These APIs may facilitate, for instance, application programs 620 reading and / or writing a trained machine learning model 616, transmitting or receiving information via communication interface 602, receiving and / or displaying information on user interface 604, and so on.

[0070] Application programs 620 may take the form of “apps” that could be downloadable to computing device 600 through one or more online application stores or application markets (via, e.g., the communication interface 602). However, application programs can also be installed on computing device 600 in other ways, such as via a web browser or through a physical interface (e.g., a USB port) of the computing device 600. IV. Example Methods

[0071] Figure 7 is a flowchart of a method 700 as described herein. The method 700 includes obtaining a model input that at least partially represents an input image, wherein the model input comprises a two-dimensional array of elements at a first resolution (710). Themethod 700 additionally includes applying a first set of one or more layers of a machine learning model to the model input to generate a first intermediate model output, wherein the first intermediate model output comprises a two-dimensional array of elements at the first resolution (720). The method 700 yet further includes upsampling the first intermediate model output to generate a second intermediate model output that comprises a two-dimensional array of elements at a second resolution, wherein the second resolution is greater than the first resolution (730). The method 700 also includes applying a second set of one or more layers of the machine learning model to the second intermediate model output to generate a third intermediate model output, wherein the third intermediate model output comprises a two- dimensional array of elements at the second resolution (740). The method 700 also includes partitioning the third intermediate model output to generate a terminal model output that comprises a two-dimensional array of elements at a third resolution, wherein the third resolution is greater than the second resolution, and wherein partitioning the third intermediate model output to generate the terminal model output comprises partitioning each element of the third intermediate model output into a respective block of elements of the terminal model output (750). The method 700 could include additional or alternative steps or features.

[0072] Figure 8 is a flowchart of a method 800 as described herein. The method 800 includes obtaining a model input that at least partially represents an input image, wherein the model input comprises a two-dimensional array of elements at a first resolution (810). The method 800 additionally includes applying a first set of one or more layers of a machine learning model to the model input to generate a first intermediate model output, wherein the first intermediate model output comprises a two-dimensional array of elements at the first resolution, and wherein at least two layers of the first set of one or more layers are residual-in- residual dense blocks (820). The method 800 yet further includes partitioning the first intermediate model output to generate a terminal model output that comprises a two- dimensional array of elements at a second resolution, wherein the second resolution is greater than the first resolution, and wherein partitioning the first intermediate model output to generate the terminal model output comprises partitioning each element of the first intermediate model output into a respective block of elements of the terminal model output (830). The method 800 could include additional or alternative steps or features.

[0073] Figure 9 is a flowchart of a method 900 as described herein. The method 900 includes obtaining a model input from an input image, wherein the input image comprises a plurality of pixels at a first resolution and the model input comprises a two-dimensional arrayof elements at a second resolution that is less than the first resolution, wherein obtaining the model input comprises combining sets of pixels of the input image to generate respective elements of the model input such that each element of the model input represents a respective block of pixels of the input image (910). The method 900 additionally includes applying a first set of one or more layers of a machine learning model to the model input to generate a first intermediate model output, wherein the first intermediate model output comprises a two- dimensional array of elements at the second resolution (920). The method 900 yet further includes partitioning the first intermediate model output to generate a terminal model output that comprises a two-dimensional array of elements at a third resolution, wherein the third resolution is greater than the second resolution, and wherein partitioning the first intermediate model output to generate the terminal model output comprises partitioning each element of the first intermediate model output into a respective block of elements of the terminal model output (930). The method 900 could include additional or alternative steps or features. V. Conclusion

[0074] The above detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying figures. In the figures, similar symbols typically identify similar components, unless the context indicates otherwise. The illustrative embodiments described in the detailed description, figures, and claims are not meant to be limiting. Other embodiments can be utilized, and other changes can be made, without departing from the scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are explicitly contemplated herein.

[0075] With respect to any or all of the message flow diagrams, scenarios, and flowcharts in the figures and as discussed herein, each step, block and / or communication may represent a processing of information and / or a transmission of information in accordance with example embodiments. Alternative embodiments are included within the scope of these example embodiments. In these alternative embodiments, for example, functions described as steps, blocks, transmissions, communications, requests, responses, and / or messages may be executed out of order from that shown or discussed, including in substantially concurrent or in reverse order, depending on the functionality involved. Further, more or fewer steps, blocks and / or functions may be used with any of the message flow diagrams, scenarios, and flowcharts discussed herein, and these message flow diagrams, scenarios, and flow charts may be combined with one another, in part or in whole.

[0076] A step or block that represents a processing of information may correspond to circuitry that can be configured to perform the specific logical functions of a herein-described method or technique. Alternatively or additionally, a step or block that represents a processing of information may correspond to a module, a segment, or a portion of program code (including related data). The program code may include one or more instructions executable by a processor for implementing specific logical functions or actions in the method or technique. The program code and / or related data may be stored on any type of computer-readable medium, such as a storage device, including a disk drive, a hard drive, or other storage media.

[0077] The computer-readable medium may also include non-transitory computer- readable media such as computer-readable media that stores data for short periods of time like register memory, processor cache, and / or random access memory (RAM). The computer- readable media may also include non-transitory computer-readable media that stores program code and / or data for longer periods of time, such as secondary or persistent long term storage, like read only memory (ROM), optical or magnetic disks, and / or compact-disc read only memory (CD-ROM), for example. The computer-readable media may also be any other volatile or non-volatile storage systems. A computer-readable medium may be considered a computer- readable storage medium, for example, or a tangible storage device.

[0078] Moreover, a step or block that represents one or more information transmissions may correspond to information transmissions between software and / or hardware modules in the same physical device. However, other information transmissions may be between software modules and / or hardware modules in different physical devices.

[0079] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and are not intended to be limiting, with the true scope being indicated by the following claims. VII. Enumerated Example Embodiments

[0080] Embodiments of the present disclosure may thus relate to one of the enumerated example embodiments (EEEs) listed below. It will be appreciated that features indicated with respect to one EEE can be combined with other EEEs.

[0081] EEE 1 is a method comprising: (i) obtaining a model input that at least partiallyrepresents an input image, wherein the model input comprises a two-dimensional array of elements at a first resolution; (ii) applying a first set of one or more layers of a machine learning model to the model input to generate a first intermediate model output, wherein the first intermediate model output comprises a two-dimensional array of elements at the first resolution; (iii) upsampling the first intermediate model output to generate a second intermediate model output that comprises a two-dimensional array of elements at a second resolution, wherein the second resolution is greater than the first resolution; (iv) applying a second set of one or more layers of the machine learning model to the second intermediate model output to generate a third intermediate model output, wherein the third intermediate model output comprises a two-dimensional array of elements at the second resolution; and (v) partitioning the third intermediate model output to generate a terminal model output that comprises a two-dimensional array of elements at a third resolution, wherein the third resolution is greater than the second resolution, and wherein partitioning the third intermediate model output to generate the terminal model output comprises partitioning each element of the third intermediate model output into a respective block of elements of the terminal model output.

[0082] EEE 2 is the method of EEE 1, wherein applying the first set of one or more layers of the machine learning model to the model input to generate the first intermediate model output comprises: (i) applying at least one convolutional layer to the model input to generate a fourth intermediate model output that comprises a two-dimensional array of elements at the first resolution; (ii) applying a third set of layers of the machine learning model to the fourth intermediate model output to generate a fifth intermediate model output that comprises a two- dimensional array of elements at the first resolution; (iii) applying at least one convolutional layer to the fifth intermediate model output to generate a sixth intermediate model output that comprises a two-dimensional array of elements at the first resolution; and (iv) combining the sixth intermediate model output and the model input to generate the first intermediate model output.

[0083] EEE 3 is the method of EEE 2, wherein the second set of one or more layers of the machine learning model comprises at least two convolutional layers.

[0084] EEE 4 is the method of any of EEEs 2-3, wherein at least two layers of the third set of one or more layers are residual-in-residual dense blocks.

[0085] EEE 5 is the method of any of EEEs 1-4, wherein at least two layers of the first set of one or more layers are residual-in-residual dense blocks.

[0086] EEE 6 is the method of any of EEEs 1-5, wherein the machine learning model is trained to generate the terminal model output as an upsampled version of the model input.

[0087] EEE 7 is the method of any of EEEs 1-5, wherein the machine learning model is trained to generate the terminal model output as a de-blurred version of the model input.

[0088] EEE 8 is the method of any of EEEs 1-5, wherein the machine learning model is trained to generate the terminal model output as a version of the model input that has had compression artifacts removed therefrom.

[0089] EEE 9 is the method of any of EEEs 1-8, wherein the input image comprises a plurality of pixels at a fourth resolution that is greater than the first resolution, and wherein obtaining the model input comprises combining sets of pixels of the input image to generate respective elements of the model input such that each element of the model input represents a respective block of pixels of the input image.

[0090] EEE 10 is the method of EEE 9, wherein combining sets of pixels of the input image to generate respective elements of the model input comprises concatenating representations of sets of the pixels of the input image such that each element of the model input represents a concatenation of the representations of a respective block of pixels of the input image.

[0091] EEE 11 is a computer-implemented method comprising: (i) obtaining a model input that at least partially represents an input image, wherein the model input comprises a two- dimensional array of elements at a first resolution; (ii) applying a first set of one or more layers of a machine learning model to the model input to generate a first intermediate model output, wherein the first intermediate model output comprises a two-dimensional array of elements at the first resolution, and wherein at least two layers of the first set of one or more layers are residual-in-residual dense blocks; and (iii) partitioning the first intermediate model output to generate a terminal model output that comprises a two-dimensional array of elements at a second resolution, wherein the second resolution is greater than the first resolution, and wherein partitioning the first intermediate model output to generate the terminal model output comprises partitioning each element of the first intermediate model output into a respective block of elements of the terminal model output.

[0092] EEE 12 is the method of EEE 11, wherein applying the first set of one or more layers of the machine learning model to the model input to generate the first intermediate model output comprises: (i) applying at least one convolutional layer to the model input to generate a second intermediate model output that comprises a two-dimensional array of elements at thefirst resolution; (ii) applying a second set of layers of the machine learning model to the second intermediate model output to generate a third intermediate model output that comprises a two- dimensional array of elements at the first resolution; (iii) applying at least one convolutional layer to the third intermediate model output to generate a fourth intermediate model output that comprises a two-dimensional array of elements at the first resolution; and (iv) combining the fourth intermediate model output and the model input to generate the first intermediate model output.

[0093] EEE 13 is the method of EEE 11, wherein applying the first set of one or more layers of the machine learning model to the model input to generate the first intermediate model output comprises: (i) applying at least one convolutional layer to the model input to generate a second intermediate model output that comprises a two-dimensional array of elements at the first resolution; (ii) applying a second set of layers of the machine learning model to the second intermediate model output to generate a third intermediate model output that comprises a two- dimensional array of elements at the first resolution; (iii) applying at least one convolutional layer to the third intermediate model output to generate a fourth intermediate model output that comprises a two-dimensional array of elements at the first resolution; (iv) combining the fourth intermediate model output and the model input to generate a fifth intermediate model output that comprises a two-dimensional array of elements at the first resolution; and (v) applying at least one convolutional layer to the fifth intermediate model output to generate the terminal model output.

[0094] EEE 14 is the method of any of EEEs 11-13, wherein the machine learning model is trained to generate the terminal model output as an upsampled version of the model input.

[0095] EEE 15 is the method of any of EEEs 11-13, wherein the machine learning model is trained to generate the terminal model output as a de-blurred version of the model input.

[0096] EEE 16 is the method of any of EEEs 11-13, wherein the machine learning model is trained to generate the terminal model output as a version of the model input that has had compression artifacts removed therefrom.

[0097] EEE 17 is the method of any of EEEs 11-16, wherein the input image comprises a plurality of pixels at a third resolution that is greater than the first resolution, and wherein obtaining the model input comprises combining sets of pixels of the input image to generate respective elements of the model input such that each element of the model input represents arespective block of pixels of the input image.

[0098] EEE 18 is the method of EEE 17, wherein combining sets of pixels of the input image to generate respective elements of the model input comprises concatenating representations of sets of the pixels of the input image such that each element of the model input represents a concatenation of the representations of a respective block of pixels of the input image.

[0099] EEE 19 is a computer-implemented method comprising: (i) obtaining a model input from an input image, wherein the input image comprises a plurality of pixels at a first resolution and the model input comprises a two-dimensional array of elements at a second resolution that is less than the first resolution, wherein obtaining the model input comprises combining sets of pixels of the input image to generate respective elements of the model input such that each element of the model input represents a respective block of pixels of the input image; (ii) applying a first set of one or more layers of a machine learning model to the model input to generate a first intermediate model output, wherein the first intermediate model output comprises a two-dimensional array of elements at the second resolution; and (iii) partitioning the first intermediate model output to generate a terminal model output that comprises a two- dimensional array of elements at a third resolution, wherein the third resolution is greater than the second resolution, and wherein partitioning the first intermediate model output to generate the terminal model output comprises partitioning each element of the first intermediate model output into a respective block of elements of the terminal model output.

[0100] EEE 20 is the method of EEE 19, wherein applying the first set of one or more layers of the machine learning model to the model input to generate the first intermediate model output comprises: (i) applying at least one convolutional layer to the model input to generate a second intermediate model output that comprises a two-dimensional array of elements at the second resolution; (ii) applying a second set of layers of the machine learning model to the second intermediate model output to generate a third intermediate model output that comprises a two-dimensional array of elements at the second resolution; (iii) applying at least one convolutional layer to the third intermediate model output to generate a fourth intermediate model output that comprises a two-dimensional array of elements at the second resolution; and (iv) combining the fourth intermediate model output and the model input to generate the first intermediate model output.

[0101] EEE 21 is the method of EEE 19, wherein applying the first set of one or more layers of the machine learning model to the model input to generate the first intermediate modeloutput comprises: (i) applying at least one convolutional layer to the model input to generate a second intermediate model output that comprises a two-dimensional array of elements at the second resolution; (ii) applying a second set of layers of the machine learning model to the second intermediate model output to generate a third intermediate model output that comprises a two-dimensional array of elements at the second resolution; (iii) applying at least one convolutional layer to the third intermediate model output to generate a fourth intermediate model output that comprises a two-dimensional array of elements at the second resolution; (iv) combining the fourth intermediate model output and the model input to generate a fifth intermediate model output that comprises a two-dimensional array of elements at the second resolution; and (v) applying at least one convolutional layer to the fifth intermediate model output to generate the terminal model output.

[0102] EEE 22 is the method of any of EEEs 20-21, wherein at least two layers of the first set of one or more layers are residual-in-residual dense blocks

[0103] EEE 23 is the method of any of EEEs 19-22, wherein the machine learning model is trained to generate the terminal model output as an upsampled version of the model input.

[0104] EEE 24 is the method of any of EEEs 19-22, wherein the machine learning model is trained to generate the terminal model output as a de-blurred version of the model input.

[0105] EEE 25 is the method of any of EEEs 19-22, wherein the machine learning model is trained to generate the terminal model output as a version of the model input that has had compression artifacts removed therefrom.

[0106] EEE 26 is the method of any of EEEs 19-25, wherein combining sets of pixels of the input image to generate respective elements of the model input comprises concatenating representations of sets of the pixels of the input image such that each element of the model input represents a concatenation of the representations of a respective block of pixels of the input image.

[0107] EEE 27 is the method of any of EEEs 19-26, wherein the first resolution and the third resolution are the same.

[0108] EEE 28 is the method of any preceding EEE, wherein the model input represents a first tile of the input image, wherein the terminal model output is a first terminal model output, and wherein the method further comprises: (i) determining, from a second tile of the input image, a second terminal model output; and (ii) generating a composite output image bycombining the first terminal model output and the second terminal model output.

[0109] EEE 29 is the method of EEE 28, further comprising: determining a size of the first tile and the second tile based on a resolution of the input image and a resolution of the composite output image such that generation of the composite output image from the input image takes less than a specified maximum latency.

[0110] EEE 30 is a non-transitory computer readable medium having stored thereon program instructions executable by at least one processor to cause the at least one processor to perform the method of any preceding EEE.

[0111] EEE 31 is a system comprising: (i) at least one processor; and (ii) a non- transitory computer-readable medium, having stored therein instructions executable by the at least one processor to cause the system to perform the method of any of EEEs 1-29.

Claims

CLAIMS What is claimed is:

1. A computer-implemented method comprising: obtaining a model input that at least partially represents an input image, wherein the model input comprises a two-dimensional array of elements at a first resolution; applying a first set of one or more layers of a machine learning model to the model input to generate a first intermediate model output, wherein the first intermediate model output comprises a two-dimensional array of elements at the first resolution; upsampling the first intermediate model output to generate a second intermediate model output that comprises a two-dimensional array of elements at a second resolution, wherein the second resolution is greater than the first resolution; applying a second set of one or more layers of the machine learning model to the second intermediate model output to generate a third intermediate model output, wherein the third intermediate model output comprises a two-dimensional array of elements at the second resolution; and partitioning the third intermediate model output to generate a terminal model output that comprises a two-dimensional array of elements at a third resolution, wherein the third resolution is greater than the second resolution, and wherein partitioning the third intermediate model output to generate the terminal model output comprises partitioning each element of the third intermediate model output into a respective block of elements of the terminal model output.

2. The method of claim 1, wherein applying the first set of one or more layers of the machine learning model to the model input to generate the first intermediate model output comprises: applying at least one convolutional layer to the model input to generate a fourth intermediate model output that comprises a two-dimensional array of elements at the first resolution; applying a third set of layers of the machine learning model to the fourth intermediate model output to generate a fifth intermediate model output that comprises a two-dimensional array of elements at the first resolution; applying at least one convolutional layer to the fifth intermediate model output to generate a sixth intermediate model output that comprises a two-dimensional array of elements at the first resolution; and combining the sixth intermediate model output and the model input to generate the first intermediate model output.

3. The method of claim 2, wherein the second set of one or more layers of the machine learning model comprises at least two convolutional layers.

4. The method of any of claims 2-3, wherein at least two layers of the third set of one or more layers are residual-in-residual dense blocks.

5. The method of claim 1, wherein at least two layers of the first set of one or more layers are residual-in-residual dense blocks.

6. The method of claim 1, wherein the machine learning model is trained to generate the terminal model output as an upsampled version of the model input. The method of claim 1, wherein the machine learning model is trained to generate the terminal model output as a de-blurred version of the model input.

8. The method of claim 1, wherein the machine learning model is trained to generate the terminal model output as a version of the model input that has had compression artifacts removed therefrom.

9. The method of any of claims 1-3 or 5-7, wherein the input image comprises a plurality of pixels at a fourth resolution that is greater than the first resolution, and wherein obtaining the model input comprises combining sets of pixels of the input image to generate respective elements of the model input such that each element of the model input represents a respective block of pixels of the input image.

10. The method of claim 9, wherein combining sets of pixels of the input image to generate respective elements of the model input comprises concatenating representations of sets of the pixels of the input image such that each element of the model input represents a concatenation of the representations of a respective block of pixels of the input image.

11. A computer-implemented method comprising: obtaining a model input that at least partially represents an input image, wherein the model input comprises a two-dimensional array of elements at a first resolution;applying a first set of one or more layers of a machine learning model to the model input to generate a first intermediate model output, wherein the first intermediate model output comprises a two-dimensional array of elements at the first resolution, and wherein at least two layers of the first set of one or more layers are residual-in-residual dense blocks; and partitioning the first intermediate model output to generate a terminal model output that comprises a two-dimensional array of elements at a second resolution, wherein the second resolution is greater than the first resolution, and wherein partitioning the first intermediate model output to generate the terminal model output comprises partitioning each element of the first intermediate model output into a respective block of elements of the terminal model output.

12. The method of claim 11, wherein applying the first set of one or more layers of the machine learning model to the model input to generate the first intermediate model output comprises: applying at least one convolutional layer to the model input to generate a second intermediate model output that comprises a two-dimensional array of elements at the first resolution; applying a second set of layers of the machine learning model to the second intermediate model output to generate a third intermediate model output that comprises a two- dimensional array of elements at the first resolution; applying at least one convolutional layer to the third intermediate model output to generate a fourth intermediate model output that comprises a two-dimensional array of elements at the first resolution; and combining the fourth intermediate model output and the model input to generate the first intermediate model output.

13. The method of claim 11, wherein applying the first set of one or more layers of the machine learning model to the model input to generate the first intermediate model output comprises: applying at least one convolutional layer to the model input to generate a second intermediate model output that comprises a two-dimensional array of elements at the first resolution; applying a second set of layers of the machine learning model to the second intermediate model output to generate a third intermediate model output that comprises a two- dimensional array of elements at the first resolution; applying at least one convolutional layer to the third intermediate model output to generate a fourth intermediate model output that comprises a two-dimensional array of elements at the first resolution; combining the fourth intermediate model output and the model input to generate a fifth intermediate model output that comprises a two-dimensional array of elements at the first resolution; and applying at least one convolutional layer to the fifth intermediate model output to generate the terminal model output.

14. The method of any of claims 11-13, wherein the machine learning model is trained to generate the terminal model output as an upsampled version of the model input.

15. The method of any of claims 11-13, wherein the machine learning model is trained to generate the terminal model output as a de-blurred version of the model input.

16. The method of any of claims 11-13, wherein the machine learning model is trained to generate the terminal model output as a version of the model input that has had compression artifacts removed therefrom.

17. The method of any of claims 11-13, wherein the input image comprises a plurality of pixels at a third resolution that is greater than the first resolution, and wherein obtaining the model input comprises combining sets of pixels of the input image to generate respective elements of the model input such that each element of the model input represents a respective block of pixels of the input image.

18. The method of claim 17, wherein combining sets of pixels of the input image to generate respective elements of the model input comprises concatenating representations of sets of the pixels of the input image such that each element of the model input represents a concatenation of the representations of a respective block of pixels of the input image.

19. A computer-implemented method comprising: obtaining a model input from an input image, wherein the input image comprises a plurality of pixels at a first resolution and the model input comprises a two-dimensional array of elements at a second resolution that is less than the first resolution, wherein obtaining the model input comprises combining sets of pixels of the input image to generate respective elements of the model input such that each element of the model input represents a respective block of pixels of the input image; applying a first set of one or more layers of a machine learning model to the model input to generate a first intermediate model output, wherein the first intermediate model output comprises a two-dimensional array of elements at the second resolution; andpartitioning the first intermediate model output to generate a terminal model output that comprises a two-dimensional array of elements at a third resolution, wherein the third resolution is greater than the second resolution, and wherein partitioning the first intermediate model output to generate the terminal model output comprises partitioning each element of the first intermediate model output into a respective block of elements of the terminal model output.

20. The method of claim 19, wherein applying the first set of one or more layers of the machine learning model to the model input to generate the first intermediate model output comprises: applying at least one convolutional layer to the model input to generate a second intermediate model output that comprises a two-dimensional array of elements at the second resolution; applying a second set of layers of the machine learning model to the second intermediate model output to generate a third intermediate model output that comprises a two- dimensional array of elements at the second resolution; applying at least one convolutional layer to the third intermediate model output to generate a fourth intermediate model output that comprises a two-dimensional array of elements at the second resolution; and combining the fourth intermediate model output and the model input to generate the first intermediate model output.

21. The method of claim 19, wherein applying the first set of one or more layers of the machine learning model to the model input to generate the first intermediate model output comprises:applying at least one convolutional layer to the model input to generate a second intermediate model output that comprises a two-dimensional array of elements at the second resolution; applying a second set of layers of the machine learning model to the second intermediate model output to generate a third intermediate model output that comprises a two- dimensional array of elements at the second resolution; applying at least one convolutional layer to the third intermediate model output to generate a fourth intermediate model output that comprises a two-dimensional array of elements at the second resolution; combining the fourth intermediate model output and the model input to generate a fifth intermediate model output that comprises a two-dimensional array of elements at the second resolution; and applying at least one convolutional layer to the fifth intermediate model output to generate the terminal model output.

22. The method of any of claims 20-21, wherein at least two layers of the first set of one or more layers are residual-in-residual dense blocks 23. The method of any of claims 19-21, wherein the machine learning model is trained to generate the terminal model output as an upsampled version of the model input.

24. The method of any of claims 19-21, wherein the machine learning model is trained to generate the terminal model output as a de-blurred version of the model input.

25. The method of any of claims 19-21, wherein the machine learning model is trained to generate the terminal model output as a version of the model input that has had compression artifacts removed therefrom.

26. The method of any of claims 19-21, wherein combining sets of pixels of the input image to generate respective elements of the model input comprises concatenating representations of sets of the pixels of the input image such that each element of the model input represents a concatenation of the representations of a respective block of pixels of the input image.

27. The method of any of claims 19-21, wherein the first resolution and the third resolution are the same.

28. The method of any of claims 1-3, 5-8, 11-13, or 19-21, wherein the model input represents a first tile of the input image, wherein the terminal model output is a first terminal model output, and wherein the method further comprises: determining, from a second tile of the input image, a second terminal model output; and generating a composite output image by combining the first terminal model output and the second terminal model output.

29. The method of claim 28, further comprising: determining a size of the first tile and the second tile based on a resolution of the input image and a resolution of the composite output image such that generation of the composite output image from the input image takes less than a specified maximum latency.

30. An article of manufacture including a non-transitory computer-readable medium, having stored therein instructions executable by a computing device to cause the computing device to perform the method of any preceding claim.

31. A system comprising: one or more processors; and a non-transitory computer-readable medium, having stored therein instructions executable by the one or more processors to cause the system to perform the method of anyof claims 1-29.

Images