Conditional Axial Transform Layer for High-Fidelity Image Transformation

Abstract

The apparatus and method relate to: receiving an input image comprising a pixel array, wherein the input image is associated with a first feature; applying a neural network to transform the input image into an output image associated with a second feature, wherein an encoder generates coded pixels for each pixel of the pixel array of the input image, provides the coded pixel array to a decoder, wherein the decoder applies axial attention to decode a given pixel, wherein the axial attention includes row attention or column attention applied to one or more previously decoded pixels in a row or column preceding the row or column associated with the given pixel, and wherein the row or column attention blends information within the respective row or column and maintains independence between the different rows or columns; and generating an output image via the neural network.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 086,945, filed October 2, 2020, the entire contents of which are incorporated herein by reference. Background Technology

[0003] Autoregressive models are a family of generative models based on exact likelihood, representing joint representations of data. The neural network models in this family achieve state-of-the-art log-likelihood on high-dimensional image and video datasets. The autoregressive model architecture can read long-term dependencies on large receptive fields and is capable of representing all joint distributions on the data. Self-attention is a building block of the autoregressive model architecture, but it comes at the cost of computational complexity. The memory and computational cost of self-attention grows quadratically with the sequence length N, making it very costly to apply self-attention directly to long sequences. Summary of the Invention

[0004] In one aspect, a computer-implemented method is provided. The method includes: receiving, via a computing device, an input image comprising a pixel array, wherein the input image is associated with a first feature. The method also includes: applying a neural network to transform the input image associated with the first feature into an output image associated with a second feature, wherein applying the neural network includes: generating coded pixels for each pixel of the pixel array of the input image via an encoder of the neural network; providing the coded pixel array to a decoder of the neural network; and having the decoder apply axial attention to decode a given pixel, wherein the axial attention includes row attention or column attention applied to one or more previously decoded pixels in a row or column preceding the row or column associated with the given pixel; and wherein the row attention or column attention blends information within the respective row or column and maintains independence between different rows or columns. The method further includes: generating the output image associated with the second feature via the neural network.

[0005] In another aspect, a computing device is provided. The computing device includes one or more processors and a data memory. The data memory stores computer-executable instructions thereon, which, when executed by the one or more processors, cause the computing device to perform a function. The function includes: receiving an input image comprising a pixel array, wherein the input image is associated with a first feature; applying a neural network to transform the input image associated with the first feature into an output image associated with a second feature, wherein applying the neural network includes: generating coded pixels for each pixel of the pixel array of the input image via an encoder of the neural network; providing the coded pixel array to a decoder of the neural network; and having the decoder apply axial attention to decode a given pixel, wherein the axial attention includes row attention or column attention applied to one or more previously decoded pixels in a row or column preceding the row or column associated with the given pixel; and wherein the row attention or column attention mixes information within the corresponding row or column and maintains independence between different rows or columns; and generating the output image associated with the second feature via the neural network.

[0006] In another aspect, a computer program is provided. The computer program includes instructions that, when executed by a computer, cause the computer to perform functions. The functions include: receiving an input image comprising a pixel array, wherein the input image is associated with a first characteristic; applying a neural network to transform the input image associated with the first characteristic into an output image associated with a second characteristic, wherein applying the neural network includes: generating coded pixels for each pixel of the pixel array of the input image via an encoder of the neural network; providing the coded pixel array to a decoder of the neural network; and applying axial attention by the decoder to decode a given pixel, wherein axial attention includes row attention or column attention applied to one or more previously decoded pixels in a row or column preceding the row or column associated with the given pixel; and wherein the row attention or column attention blends information within the respective row or column and maintains independence between different rows or columns; and generating the output image associated with the second characteristic via the neural network.

[0007] In another aspect, an article of manufacture is provided. The article of manufacture includes one or more computer-readable media having stored computer-readable instructions thereon, which, when executed by one or more processors of a computing device, cause the computing device to perform a function. The function includes: receiving an input image comprising a pixel array, wherein the input image is associated with a first feature; applying a neural network to transform the input image associated with the first feature into an output image associated with a second feature, wherein applying the neural network includes: generating coded pixels for each pixel of the pixel array of the input image via an encoder of the neural network; providing the coded pixel array to a decoder of the neural network; and having the decoder apply axial attention to decode a given pixel, wherein the axial attention includes row attention or column attention applied to one or more previously decoded pixels in a row or column preceding the row or column associated with the given pixel; and wherein the row attention or column attention blends information within the respective row or column and maintains independence between different rows or columns; and generating the output image associated with the second feature via the neural network.

[0008] In another aspect, a system is provided. The system includes: means for receiving an input image comprising a pixel array, wherein the input image is associated with a first characteristic; means for applying a neural network to transform the input image associated with the first characteristic into an output image associated with a second characteristic, wherein applying the neural network includes: generating coded pixels for each pixel of the pixel array of the input image via an encoder of the neural network; providing the coded pixel array to a decoder of the neural network; and having the decoder apply axial attention to decode a given pixel, wherein the axial attention includes row attention or column attention applied to one or more previously decoded pixels in a row or column preceding the row or column associated with the given pixel; and wherein the row attention or column attention mixes information within the respective row or column and maintains independence between respective different rows or columns; and means for generating the output image associated with the second characteristic via the neural network.

[0009] The above summary is illustrative only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent from the accompanying drawings and the following detailed description. Attached Figure Description

[0010] The patent or application documents must contain at least one color drawing. Upon request and payment of the necessary fees, the Patent Office will provide a copy of the patent or patent application publication and the color drawing.

[0011] Figure 1 An exemplary block diagram illustrating a conditional axial self-attention-based neural network according to an exemplary embodiment is shown.

[0012] Figure 2 This is a diagram illustrating an exemplary axial transformer with a multidimensional characteristic vector according to an exemplary embodiment.

[0013] Figure 3 This is a diagram illustrating an exemplary mask operation applied to a multidimensional feature vector according to an exemplary embodiment.

[0014] Figure 4A An exemplary autoregressive shader according to an exemplary embodiment is shown.

[0015] Figure 4B An exemplary color upsampler according to an exemplary embodiment is shown.

[0016] Figure 4C An exemplary spatial upsampler according to an exemplary embodiment is shown.

[0017] Figure 5 An example image of coloring applied to a grayscale image according to an example embodiment is shown.

[0018] Figure 6A and Figure 6B An exemplary graphical representation of hyperparameter selection during training of a neural network is shown, according to an exemplary embodiment.

[0019] Figure 7 An example graphical representation of the predicted probability distribution based on a grayscale image is shown according to an example embodiment.

[0020] Figure 8 An example of a real data image and a predicted coloring of an image, according to an exemplary embodiment, is shown.

[0021] Figure 9 An example of a real data image and a predicted coloring of an image, according to an exemplary embodiment, is shown.

[0022] Figure 10 This is a diagram illustrating the training and inference phases of a machine learning model according to an example embodiment.

[0023] Figure 11 A distributed computing architecture based on an example embodiment is described.

[0024] Figure 12 This is a block diagram of a computing device according to an example embodiment.

[0025] Figure 13 A computing cluster network, arranged as a cloud-based server system according to an example embodiment, is described.

[0026] Figure 14 This is a flowchart of a method according to an example embodiment. Detailed Implementation

[0027] This application relates in one aspect to a neural network based on conditional axial self-attention. In another aspect, this application relates to using machine learning techniques (such as, but not limited to, neural network techniques) to transform an input image associated with a first characteristic into an output image associated with a second characteristic. For example, a grayscale image can be transformed into one or more high-fidelity shaders of a grayscale image. Furthermore, for example, a low-resolution image can be transformed into a high-resolution image. When a user of a mobile computing device takes an image, the resulting image may not always have the desired characteristics, such as shading, resolution, etc. For example, the image may be a grayscale image or a low-resolution image, or it may have color depth and / or intensity that may need to be modified. Furthermore, even if the image does have the desired shading at a certain time, the user may wish to change the shading at a later time. Therefore, image processing-related technical problems arise involving adjusting the shading of an already acquired image.

[0028] Image colorization can be a challenging and inherently stochastic task, potentially based on a semantic understanding of the scene and knowledge of the world. Core immediate applications of the technology include organically recoloring existing image and video content, and bringing raw grayscale media to life, such as old archival images, videos, and black-and-white cartoons. Colorization also has technical uses, serving as a method for learning meaningful representations without explicit supervision, or as an unsupervised data augmentation technique that generates various semantically preserved colorizations of labeled images by training a colorization model on potentially larger sets of unlabeled images.

[0029] Some automatic coloring techniques may be based on neural generative methods, which rely on log-likelihood estimation. Probabilistic models can also be used for one-to-many image coloring tasks. This approach may produce better results than previously applied deterministic methods. Probabilistic models are also better suited to generating a wide variety of colors sampled from learned distributions.

[0030] Colorization methods typically rely on human-in-the-loop approaches that provide cues in the form of doodles, as well as example-based techniques that involve identifying reference source images to reproduce colors. Example-based techniques can be extended to video. In automated neural colorization methods, a range of deterministic approaches may involve variations of convolutional neural networks (CNNs) to model per-pixel color information conditioned on intensity.

[0031] Probabilistic methods typically extend unconditional image generation models, such as variational autoencoders (VAEs), generative adversarial networks (GANs), and normalized flows, to incorporate conditional information from grayscale images. Due to CNN-based upsampling strategies, some autoregressive methods, such as pixel recursive coloring (PixColor) and PIC with PixColor, may offer slightly better results than PIC.

[0032] As described in this paper, an axial attention transformer is presented for self-attention-based autoregressive models organized as high-dimensional tensors of images and other data. Existing autoregressive models may consume large amounts of computational resources for high-dimensional data, or may reduce resource requirements by making quality-based choices regarding distributional expressiveness or ease of implementation. In contrast, the architecture described in this paper maintains full expressiveness over the joint distribution of the data and is easy to implement using standard deep learning frameworks. This can be achieved with reasonable memory and computational resources, and this efficiency is demonstrated by state-of-the-art results on standard generative modeling benchmarks. The model described in this paper is based on axial attention, a type of self-attention that naturally aligns with multiple dimensions of the tensor in both encoding and decoding settings. Notably, the proposed layer structure allows for parallel computation of the vast majority of contexts during decoding without introducing any independent assumptions. This semi-parallel structure makes decoding widely applicable from very large axial attention transformers.

[0033] Neural networks such as the Shading Transformer (ColTran) can be trained in a completely unsupervised manner. ColTran relies on an axis transformer and can utilize two layers to model interactions between pixels, thus requiring minimal depth. Some techniques use different architectures for conditioning, shading, and super-resolution. In contrast, ColTran has a conceptually simpler architecture because self-attention blocks are used for both shading and super-resolution. The multi-level generation process in ColTran can be upsampled in both depth and size based on a fully parallel network. While example embodiments can be described in terms of transforming a grayscale image into a color version of the grayscale image, similar techniques can be applied to transform other image properties, such as image resolution, color depth, and / or color intensity.

[0034] Network Model

[0035] Image coloring is the process of combining the first characteristic The associated input image (e.g., a grayscale image) is transformed to match the second characteristic. The task involves associating output images (e.g., color images). Such tasks may have inherent randomness; for a given image... (For example, grayscale images) exist The conditional distribution is expressed as In some embodiments, instead of from Direct generation Two intermediate low-resolution images with different color depths and They can be predicted sequentially first. In addition to simplifying individual tasks, smaller resolutions can enable the training of larger models.

[0036] Figure 1 An exemplary block diagram illustrating a conditional axial self-attention-based neural network according to an exemplary embodiment is shown. In some embodiments, reasonable and diverse colorings can be generated. An input image can be received. 105 includes a pixel array, which can be represented as rows and columns. In some aspects, the input image 105 can be a low-resolution image or a grayscale image. Although grayscale images are used herein for illustrative purposes, the techniques disclosed herein can generally be applied to augment any digital object that can be represented by a high-dimensional vector array. A neural network can be applied to the input image 105, wherein the application of the neural network involves generating coded pixels for each pixel of the pixel array of the input image 105 by an encoder of the neural network. For example, one or more embeddings 110 of the input image 105 can be generated. In some embodiments, the array of coded pixels can be provided to a decoder of the neural network. In some embodiments, the decoder can apply axial attention to decode a given pixel. Axial attention can include row attention or column attention applied to one or more previously decoded pixels in a row or column preceding the row or column associated with the given pixel. Generally, the term "axial attention" as used herein refers to the mixing of information within a row or column while maintaining independence between different rows or columns.

[0037] For example, autoregressive shader 115 can model the distribution on a low-resolution, coarsely colored image 105 conditioned on the input image 105. The image can be sampled and fed to color upsampler 125, which is a fully parallel self-attention-based decoder for color super-resolution. Furthermore, for example, spatial upsampler 135 can be a fully parallel self-attention-based decoder for spatial super-resolution. In some embodiments, the autoregressive shader (also referred to herein as a ColTran kernel) can be an instance of an axial transformer with a conditional self-attention layer, as described in more detail below. The encoder and upsampler can be stacked, row-wise or column-wise self-attention blocks. In some embodiments, the attention blocks can be residuals and can be trained to perform operations such as conditional self-attention, conditional multilayer perceptron (MLP), and / or conditional layer norm. In some embodiments, the neural network includes two pointwise feedforward layers following the self-attention layer. For example, components of a transformer architecture can be two pointwise feedforward layers following the self-attention layer.

[0038] In some embodiments, the application of axial attention involves predicting one or more of a global per-channel scale and a global per-channel shift. In such embodiments, one or more of the global per-channel scale or global per-channel shift can be learnable parameters. For example, layer normalization can be applied to globally scale and shift a given normalized input using learnable vectors.

[0039] As shown in the figure, a neural network such as ColTran can be a colorization probabilistic model consisting of an autoregressive generating kernel (ColTran kernel) (such as autoregressive shader 115) and two fully parallel upsampling network (ColTran upsampling) networks (such as color upsampling 125 and spatial upsampling 135). In some embodiments, one or more of the three networks may use appropriately modified axial self-attention blocks, where a row-level self-attention layer may be followed by a column-level self-attention layer. To improve grayscale adjustment, ColTran can be configured to perform an adjustment scheme such that each self-attention block can be additionally adjusted by the input image 105 through multiple components. In some embodiments, a colored output image 145 comprising a pixel array of the input image 105 can be generated by the neural network.

[0040] Autoregressive models are a series of generative models based on exact likelihood to represent data. The joint distribution as a condition The product of [the product of the two elements]. In some embodiments, the neural network models in this series can achieve state-of-the-art log-likelihood on high-dimensional image and video datasets. This is likely partly due to architectural innovations that enable each pixel to [achieve a higher level of accuracy]. They all have large, high-bandwidth receptive fields, capable of representing the information from previous pixels. It has long-term dependencies, high computational efficiency, and vectorized computation of log-likelihood and its gradient.

[0041] Autoregressive model architectures capable of reading long-term dependencies over a large receptive field may be able to represent the joint distribution of the data. Meanwhile, architectures that allow for fast log-likelihood gradient computation may be suitable for training on a maximum likelihood target using stochastic gradient methods, which is a stable training process for generative models.

[0042] Self-attention is a building block of autoregressive model architecture. The term "self-attention" used in this paper describes the ability to focus a sequence... Transform into a sequence , ..., The neural network operation, where each is computed through a single vectorizable computation. Depends Generally, self-attention can be effective when learning the long-term dependencies between data dimensions and neural networks that incorporate self-attention into their design.

[0043] Intermediate representation

[0044] In some embodiments, applying axial attention involves applying one or more convolutions to each layer of the decoder to generate multiple affine transformations. Such embodiments also involve applying multiple affine transformations to keys, queries, and values. For example, a self-attention layer can take a sequence of length N of a D-dimensional embedding X as input, which can be represented as an N × D matrix, and generate an output sequence Y, which can also be represented as an N × D matrix. These operations can be performed as follows:

[0045] (Equation 1)

[0046] (Equation 2)

[0047] , and It is a D × D parameter matrix that projects the entries of sequence X onto the query, key, and value, respectively. Each entry of the output sequence Y is a linear combination of values ​​in V weighted by the attention matrix A. In some respects, the attention matrix A can be computed based on the similarity between all pairs of query and key vectors.

[0048] The expressive power and resource cost of self-attention typically stem from the computation of A and Y. For example, it might cost... Time and space are used to compute pairwise similarities between Q and K, and to compute linear combinations of V vectors. Typically, this quadratic complexity can make applying self-attention directly as flat vectors to images and videos computationally intensive. For example, a small 3 images have Size. Sequences like these can be too long for self-attention, so self-attention may be scaled according to these patterns. For example, the length of these sequences can be limited in a modality-aware manner while trying to maintain modeling performance.

[0049] One approach might be to use conditional contexts This approach restricts attention to small subsets of carefully designed data dimensions. While this reduces attentional costs—it's performed on these small subsets rather than the full dataset—the model may fail to represent all joint distributions of the data. Some proposed image models incorporate conditional context. Limiting to small windows of the entire image is possible, but this implementation could require redundant data duplication to extract and process these small windows. Furthermore, video autoregressive models, for example, can be constrained by conditional contexts. Scaling, again, prevents such a model from representing all joint distributions of pixels. The model described in this paper is not context-dependent, and therefore achieves better log-likelihood.

[0050] Another approach might be to stack multiple sparse attention layers, each with a constrained context to improve computational efficiency, but stacking these layers in this way results in a full-context model. However, this architecture typically requires a custom sparse attention graphics processing unit (GPU) kernel to implement specific blocks of sparse variables for matrix multiplication operations. These models are generally not easily implemented on other hardware, such as tensor processing units (TPUs).

[0051] As described in this paper, attention-based autoregressive models can be designed to address the aforementioned challenges. For example, the axial transformer can be configured with a fully regulated scenario, thus its ability to represent joint distributions is unrestricted. The axial transformer also does not require any redundant data duplication or custom kernels for efficient implementation.

[0052] Axial converter

[0053] Typically, the axial transformer described in this paper can be a simple yet effective self-attention-based autoregressive model for data organized as multidimensional tensors. Instead of applying attention to a flattened string of tensor elements, the model described in this paper applies attention along a single axis of the tensor without flattening. This operation is referred to as axial attention in this paper. Since the length of any single axis (i.e., the height or width of the image) is typically much smaller than the total number of elements, the axial attention operation can significantly save computation and memory compared to standard self-attention.

[0054] The described axial converter architecture allows for most conditional scenarios. Embedding offers high parallelism without introducing conditional independence assumptions between any locations and is also suitable for fast sampling processes that are easy to implement. To sample a row of an image, the axial transform operates only on that row, without re-embedding pixels from the previous row. In some embodiments, the axial transform can be configured to define a joint distribution that is fully expressed. Therefore, dependencies on previous pixels are not lost.

[0055] As described in this article, one or more intermediate representations can be generated. For example, this can be achieved by performing region interpolation. Spatial downsampling representation In this description, the down arrow... The up arrow indicates downsampling. This indicates upsampling. Additionally, for example, the letter "c" represents color, "s" represents resolution, and "g" represents ground truth or input. Furthermore, for example, it can generate... 3 bits per channel representation For example, each color channel has 8 intensities. Therefore, each pixel has... A coarse color, which can be directly predicted as a single "color" channel. In some embodiments, The conditional distribution, or conditional likelihood, is expressed as Used to merge intermediate representations, it can be represented as:

[0056]

[0057] (Equation 3)

[0058]

[0059] (Equation 4)

[0060] In some aspects, the autoregressive shader 115 may be a conditional autoregressive axial transformer. In some embodiments, the autoregressive shader 115 may be configured to model the intermediate distribution of each pixel across 512 coarse colors. The condition is that the colors from previously predicted pixels follow the raster order in Equation 5 below. Alternatively, for example, a parallel prediction head can be trained. This can facilitate regularization, as shown in Equation 6 below:

[0061]

[0062] (Equation 5)

[0063]

[0064] (Equation 6)

[0065] where the symbol "<i" represents one or more elements in the rows or columns before the i-th row or column, respectively. In some embodiments, for all pixels in parallel, for upsampling color and for upsampling resolution alternate axial self-attention blocks, respectively. For example, can be used to upsample a rough image provided that the input image is pixel-wise and channel-wise distributed over 256 color intensities, as described in Equation 7 below. Additionally, for example, can spatially upsample to , as described in Equation 8 below:

[0066] (Equation 7)

[0067] (Equation 8)

[0068] Axial attention

[0069] The methods described herein do not change the original shape of the multi-dimensional data tensor and perform masked attention or unmasked attention on a single axis of the tensor at a time. As previously described, such an operation can be referred to as axial attention and is denoted by Attention can be performed on the axis of the tensor (e.g., an input image including a pixel array), mixing information along the axis while keeping the information along the other axes independent. In some embodiments, axial attention is called by transposing all axes other than to the axis of the batch axis and then undoing the transpose. An alternative approach can be to use the "einsum" operation provided in deep learning libraries.

[0070] When the data is an image, can be referred to as column attention because it mixes information in the columns while keeping the unrelated columns independent. Additionally, for example, can be referred to as row attention because it mixes information within the rows while keeping the unrelated rows independent. Axial attention on a square image of size can be of length of Each sequence is executed with attention. Therefore, a total of [number] sequences are executed. This can save on calculations compared to standard self-attention. Generally, for of 2D tensor, axial attention can save order compared to standard attention. This is a computational step. A single layer of axial attention along a certain axis k may not have a complete receptive field because it covers a single axis; however, stacking two axial attention layers allows the model to obtain a global receptive field.

[0071] Mask attention

[0072] As described in this article, mask attention is represented as... It can be Causal mask variants. For example, along the axis of Components of the result It can depend only on the axis of The amount . Figure 2 The receptive fields of these attention patterns (unmasked and masked) are shown in the diagram. Figure 1 Such a mask module is used in the autoregressive model described in the paper.

[0073] Figure 2 This is a diagram illustrating an exemplary axial transformer for a multidimensional feature vector according to an exemplary embodiment. For example, an axial transformer model for a two-dimensional tensor is shown. Before sampling a channel, previous channels and frames can be encoded using 8 blocks of maskless row attention and maskless column attention, as shown in 205. Then, for each row, 4 blocks of maskless row (i.e., full row) attention and masked column (i.e., partial column) attention can be applied to integrate the previously sampled rows for the active channel into the encoded representation, as shown in 210. Finally, as shown in 215, the encoded representation can be shifted up such that the conditioning information satisfies causality, and then, as shown in 220, an internal decoder consisting of 4 blocks of masked row attention can be run to sample new rows in the image.

[0074] Axial attention can be used within standard transform layers to produce axial transform layers. Basic building blocks can share one or more aspects with building blocks in standard transform architectures, such as layer normalization. , and on the last axis of the input The dense layer of the above operation is represented as .letter Indicates the dimension of the output activation. If the input has a shape... This operation is the same as The convolutions are the same, and the output has a shape. In some respects, The ResNet axial attention block operating on the tensor of dimensional embeddings can be defined as follows:

[0075]

[0076] (Equation 9)

[0077]

[0078] (Equation 10)

[0079]

[0080] (Equation 11)

[0081] in, It can be chosen as a constant factor greater than 0. From 1 to 4. In some embodiments, By using Replace the equation in equation 7-9 To define it. Although maskless axial attention is used in some methods, the neural network described in this paper uses autoregressive image modeling based on masked axial attention.

[0082] Figure 3 This is a diagram illustrating an exemplary mask operation applied to a multidimensional feature vector according to an exemplary embodiment. Figure 3 It was shown as Figure 1 The axial attention layer 305 of the building blocks of the axial transformer is of type. A full row 305A is associated with a given active pixel 320, where the receptive field 325 is the entire row of pixels. A mask row 305B is associated with a given active pixel 320, where the receptive field 325 is the partial row of pixels preceding the active pixel 320. A full column 305C is associated with a given active pixel 320, where the receptive field 325 is the entire column of pixels. A mask column 305D is associated with a given active pixel 320, where the receptive field 325 is the partial column of pixels preceding the active pixel 320. The shading position corresponds to the receptive field of the output position, also referred to herein as the given pixel or active pixel 320.

[0083] The arrangement of the inputs to the encoding network of the axis transformer is also shown at frame 310. Previously available or generated channels of image or video can be sequentially stacked in the input. For example, frame 310A represents the previous RGB channels of the current frame. The RGB channels of the frame at time T-1 are represented by frame 310B. The RGB channels of the frame at time T-2 are represented by frame 310C. The RGB channels of the frame at time T-3 are represented by frame 310D. A variable number of fill planes 310E can be used as placeholders for channels generated in the future. The final integer plane 310F can signal the axis transformer to the channels generated in this step. Figure 315 is provided for reference to the corresponding colors of the RGB frames.

[0084] Figure 4A An exemplary autoregressive shader 115 according to an exemplary embodiment is illustrated. In some embodiments, an input image 402 may be received, and one or more embeddings 404 may be generated by an encoder (e.g., a grayscale encoder). In some embodiments, a first encoder 406 may represent layers of column attention 408 and row attention 410. In some embodiments, the neural network may include multiple alternating row-wise and column-wise self-attention layers. For example, the first encoder 406 may include multiple layers, such as N layers. A first softmax operation 430 may be applied to the output from the first encoder 406 to generate a representation as... The intermediate condition distribution is 432.

[0085] In some embodiments, the neural network includes an autoregressive shader comprising an encoder (e.g., a grayscale encoder) configured to apply alternating row and column attention and an axial transformer configured to apply alternating conditional mask column attention and conditional row attention. For example, an intermediate image. 412 can be encoded to generate embedding 414. A second encoder 416 can represent another layer of conditional row attention 418 and conditional mask column attention 420. In some embodiments, the second encoder 416 can receive the output of the first encoder 406 and perform conditional row attention 418 and conditional mask column attention 420. In some embodiments, the second encoder 416 can include multiple layers, such as N layers. A downshift operation 422 can be applied to the outputs of the first encoder 406 and the second encoder 416. This can be followed by a leftshift operation 424, and this process can be recursively repeated on the intermediate image 412.

[0086] In some embodiments, the autoregressive shader further includes one or more layers to apply conditional mask line attention. For example, the outputs of the downshift operation 422 and the leftshift operation 424 can be provided to a third encoder 426. The third encoder 426 can represent a layer of conditional mask line attention 428. Like the first encoder 406 and the second encoder 416, the third encoder 426 can include multiple layers, such as N layers. Another softmax operation 436 can be applied to the output from the third encoder 426 to generate a representation as... Another intermediate condition distribution.

[0087] The term "axial transformer" used in this article generally refers to an axial attention-based autoregressive model used for images and videos. In form In multi-level autoregressive models, the axial attention operation described herein can be used as a building block, following the raster scan order of pixels. In some embodiments, the autoregressive model can be configured on rows, and each row can then be adjusted based on previous rows, and then based on previous channels and frames. Decomposing the model in this way can produce a simple, fast, and partially parallel sampling process.

[0088] Single-channel image model

[0089] For shape single-channel image Each pixel uses The integer value represents its intensity. Pixel intensity can be embedded first. Dimensional Embedded Tensor, which can be represented as A model can be trained to... Transform into a form suitable for classification or sampling. These logit tensors may depend only on the inputs along the raster scan order. The previous pixels in the model are used to ensure that the model defines an effective autoregressive model.

[0090] A. Internal decoder – progressive mode

[0091] Some embodiments involve row-by-row self-attention based on either mask row attention or mask column attention, conditioned on both the encoded and decoded pixel arrays via an internal decoder. For example, a "row-by-row" model can be generated by applying a mask row attention layer:

[0092]

[0093]

[0094] (Equation 12)

[0095] here, It is applied to The number of mask row attention blocks. PositionEmbeddings are the position embeddings that inform the attention layer of the position. Tensors. For parameter efficiency, they can be parameterized using "additive decomposition" positional embeddings. Line embedding and The sum of embedded broadcasts.

[0096] The ShiftRight operation shifts the input one pixel to the right, which in turn shifts the receptive field one pixel to the left. This ensures that the mask row attention layer excludes the current pixel from its receptive field, thus allowing the model architecture to define the correct autoregressive model.

[0097] This model is computationally efficient because it only uses line attention. However, it may not define a full-context model, as each position in the output may not depend on the input pixels in the previous row. If the result is... The logit used for pixel intensity prediction can then generate A collection of independent autoregressive models, such as each row , Instead of a single autoregressive model with full context.

[0098] B. External decoder – captures the above line

[0099] In some embodiments, providing the array involves providing the array to one or more self-attention layers of an external decoder of the neural network. The application of axial attention may include applying row-by-row self-attention to a given pixel by the external decoder, wherein row attention is maskless row attention, column attention is masked column attention, and one or more previously decoded pixels are located in the row preceding the row associated with the given pixel. In some embodiments, the application of axial attention may include applying column-by-column self-attention to a given pixel by the external decoder, wherein row attention is masked row attention, column attention is maskless column attention, and one or more previously decoded pixels are located in the column preceding the column associated with the given pixel.

[0100] Some embodiments involve generating a decoded pixel array conditioned on an encoded pixel array by an external decoder. Such embodiments also involve providing a decoded pixel array to one or more self-attention layers of an internal decoder of a neural network. Applying axial attention may include applying row-by-row self-attention to a given pixel by an internal decoder, where row attention is masked row attention, and one or more previously decoded pixels precede the given pixel in the row associated with it. In some embodiments, applying axial attention may include applying column-by-column self-attention to a given pixel by an internal decoder, where column attention is masked column attention, and one or more previously decoded pixels precede the given pixel in the column associated with it.

[0101] For example, each pixel from the model above Depends on the previous pixels in its own row Therefore, in order to generate an autoregressive model with full context, it is possible to add a model based on previous rows. Additional conditions. In some embodiments, maskless row and mask column layers can be inserted at the beginning of the model described in Equation 12, as follows:

[0102]

[0103]

[0104] (Equation 13)

[0105] tensor This represents the context captured above the current (or given) pixel. It can be computed by unmasked row and masked column attention layers, repeated in total. Layers are used to increase model capacity, which makes It can cover the receptive field of all rows above, including the current pixel. The ShiftDown operation will... Shifting down one pixel shifts its receptive field up one pixel. Therefore, for adjustment purposes, all pixels above can be covered while excluding pixels in the current row, which can then be added to... This serves as the input to the mask row layer. By applying this process, the above row-by-row model can be transformed into a fully expressive autoregressive model that captures not only the pixels in the previous row above the current row, but also the pixels in the current row.

[0106] In some embodiments, finally Layer normalization and a final dense layer can be used to generate layers with... The shape's logit. The logit for each position depends on the positions of all previous pixels in raster scan order.

[0107] C. Semi-parallel sampling

[0108] Simple implementations of sampling from sequence models can be slow because they require re-evaluating the entire network to sample at each location. In the case of the model described in this paper, for size... A square image, each network evaluation may cost Time. Therefore, sampling the entire image may take time. Time, this may be in The time complexity is very large. Therefore, the architecture described in this paper can be configured to perform faster sampling, thereby enabling parallel computation of a larger portion of the model. The sampling algorithm can be based on the following pseudocode:

[0109] 1. For each row :

[0110] (a) Using upper-level computation to construct upper-level scenarios , including about all Information

[0111] (b) For each column :

[0112] i. Sample by The first element of the row As a condition.

[0113] Row-by-row layers are independent of rows because they depend on other rows only through the context of the higher-level layers. Therefore, sampling of a row can be achieved by evaluating only the row-by-row layers of that row, rather than the row-by-row layers of other rows. Thus, in In a row of pixels, you can Each pixel is sampled over time. Therefore, it is possible to... All pixels are sampled during time. Additionally, for example, it is possible to use... Before each row, for the total number of rows in the process , can The time is used to calculate the upper-level context. Therefore, the total computation time can be estimated as follows: Time, this is This is faster than local implementations. This results in a significant improvement in the computation time for image and video sampling, while potentially reducing the computational resources consumed.

[0114] Multi-channel image and video models

[0115] As mentioned above, for shape The architecture of a single-channel image can be configured. Such an architecture can be extended to shapes. Multi-channel images or videos, among which, It can represent the number of channels in a multi-channel image, or the product of the number of channels and the time step in a video. For shape... Modeling this type of data could involve stacking channels on top of each other to form a shape. or Single-channel images. Such a process can be easily implemented, but it may increase the sequence length of column-attention or row-attention, which is problematic for large images. This could be quite large. In some embodiments, a channel can be modeled as a single-channel image at a time, provided that an additional set of maskless row and column attention layers is used for the previous channels. Therefore, the form can be configured... The model in which the previous channel It can be processed as contextual information Tensors can then be added to the first encoding block of a single-channel encoder model. These parameters cannot be shared across these layers.

[0116] Some embodiments involve training a neural network to receive a given input image (e.g., a grayscale image) associated with a first characteristic and applying axial attention to transform the given input image into an output image associated with a second characteristic (e.g., colorization of the grayscale image). During training, training can be performed on random channel slices of each image. For example, previous slices using these maskless attention layers can be used to generate a context tensor, and the probability of randomly selected slices conditioned on that context can be maximized. In some respects, this may be analogous to training an unbiased estimate of the log probability of the entire data tensor. The architecture can be trained to minimize the negative log probability of the data (Equation 7). Note that... , and It can be maximized independently, therefore different models can be referenced. The log probability can be determined as:

[0117] (Equation 14)

[0118] In some embodiments, training a neural network involves training the neural network based on an object detection network trained to detect one or more objects in a given input image.

[0119] In some embodiments, generating an output image involves generating an intermediate output image by applying parallel downsampling to each pixel of the pixel array, wherein the first intermediate output image includes one or more of an intermediate color depth, an intermediate color intensity, an intermediate spatial resolution, or an intermediate color resolution. Such embodiments also involve upsampling each pixel in the intermediate output image in parallel to one or more of a corresponding output color depth, output color intensity, output spatial resolution, or output color resolution.

[0120] Some embodiments involve generating a first intermediate output image including an intermediate color depth by applying parallel downsampling to each pixel of a pixel array. These embodiments also involve generating a second intermediate output image including an intermediate color intensity by applying parallel downsampling to each pixel of the first intermediate output image. Some embodiments involve upsampling each pixel in the second intermediate output image in parallel from the intermediate color intensity to a target color intensity. Such embodiments also involve upsampling each pixel in the upsampled second intermediate output image in parallel from the intermediate color depth to the target color depth.

[0121] Figure 4B An exemplary color upsampler 125 according to an exemplary embodiment is shown. This is for upsampling low-resolution, coarse color images—for example, intermediate images. 212 and a given high-resolution input image, for example, the input image 402 — Generate a high-fidelity color image, which can be trained by color upsampling unit 125 and spatial upsampling unit 135. In some embodiments, these two models can share the same architecture, while differing in their respective input and operating resolutions. Similar to the encoder of autoregressive shader 115, color upsampling unit 125 and spatial upsampling unit 135 include multiple layers of alternating row / column self-attention layers. The encoder output is projected to compute the logit of the per-pixel color probability of the respective upsampling unit.

[0122] Coarse color images, such as intermediate images of 512 colors, 212. This can be converted back to a 3-bit RGB image with 8 symbols per channel. For example, the image generated by the autoregressive shader 115 is represented as... The intermediate condition distribution 232 can be used to generate intermediate images 212 The channel embedding is 240. Channels can be embedded using separate embedding matrices. (For example, channel embedding 240), where, Indicator channels. In some embodiments, each channel can be upsampled by adjusting the embedding of the corresponding channel individually. Channel embedding 240 can be added to the corresponding grayscale and position embedding 242 for each pixel in step 244 and used as input to the subsequent self-attention layer, the fourth encoder 246. For example, the fourth encoder 246 can represent a layer of column attention 248 and row attention 250. In some embodiments, the fourth encoder 246 can include multiple layers, such as N layers. A third softmax 252 can be applied to the output of the fourth encoder 246 to project the output onto a per-pixel channel probability distribution 254, denoted as 256 color intensities, and used for each channel .

[0123] Figure 4C An exemplary spatial upsampler 135 according to an exemplary embodiment is shown. In some embodiments, parallel upsampling can be effectively used for high-quality shading. To avoid slight color inconsistencies between pixels that may stem from parallel sampling, in some embodiments, instead of sampling, the most probable color can be predicted for each pixel. While this may seem to limit the potential diversity of shading to some extent, sampling only coarse colors can often generate a wide variety of shading. Parallel upsampling may have the technical advantage of faster color generation, thereby significantly improving the speed of high-resolution fully autoregressive models.

[0124] In some embodiments, the probability distribution per pixel channel is 254, represented as follows: It can receive from the color upsampler 125 to represent the downsampled intermediate image 256 using region interpolation 258. This is simply upsampled into a blurred, high-resolution RGB image. (See reference...) Figure 1 As described in color upsampling 125, each channel of the blurred RGB image can be embedded. For example, channels can be embedded into a separate embedding matrix. (For example, channel embedding 262), where, Indicating channels. In some embodiments, each channel can be upsampled by adjusting the embedding of the corresponding channel individually. Channel embedding 262 can be added at step 268 to the corresponding grayscale and position embedding 266 for each pixel and used as input to the subsequent self-attention layer, the fifth encoder 270. For example, the fifth encoder 270 can represent a layer of column attention 274 and row attention 272. In some embodiments, the fifth encoder 270 can include multiple layers, such as N layers. A fourth softmax 276 can be applied to the output of the fifth encoder 270 to project the output onto a per-pixel channel probability distribution 278, denoted as 256 color intensities, and for each channel .

[0125] Figure 5 An example image of coloring applied to a grayscale image according to an exemplary embodiment is shown. In each row of images 510, 515, 520, 525, 530, 535, 540, and 545, a first grayscale image and three colors of the grayscale image are shown.

[0126] Figure 6A and Figure 6B An exemplary graphical representation of hyperparameter selection during neural network training is shown according to an exemplary embodiment. Figure 605 shows that at t = 0.01, the model converges to a better FID faster. Figure 610 shows that at higher values ​​of t, the model's performance may deteriorate rapidly.

[0127] Figure 7 An example graphical representation of the predicted probability distribution for a grayscale image according to an example embodiment is shown. The maximum predicted probability per pixel for 512 colors is shown as a proxy for uncertainty. For example, image 710 is the maximum predicted probability per pixel for image 705 across 512 colors. As another example, image 720 is the maximum predicted probability per pixel for image 715 across 512 colors.

[0128] Evaluate

[0129] Axis transformers can be evaluated against image and video modeling benchmarks. Typically, axis transformers achieve state-of-the-art results on ImageNet-32 and ImageNet-64. By stacking videos along the channel dimensions, axis transformers can be directly applied to channel-stacked videos with minimal modification. Axis transformers also outperform previous results without using architectures specifically designed for video. Samples generated on these datasets generally exhibit the expected high quality.

[0130] Axis transformers do not require sub-threads for GPUs or TPUs, which can exhibit unfavorable memory bandwidth and computational tradeoffs. Using efficient operations provided in deep learning frameworks, axis transformers are easy to implement.

[0131] In some respects, the combination of the axis transformer and the LSTM internal decoder performs well on ImageNet32 and ImageNet64 benchmarks, thus also demonstrating the effectiveness of the rest of the axis transformer in capturing context from the preceding rows. Furthermore, for example, the full four-layer internal decoder of the axis transformer provides additional performance gains and significantly faster training. The combination of the axis transformer and the LSTM internal decoder has the advantage of requiring a small number of matrix-vector products to compute layers at each autoregressive step, which is advantageous compared to the approximately 12 matrix-vector products required by the axis transformer itself.

[0132] Example Application

[0133] In one aspect, an input dataset, such as a set of multidimensional feature vectors, can be transformed to output a modified set of multidimensional feature vectors. For example, an input image associated with a first feature can be transformed to an output image associated with a second feature. For example, a grayscale image can be transformed to output high-fidelity color. For example, black and white photographs, low-resolution images, blurred images, and / or images with imperfect and / or undesirable coloring can be converted to output high-fidelity color.

[0134] In some example embodiments, an input image can be transformed to produce an enhanced version of the output image. For example, a grayscale image can be enhanced to a high-resolution black and white image. Furthermore, for example, a color image can be enhanced to a high-resolution color image.

[0135] On the other hand, grayscale videos can be converted to output high-fidelity colorized videos. For example, old black and white videos can be converted to obtain realistic colorization. Furthermore, for example, the image resolution of a video can be increased. As another example, the color depth or color intensity of a video can be enhanced.

[0136] In another scenario, the conversion from a first image to a target (e.g., high-resolution, high-fidelity, color) image can be performed by a mobile device, such as a mobile phone device. For example, a user can access an image captured by the camera of a mobile phone device and indicate their desired modifications to the accessed image. The mobile phone device can then convert the captured image to output the desired image. In some implementations, one or more output images can be provided to the user, and the user can select from one or more output images.

[0137] In some example embodiments, a user can access images from an image library. In other example embodiments, a user can receive images via a mobile phone device, for example, for another user, or download images from an image server.

[0138] For example, a user can retrieve a black-and-white video from a video server to a mobile phone device, and then the user can specify their desire to colorize the retrieved video. Therefore, the mobile phone device can convert the retrieved video to output a high-fidelity color video.

[0139] In some example implementations, neural network architectures can be used for image editing.

[0140] In some example embodiments, neural networks can be used as a preprocessing step for another machine learning model.

[0141] In some example embodiments, neural network architectures can be used to transform and / or augment any dataset represented as multidimensional feature vectors.

[0142] These and other exemplary applications are considered within the scope of this disclosure.

[0143] Figure 8 Examples of real data images and predicted coloring of images according to exemplary embodiments are shown. In each column, the real data image is described by two samples. For example, the image in column 810 corresponds to the real data image. The images in columns 820 and 830 illustrate various actual colorings. The images illustrate how the realism improves from the first sample in column 820 to the second sample in column 830.

[0144] Figure 9 Examples of real data images and predicted coloring of images according to exemplary embodiments are shown. In each image set, the bottom row 910 corresponds to the real data, and the top row 920 corresponds to the predicted coloring.

[0145] Training machine learning models to generate inference / prediction

[0146] Figure 10 Figure 1000 illustrates the training phase 1002 and inference phase 1004 of a trained machine learning model (or more) 1032 according to an example embodiment. Some machine learning techniques involve training one or more machine learning algorithms on an input set of training data to identify patterns in the training data and provide output inferences and / or predictions about the patterns in the training data. The resulting trained machine learning algorithm can be referred to as a trained machine learning model. For example, Figure 10 The training phase 1002 is shown, in which one or more machine learning algorithms 1020 are trained on training data 1010 to become a trained machine learning model 1032. Then, during the inference phase 1004, the trained machine learning model 1032 may receive input data 1030 and one or more inference / prediction requests 1040 (possibly as part of the input data 1030), and respond by providing one or more inferences and / or predictions 1050 as outputs.

[0147] Therefore, the trained machine learning model(s) 1032 may include one or more models of one or more machine learning algorithms(s) 1020. The machine learning algorithms(s) 1020 may include, but are not limited to, artificial neural networks (e.g., convolutional neural networks, recurrent neural networks, Bayesian networks, hidden Markov models, Markov decision processes, logistic regression functions, support vector machines, suitable statistical machine learning algorithms, and / or heuristic machine learning systems described herein). The machine learning algorithms(s) 1020 may be supervised or unsupervised and may implement any suitable combination of online and offline learning.

[0148] In some examples, machine learning algorithms (or more) 1020 and / or trained machine learning models (or more) 1032 can be accelerated using on-device coprocessors, such as graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), and / or application-specific integrated circuits (ASICs). Such on-device coprocessors can be used to accelerate machine learning algorithms (or more) 1020 and / or trained machine learning models (or more) 1032. In some examples, trained machine learning models (or more) 1032 can be trained, resided, and executed to provide inference on a specific computing device, and / or otherwise enable inference on a specific computing device.

[0149] During training phase 1002, machine learning algorithms (or more) 1020 may be trained by providing at least training data 1010 as training input using unsupervised, supervised, semi-supervised, and / or reinforcement learning techniques. Unsupervised learning involves providing a portion (or all) of the training data 1010 to the machine learning algorithms (or more) 1020, and the machine learning algorithms (or more) 1020 determining one or more output inferences based on the provided portion (or all) of the training data 1010. Supervised learning involves providing a portion of the training data 1010 to the machine learning algorithms (or more) 1020, the machine learning algorithms (or more) 1020 determining one or more output inferences based on the provided portion of the training data 1010, and the output inferences (or more) being accepted or corrected based on correct results associated with the training data 1010. In some examples, the supervised learning of the machine learning algorithm (or more) 1020 can be supervised by a set of rules and / or a set of labels used for training inputs, and the set of rules and / or the set of labels can be used to correct the inference of the machine learning algorithm (or more) 1020.

[0150] Semi-supervised learning involves obtaining correct results on a portion (but not all) of the training data 1010. During semi-supervised learning, supervised learning is used on the portion of the training data 1010 that yields correct results, while unsupervised learning is used on the portion of the training data 1010 that does not yield correct results. Reinforcement learning involves machine learning algorithms (or more) 1020 receiving a reward signal about a previously inferred value, where the reward signal can be a numerical value. During reinforcement learning, machine learning algorithms (or more) 1020 can output an inference and receive a reward signal in response, where the machine learning algorithms (or more) 1020 are configured to attempt to maximize the numerical value of the reward signal. In some examples, reinforcement learning also utilizes a numerical function that provides a numerical value representing the expected sum of the values ​​provided by the reward signal over time. In some examples, the machine learning algorithms (or more) 1020 and / or the trained machine learning models (or more) 1032 can be trained using other machine learning techniques, including but not limited to incremental learning and curriculum learning.

[0151] In some examples, machine learning algorithms (or more) 1020 and / or trained machine learning models (or more) 1032 may use transfer learning techniques. For example, transfer learning techniques may involve pre-training the trained machine learning models (or more) 1032 on a dataset and then further training them using training data 1010. More specifically, machine learning algorithms (or more) 1020 may be pre-trained on data from one or more computing devices and the results provided to computing device CD1 to obtain the trained machine learning model, where CD1 intends to execute the trained machine learning model during inference phase 1004. Then, during training phase 1002, the pre-trained machine learning model may be further trained using training data 1010, which may be derived from both kernel and non-kernel data of computing device CD1. Supervised or unsupervised learning may be used to perform further training of the machine learning algorithms (or more) 1020 and / or the pre-trained machine learning model using training data 1010 from CD1 data. Once the machine learning algorithm(s) 1020 and / or the pre-trained machine learning model have been trained on at least the training data 1010, the training phase 1002 can be completed. The machine learning model obtained from the training result can be used as at least one of the trained machine learning models(s) 1032.

[0152] Specifically, once the training phase 1002 has been completed, the trained machine learning model (or more) 1032 can be provided to the computing device (if it is not already on the computing device). The inference phase 1004 can begin after the trained machine learning model (or more) 1032 has been provided to the computing device CD1.

[0153] During the inference phase 1004, the trained machine learning model (or more) 1032 may receive input data 1030 and generate and output one or more corresponding inferences and / or predictions 1050 about the input data 1030. Therefore, the input data 1030 can be used as input to the trained machine learning model (or more) 1032 to provide corresponding inferences and / or predictions 1050 to kernel components and non-kernel components. For example, the trained machine learning model (or more) 1032 may generate inferences and / or predictions 1050 in response to one or more inference / prediction requests 1040. In some examples, the trained machine learning model (or more) 1032 may be executed by a portion of other software. For example, the trained machine learning model (or more) 1032 may be executed by an inference or prediction daemon to provide inferences and / or predictions upon request. The input data 1030 may include data from the computing device CD1 executing the trained machine learning model (or more) 1032 and / or input data from one or more computing devices other than CD1.

[0154] Input data 1030 may include a collection of images provided by one or more sources. The collection of images may include video frames, images residing on the computing device CD1, and / or other images. Other types of input data are also possible.

[0155] The inferences (or multiple) and / or predictions (or multiple) 1050 may include output images, output intermediate images, numerical values, and / or other output data generated by the trained machine learning models (or multiple) 1032 operating on the input data 1030 (and training data 1010). In some examples, the trained machine learning models (or multiple) 1032 may use the output inferences (or multiple) and / or predictions (or multiple) 1050 as input feedback 1060. The trained machine learning models (or multiple) 1032 may also rely on past inferences as input to generate new inferences.

[0156] A conditional axial self-attention-based neural network can be an example of machine learning algorithms (or more) 1020. After training, a trained version of the neural network can be an example of a trained machine learning model (or more) 1032. In this approach, an example of inference / prediction requests (or more) 1040 can be a request to predict the colorization of a grayscale image, and a corresponding example of inference and / or prediction (or more) 1050 can be an output image including the colorization of the grayscale image. Another example of inference / prediction requests (or more) 1040 can be a request to predict spatial magnification of an image, and a corresponding example of inference and / or prediction (or more) 1050 can be an output image including the spatially magnified input image.

[0157] In some examples, a computing device CD_SOLO may include a trained version of a conditional axial self-attention-based neural network, possibly after training. The computing device CD_SOLO can then receive a request to transform an input image associated with a first feature into an output image associated with a second feature, and generate the output image associated with the second feature using the trained version of the neural network.

[0158] In some examples, two or more computing devices, CD_CLI and CD_SRV, can be used to provide the output image; for example, the first computing device, CD_CLI, can generate and send a request to transform a grayscale image into a second computing device, CD_SRV. CD_SRV can then use a trained version of a neural network to generate a colored output image that transforms the grayscale image into a grayscale image, and respond to CD_CLI's request for the output image. Upon receiving a response to the request, CD_CLI can then provide the requested output image (e.g., using a user interface and / or a display, a printed copy, electronic communication, etc.).

[0159] Example Data Network

[0160] Figure 11 A distributed computing architecture 1100 according to an example embodiment is depicted. The distributed computing architecture 1100 includes server devices 1108 and 1110, configured to communicate with programmable devices 1104a, 1104b, 1104c, 1104d, and 1104e via a network 1106. The network 1106 may correspond to a local area network (LAN), a wide area network (WAN), a WLAN, a WWAN, a corporate intranet, the public internet, or any other type of network configured to provide communication paths between networked computing devices. The network 1106 may also correspond to a combination of one or more LANs, WANs, corporate intranets, and / or the public internet.

[0161] although Figure 11Only five programmable devices are shown, but the distributed application architecture can serve dozens, hundreds, or thousands of programmable devices. Furthermore, programmable devices 1104a, 1104b, 1104c, 1104d, 1104e (or any other programmable device) can be any type of computing device, such as mobile computing devices, desktop computers, wearable computing devices, head-mounted displays (HMDs), network terminals, etc. In some examples, such as programmable devices 1104a, 1104b, 1104c, and 1104e, the programmable devices can be directly connected to network 1106. In other examples, such as programmable device 1104d, the programmable device can be indirectly connected to network 1106 through an associated computing device, such as programmable device 1104c. In this example, programmable device 1104c can act as an associated computing device to transmit electronic communication between programmable device 1104d and network 1106. In other examples, such as programmable device 1104e, the computing device may be part of and / or inside a vehicle, such as a car, truck, bus, boat or ship, airplane, etc. Figure 11 In other examples not shown, the programmable device may be directly or indirectly connected to network 1106.

[0162] Server devices 1108 and 1110 can be configured to perform one or more services upon request from programmable devices 1104a-1104e. For example, server devices 1108 and / or 1110 can provide content to programmable devices 1104a-1104e. Content may include, but is not limited to, web pages, hypertext, scripts, binary data such as compiled software, images, audio, and / or video. Content may include compressed and / or uncompressed content. Content may be encrypted and / or unencrypted. Other types of content are also possible.

[0163] As another example, server devices 1108 and / or 1110 can provide programmable devices 1104a-1104e with access to software for database, search, computing, graphics, audio, video, World Wide Web / Internet utilization and / or other functions. Many other examples of server devices are also possible.

[0164] Computing device architecture

[0165] Figure 12 This is a block diagram of an exemplary computing device 1200 according to an exemplary embodiment. Specifically, Figure 12 The computing device 1200 shown can be configured to perform at least one function and / or related functions of a conditional axial self-attention-based neural network and / or method 1400.

[0166] The computing device 1200 may include a user interface module 1201, a network communication module 1202, one or more processors 1203, a data storage device 1204, one or more cameras 1218, one or more sensors 1220, and a power system 1222, all of which may be linked together via a system bus, network, or other connection mechanism 1205.

[0167] User interface module 1201 can be used to send data to and / or receive data from external user input / output devices. For example, user interface module 1201 can be configured to send and / or receive data from user input devices such as touchscreens, computer mice, keyboards, keypads, touchpads, trackballs, joysticks, voice recognition modules, and / or other similar devices. User interface module 1201 can also be configured to provide output to user display devices such as one or more cathode ray tubes (CRTs), liquid crystal displays, light-emitting diodes (LEDs), display technologies using digital light processing (DLP), printers, light bulbs, and / or other similar devices, whether now known or later developed. User interface module 1201 can also be configured to generate audio output using devices such as speakers, speaker jacks, audio output ports, audio output devices, headphones, and / or other similar devices. User interface module 1201 can be further configured with one or more tactile devices that can generate tactile output, such as vibration and / or other outputs detectable by touching computing device 1200 and / or physical contact with computing device 1200. In some examples, the user interface module 1201 may be used to provide a graphical user interface (GUI) for utilizing the computing device 1200, such as a mobile phone device.

[0168] The network communication module 1202 may include one or more devices providing one or more wireless interfaces 1207 and / or one or more wired interfaces 1208, configurable for communication over a network. The wireless interfaces (or multiple interfaces) 1207 may include one or more wireless transmitters, receivers, and / or transceivers, such as Bluetooth™ transceivers, Zigbee® transceivers, Wi-Fi™ transceivers, WiMAX™ transceivers, LTE™ transceivers, and / or other types of wireless transceivers configurable for communication over a wireless network. The wired interfaces (or multiple interfaces) 1208 may include one or more wired transmitters, receivers, and / or transceivers, such as Ethernet transceivers, Universal Serial Bus (USB) transceivers, or similar transceivers, configurable for communication over twisted-pair, coaxial, fiber optic links, or similar physical connections to a wired network.

[0169] In some examples, network communication module 1202 can be configured to provide reliable, secure, and / or authenticated communication. For each communication described herein, information may be provided to facilitate reliable communication (e.g., guaranteed message delivery), possibly as part of message headers and / or footers (e.g., packet / message sequence information, encapsulated headers and / or footers, size / time information, and transmission verification information such as Cyclic Redundancy Check (CRC) and / or parity values). Communication may be securely (e.g., encoded or encrypted) and / or decrypted / decoded using one or more encryption protocols and / or algorithms, such as, but not limited to, Data Encryption Standard (DES), Advanced Encryption Standard (AES), Rivest-Shamir-Adelman (RSA) algorithm, Diffie-Hellman algorithm, Secure Sockets Protocols such as Secure Sockets Layer (SSL) or Transport Layer Security (TLS), and / or Digital Signature Algorithm (DSA). In addition to those protocols and / or algorithms listed herein, other cryptographic protocols and / or algorithms may be used to secure (and then decrypt / decode) communication.

[0170] One or more processors 1203 may include one or more general-purpose processors and / or one or more special-purpose processors (e.g., digital signal processors, tensor processing units (TPUs), graphics processing units (GPUs), application-specific integrated circuits, etc.). One or more processors 1203 may be configured to execute computer-readable instructions 1206 and / or other instructions as described herein, contained in data memory 1204.

[0171] Data storage 1204 may include one or more non-transitory computer-readable storage media that can be read and / or accessed by at least one of one or more processors 1203. The one or more computer-readable storage media may include volatile and / or non-volatile storage components, such as optical, magnetic, organic, or other memory or disk storage, which may be wholly or partially integrated with at least one of the one or more processors 1203. In some examples, data storage 1204 may be implemented using a single physical device (e.g., a single optical, magnetic, organic, or other memory or disk storage unit), while in other examples, data storage 1204 may be implemented using two or more physical devices.

[0172] Data memory 1204 may include computer-readable instructions 1206 and possible additional data. In some examples, data memory 1204 may include storage required to perform at least a portion of the methods, scenarios, and techniques described herein and / or at least a portion of the functionality of the devices and networks described herein. In some examples, data memory 1204 may include memory for training a neural network model 1212 (e.g., a model for training a neural network, such as a conditional axial self-attention-based neural network). Specifically, in these examples, computer-readable instructions 1206 may include instructions that, when executed by processor 1203, enable computing device 1200 to provide some or all of the functionality for training neural network model 1212.

[0173] In some examples, computing device 1200 may include one or more cameras 1218. Camera 1218 may include one or more image capturing devices, such as still and / or video cameras, equipped to capture light and record the captured light in one or more images; that is, camera 1218 can generate images of the captured light. The one or more images may be one or more still images and / or video images used in one or more images. Camera 1218 may capture light and / or electromagnetic radiation emitted as visible light, infrared radiation, ultraviolet light, and / or as one or more other light frequencies.

[0174] In some examples, computing device 1200 may include one or more sensors 1220. Sensors 1220 may be configured to measure conditions within computing device 1200 and / or conditions in the environment of computing device 1200, and provide data about these conditions. For example, sensors 1220 may include one or more of the following: (i) sensors for obtaining data about computing device 1200, such as, but not limited to, thermometers for measuring the temperature of computing device 1200, battery sensors for measuring the power of one or more batteries in power system 1222, and / or other sensors for measuring the conditions of computing device 1200; (ii) identification sensors for identifying other objects and / or devices, such as, but not limited to, radio frequency identification (RFID) readers, proximity sensors, one-dimensional barcode readers, two-dimensional barcode (e.g., quick response (QR) code) readers, and laser trackers, wherein the identification sensors may be configured to read identifiers such as RFID tags, barcodes, QR codes, and / or other devices and / or objects configured to read and at least provide identification information; (iii) sensors for measuring computing device... The computing device 1200 includes position and / or motion sensors, such as, but not limited to, tilt sensors, gyroscopes, accelerometers, Doppler sensors, GPS devices, sonar sensors, radar devices, laser displacement sensors, and compasses; (iv) environmental sensors for obtaining data indicating the environment of the computing device 1200, such as, but not limited to, infrared sensors, optical sensors, light sensors, biosensors, capacitive sensors, touch sensors, temperature sensors, wireless sensors, radio sensors, motion sensors, microphones, sound sensors, ultrasonic sensors, and / or smoke sensors; and / or (v) force sensors for measuring one or more forces (e.g., inertial forces and / or gravity) acting on the computing device 1200, such as, but not limited to, one or more sensors for measuring: one-dimensional or multi-dimensional forces, torque, ground forces, frictional forces, and / or zero-torque point (ZMP) sensors for identifying the ZMP and / or the ZMP's location. Many other examples of sensors 1220 are also possible.

[0175] Power system 1222 may include one or more batteries 1224 and / or one or more external power interfaces 1226 for providing power to computing device 1200. When electrically coupled to computing device 1200, each of the one or more batteries 1224 can serve as a source of stored power for computing device 1200. The one or more batteries 1224 of power system 1222 may be configured to be portable. Some or all of the one or more batteries 1224 may be easily removed from computing device 1200. In other examples, some or all of the one or more batteries 1224 may be located inside computing device 1200 and therefore may not be easily removed from computing device 1200. Some or all of the one or more batteries 1224 may be rechargeable. For example, a rechargeable battery may be rechargeable via a wired connection between the battery and another power source, such as one or more power sources external to computing device 1200 and connected to computing device 1200 via one or more external power interfaces. In other examples, some or all of the one or more batteries 1224 may be non-rechargeable batteries.

[0176] One or more external power interfaces 1226 of the power system 1222 may include one or more wired power interfaces, such as USB cables and / or power cords, enabling wired power connection to one or more power sources external to the computing device 1200. One or more external power interfaces 1226 may include one or more wireless power interfaces, such as Qi wireless chargers, enabling wireless power connections (e.g., via Qi wireless chargers) to one or more external power sources. Once a power connection to an external power source is established using one or more external power interfaces 1226, the computing device 1200 can draw power from the external power source to maintain the established power connection. In some examples, the power system 1222 may include associated sensors, such as battery sensors associated with one or more batteries or other types of electrical power sensors.

[0177] Cloud-based servers

[0178] Figure 13 A network 1106 is depicted as a computing cluster 1309a, 1309b, 1309c arranged as a cloud-based server system according to an example embodiment. The computing clusters 1309a, 1309b, 1309c may be cloud-based devices that store program logic and / or data for cloud-based applications and / or services; for example, executing at least one function and / or related functions of a neural network, a conditional axial self-attention-based neural network, and / or method 2400.

[0179] In some embodiments, computing clusters 1309a, 1309b, and 1309c may be a single computing device residing in a single computing center. In other embodiments, computing clusters 1309a, 1309b, and 1309c may include multiple computing devices in a single computing center, or even multiple computing devices in multiple computing centers located in different geographical locations. For example, Figure 13 Each of the computing clusters 1309a, 1309b, and 1309c residing in different physical locations is depicted.

[0180] In some embodiments, the data and services of computing clusters 1309a, 1309b, and 1309c may be encoded as computer-readable information stored in a non-transitory tangible computer-readable medium (or computer-readable storage medium) and accessible by other computing devices. In some embodiments, computing clusters 1309a, 1309b, and 1309c may be stored on a single disk drive or other tangible storage medium, or may be implemented on multiple disk drives or other tangible storage media located in one or more different geographical locations.

[0181] Figure 13 A cloud-based server system according to an example embodiment is described. Figure 13 In this configuration, the functionality of conditionally axially self-attention-based neural networks and / or computing devices can be distributed among computing clusters 1309a, 1309b, and 1309c. Computing cluster 1309a may include one or more computing devices 1300a, cluster storage array 1310a, and cluster router 1311a connected by a local cluster network 1312a. Similarly, computing cluster 1309b may include one or more computing devices 1300b, cluster storage array 1310b, and cluster router 1311b connected by a local cluster network 1312b. Similarly, computing cluster 1309c may include one or more computing devices 1300c, cluster storage array 1310c, and cluster router 1311c connected by a local cluster network 1312c.

[0182] In some embodiments, each of computing clusters 1309a, 1309b, and 1309c may have an equal number of computing devices, an equal number of cluster storage arrays, and an equal number of cluster routers. However, in other embodiments, each computing cluster may have a different number of computing devices, a different number of cluster storage arrays, and a different number of cluster routers. The number of computing devices, cluster storage arrays, and cluster routers in each computing cluster depends on one or more computing tasks assigned to each computing cluster.

[0183] For example, in computing cluster 1309a, computing device 1300a can be configured to perform various computational tasks of conditional axial self-attention-based neural networks and / or computing devices. In one embodiment, the various functions of the neural network and / or computing device can be distributed among one or more of computing devices 1300a, 1300b, and 1300c. Computing devices 1300b and 1300c in the corresponding computing clusters 1309b and 1309c can be configured similarly to computing device 1300a in computing cluster 1309a. On the other hand, in some embodiments, computing devices 1300a, 1300b, and 1300c can be configured to perform different functions.

[0184] In some embodiments, the computational tasks associated with the neural network and / or the computing devices and the stored data may be distributed among the computing devices 1300a, 1300b, and 1300c based at least in part on the processing requirements of the neural network and / or the computing devices, the processing capabilities of the computing devices 1300a, 1300b, and 1300c, the latency of network links between computing devices in each computing cluster and between the computing clusters themselves, and / or other factors that may affect cost, speed, fault tolerance, resilience, efficiency, and / or other design goals of the overall system architecture.

[0185] The clustered storage arrays 1310a, 1310b, and 1310c of computing clusters 1309a, 1309b, and 1309c can be data storage arrays, including disk array controllers configured to manage read and write access to hard disk drive groups. The disk array controllers, used alone or in conjunction with their respective computing devices, can also be configured to manage backups or redundant copies of data stored in the clustered storage arrays to prevent one or more computing devices from accessing one or more clustered storage arrays due to disk drive or other clustered storage array failures and / or network failures.

[0186] Similar to how the functionality of conditional axially self-attention-based neural networks and / or computing devices can be distributed across computing devices 1300a, 1300b, and 1300c in computing clusters 1309a, 1309b, and 1309c, various active and / or backup portions of these components can be distributed across clustered storage arrays 1310a, 1310b, and 1310c. For example, some clustered storage arrays can be configured to store a portion of the data for the first layer of the neural network and / or computing device, while other clustered storage arrays can store other portions of the data for the second layer of the neural network and / or computing device. Furthermore, for example, some clustered storage arrays can be configured to store data for the encoder of the neural network, while other clustered storage arrays can store data for the decoder of the neural network. Additionally, some clustered storage arrays can be configured to store backup versions of the data stored in other clustered storage arrays.

[0187] Cluster routers 1311a, 1311b, and 1311c in computing clusters 1309a, 1309b, and 1309c may include network devices configured to provide internal and external communication for the computing clusters. For example, cluster router 1311a in computing cluster 1309a may include one or more Internet switching and routing devices configured to provide (i) local area network communication between computing device 1300a and cluster storage array 1310a, and (ii) wide area network communication between computing cluster 1309a and computing clusters 1309b and 1309c via wide area network link 1313a to network 1106 via local cluster network 1312a. Cluster routers 1311b and 1311c may include network devices similar to those in cluster router 1311a, and cluster routers 1311b and 1311c may perform similar networking functions for computing cluster 1309b and 1309b as those performed by cluster router 1311a for computing cluster 1309a.

[0188] In some embodiments, the configuration of cluster routers 1311a, 1311b, and 1311c may be based at least in part on the data communication requirements of computing devices and cluster storage arrays, the data communication capabilities of network devices in cluster routers 1311a, 1311b, and 1311c, the latency and throughput of local cluster networks 1312a, 1312b, and 1312c, the latency, throughput, and cost of WAN links 1313a, 1313b, and 1313c, and / or other factors that may lead to adjustments in the cost, speed, fault tolerance, resilience, efficiency, and / or other design criteria of the system architecture.

[0189] Example operation method

[0190] Figure 14This is a flowchart of method 1400 according to an example embodiment. Method 1400 can be performed by a computing device such as computing device 1200. Method 1400 can begin at block 1410, wherein the method involves receiving an input image comprising a pixel array, wherein the input image is associated with a first feature.

[0191] In box 1420, the method relates to applying a neural network to transform an input image associated with a first feature into an output image associated with a second feature, wherein applying the neural network involves generating an encoded pixel for each pixel of the pixel array of the input image via an encoder of the neural network, providing the encoded pixel array to a decoder of the neural network, and applying axial attention via the decoder to decode a given pixel, wherein the axial attention includes row attention or column attention applied to one or more previously decoded pixels in a row or column preceding the row or column associated with the given pixel, and wherein the row attention or column attention blends information in the respective row or column and maintains independence between the different rows or columns.

[0192] In box 1430, the method involves generating an output image associated with the second characteristic via a neural network.

[0193] In some embodiments, the neural network includes multiple alternating row-by-row self-attention layers and column-by-column self-attention layers.

[0194] In some embodiments, providing the array involves providing the array to one or more self-attention layers of an external decoder of a neural network, and wherein applying axial attention includes applying one of the following via the external decoder: applying row-wise self-attention to a given pixel, wherein row attention is maskless row attention, wherein column attention is masked column attention, and wherein one or more previously decoded pixels are in the row preceding the row associated with the given pixel; or applying column-wise self-attention to a given pixel, wherein row attention is masked row attention, wherein column attention is maskless column attention, and wherein one or more previously decoded pixels are in the column preceding the column associated with the given pixel.

[0195] Some embodiments involve generating a decoded pixel array by an external decoder conditioned on an encoded pixel array. These embodiments also involve providing a decoded pixel array to one or more self-attention layers of an internal decoder of a neural network, wherein applying axial attention involves applying one of the following via the internal decoder: applying row-wise self-attention to a given pixel, wherein row attention is masked row attention, and wherein one or more previously decoded pixels precede the given pixel in the row associated with the given pixel; or applying column-wise self-attention to a given pixel, wherein column attention is masked column attention, and wherein one or more previously decoded pixels precede the given pixel in the column associated with the given pixel.

[0196] Some embodiments involve row-by-row self-attention conditioned on an array of encoded and decoded pixels, using an internal decoder and based on either mask row attention or mask column attention.

[0197] In some embodiments, the input image associated with the first feature may be an image with a first resolution, and the output image associated with the second feature may be an image with a second resolution, wherein the second resolution is greater than the first resolution.

[0198] In some embodiments, the input image associated with the first characteristic may be a grayscale image, and the output image associated with the second characteristic may be a colored version of the grayscale image. In some embodiments, the coloring of the grayscale image involves high-fidelity coloring of the grayscale image.

[0199] In some embodiments, the input image includes multiple video frames.

[0200] In some embodiments, generating an output image involves generating an intermediate output image by applying parallel downsampling to each pixel of the pixel array, wherein the first intermediate output image includes one or more of an intermediate color depth, an intermediate color intensity, an intermediate spatial resolution, or an intermediate color resolution. Such embodiments also involve upsampling each pixel in the intermediate output image in parallel to a corresponding one or more of an output color depth, an output color intensity, an output spatial resolution, or an output color resolution.

[0201] Some embodiments involve generating a first intermediate output image that includes intermediate color depth by applying parallel downsampling to each pixel of the pixel array. Such embodiments also involve generating a second intermediate output image that includes intermediate color intensity by applying parallel downsampling to each pixel of the first intermediate output image.

[0202] Some embodiments involve upsampling each pixel in the second intermediate output image in parallel from an intermediate color intensity to a target color intensity. Such embodiments also involve upsampling each pixel in the upsampled second intermediate output image in parallel from an intermediate color depth to a target color depth.

[0203] Some embodiments involve providing a neural network as a preprocessing network for another machine learning model.

[0204] In some embodiments, applying axial attention involves applying one or more convolutions to each layer of the decoder to generate multiple affine transformations. Such embodiments also involve applying multiple affine transformations to keys, queries, and values.

[0205] In some embodiments, the neural network includes two pointwise feedforward layers following a self-attention layer.

[0206] In some embodiments, applying axial attention involves predicting one or more of the global per-channel scale and the global per-channel shift. In such embodiments, one or more of the global per-channel scale or the global per-channel shift can be learnable parameters.

[0207] Some embodiments involve training a neural network to receive a given input image associated with a first characteristic and applying axial attention to transform the given input image into an output image associated with a second characteristic. In some embodiments, training the neural network involves training the neural network based on an object detection network trained to detect one or more objects in a given grayscale image.

[0208] In some embodiments, the neural network includes an autoregressive shader comprising an encoder configured to apply alternating row attention and column attention, and an axial transformer configured to apply alternating conditional mask column attention and conditional row attention. In some embodiments, the autoregressive shader further includes one or more layers to apply conditional mask row attention.

[0209] Some embodiments involve a computing device determining a request to transform an input image associated with a first feature into an output image associated with a second feature. Such embodiments also involve sending a request from the computing device to a second computing device, the second computing device comprising a trained version of a neural network. Such embodiments further involve the computing device receiving an output image from the second computing device after the request has been sent.

[0210] In some embodiments, the computing device includes a camera, and the method involves accessing an input image using the camera and receiving an output image from the camera at the computing device.

[0211] Some embodiments involve using computing devices to provide output images.

[0212] In some embodiments, generating an output image involves obtaining a trained version of a neural network on a computing device. The computing device can then use the obtained neural network to perform applications of the neural network and generate the output image. In some embodiments, the neural network can be trained at the computing device.

[0213] This invention is not limited to the specific embodiments described herein, which are intended as illustrations of various aspects. As will be apparent to those skilled in the art, many modifications and variations can be made without departing from its spirit and scope. From the foregoing description, it will be clear to those skilled in the art that functionally equivalent methods and apparatuses exist within the scope of this invention, in addition to those listed herein. Such modifications and variations are intended to fall within the scope of the appended claims.

[0214] The above detailed description, with reference to the accompanying drawings, describes various features and functions of the disclosed systems, devices, and methods. In the drawings, similar symbols generally identify similar components unless the context otherwise requires. The exemplary embodiments described in the detailed description, drawings, and claims are not intended to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter described herein. It will be readily understood that, as generally described herein and illustrated in the figures, various aspects of the invention can be arranged, substituted, combined, separated, and designed in a variety of different configurations, all of which are expressly contemplated herein.

[0215] Regarding any or all ladder diagrams, scenarios, and flowcharts discussed herein, each block and / or communication may represent information processing and / or information transmission according to exemplary embodiments. Alternative embodiments are included within the scope of these exemplary embodiments. In these alternative embodiments, for example, functions described as blocks, transmissions, communications, requests, responses, and / or messages, depending on the functionality involved, may be executed in the order shown or discussed, including substantially concurrent or reverse execution. Furthermore, more or fewer blocks and / or functions may be associated with any ladder diagrams, scenarios, and flowcharts discussed herein. Figure 1 They can be used together, and these ladder diagrams, scenarios, and flowcharts can be partially or fully combined with each other.

[0216] A block representing information processing may correspond to a circuit that can be configured to perform a specific logical function of the method or technique described herein. Optionally or additionally, a block representing information processing may correspond to a module, segment, or portion of program code (including associated data). The program code may include one or more processor-executable instructions for implementing a specific logical function or action in the method or technique. The program code and / or associated data may be stored on any type of computer-readable media, such as storage devices including disks or hard disk drives or other storage media.

[0217] Computer-readable media may also include non-transitory computer-readable media, such as non-transitory computer-readable media that store data for short periods of time, such as register memory, processor cache, and random access memory (RAM). Computer-readable media may also include non-transitory computer-readable media that store program code and / or data for longer periods of time, such as auxiliary or persistent long-term storage, such as read-only memory (ROM), optical discs or magnetic disks, and optical disc read-only memory (CD-ROM). Computer-readable media may also be any other volatile or non-volatile storage system. Computer-readable media can be considered, for example, computer-readable storage media or tangible storage devices.

[0218] Furthermore, a block representing one or more information transfers can correspond to information transfers between software and / or hardware modules within the same physical device. However, other information transfers can occur between software and / or hardware modules in different physical devices.

[0219] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are provided for illustrative purposes and are not intended to be limiting; their true scope is indicated by the appended claims.

Claims

1. A computer-implemented method comprising: receiving, via a computing device, an input grayscale image comprising an array of pixels; applying a neural network to transform the input grayscale image into a colored high resolution version output image, wherein applying the neural network comprises: generating a low resolution rough colored version of the input grayscale image based at least in part on an axial transformer configured to apply alternating conditional mask column attention and conditional row attention, wherein the axial transformer comprises N conditional self-attention layers; generating the colored high resolution version output image of the input grayscale image by applying a color up-sampler and a spatial up-sampler to the low resolution rough colored version of the input grayscale image, wherein each of the color up-sampler and the spatial up-sampler comprises N layers corresponding to the N conditional self-attention layers, respectively; and providing, via the computing device, the generated colored high resolution version output image.

2. The computer-implemented method of claim 1, wherein, the colored version of the grayscale image comprises a high-fidelity colorization of the grayscale image.

3. The computer-implemented method of claim 1, wherein, the input grayscale image comprises a plurality of video frames.

4. The computer-implemented method of claim 1, wherein, generating the colored high resolution version output image further comprises: generating an intermediate output image by applying parallel down-sampling to each pixel of the array of pixels, wherein the intermediate output image comprises one or more of an intermediate color depth, an intermediate color intensity, an intermediate spatial resolution, or an intermediate color resolution; and parallel up-sampling each pixel in the intermediate output image to a respective one or more of an output color depth, an output color intensity, an output spatial resolution, or an output color resolution.

5. The computer-implemented method of claim 1, wherein, generating the colored high resolution version output image further comprises: generating a first intermediate output image comprising an intermediate color depth by applying parallel down-sampling to each pixel of the array of pixels; and generating a second intermediate output image comprising an intermediate color intensity by applying parallel down-sampling to each pixel of the first intermediate output image.

6. The computer-implemented method of claim 5, further comprising: parallel up-sampling each pixel in the second intermediate output image from the intermediate color intensity to a target color intensity; and parallel up-sampling each pixel in the up-sampled second intermediate output image from the intermediate color depth to a target color depth.

7. The computer-implemented method of claim 1, further comprising: training the neural network to receive a given input grayscale image and apply axial attention in order to transform the given input grayscale image into a colored high resolution version output image.

8. The computer-implemented method of claim 1, wherein, the neural network comprises an autoregressive colorizer that includes an encoder configured to apply alternating row attention and column attention.

9. The computer-implemented method of claim 8, wherein, the autoregressive colorizer further comprises one or more layers to apply conditional mask row attention.

10. The computer-implemented method of claim 1, wherein, generating the colored high resolution version output image further comprises: determining, by the computing device, a request to transform the input grayscale image into a colored high resolution version output image; sending the request from the computing device to a second computing device, the second computing device comprising a trained version of the neural network; and after sending the request, the computing device receives the output colored high resolution version image from the second computing device.

11. The computer-implemented method of claim 1, wherein, the computing device comprises a camera, and the method further comprises: accessing the input grayscale image using the camera; and At a computing device, receiving a colored high-resolution version output image from a camera.

12. The computer-implemented method of claim 1, wherein, Generating the colored high-resolution version output image includes: At the computing device, obtaining a trained version of a neural network, and wherein applying the neural network and generating the colored high-resolution version output image are performed by the computing device using the obtained trained version of the neural network.

13. A computing device comprising: one or more processors; and a data memory, wherein the data memory has stored thereon computer executable instructions that, when executed by the one or more processors, cause the computing device to perform functions comprising: receiving an input grayscale image comprising an array of pixels; applying a neural network to transform the input grayscale image into a colored high-resolution version output image, wherein applying the neural network includes: generating a low-resolution coarse coloring of the input grayscale image based at least in part on an axial transformer configured to apply alternating conditional mask column attention and conditional row attention, wherein the axial transformer includes N conditional self-attention layers; generating the colored high-resolution version output image of the input grayscale image by applying a color up-sampler and a spatial up-sampler to the low-resolution coarse coloring of the input grayscale image, wherein each of the color up-sampler and the spatial up-sampler includes N layers respectively corresponding to the N conditional self-attention layers; and providing, via the computing device, the generated colored high-resolution version output image.

14. An article of manufacture comprising one or more non-transitory computer- readable media having stored thereon computer-readable instructions that, when executed by one or more processors of a computing device, cause the computing device to perform functions comprising: receiving an input grayscale image comprising an array of pixels; applying a neural network to transform the input grayscale image into a colored high-resolution version output image, wherein applying the neural network includes: generating a low-resolution coarse coloring of the input grayscale image based at least in part on an axial transformer configured to apply alternating conditional mask column attention and conditional row attention, wherein the axial transformer includes N conditional self-attention layers; generating the colored high-resolution version output image of the input grayscale image by applying a color up-sampler and a spatial up-sampler to the low-resolution coarse coloring of the input grayscale image, wherein each of the color up-sampler and the spatial up-sampler includes N layers respectively corresponding to the N conditional self-attention layers; and providing, via the computing device, the generated colored high-resolution version output image.

Images