Machine Learning Discretization Level Reduction

Abstract

A computer-implemented method for providing level-reduced tensor data having an improved information representation may include: obtaining input tensor data; providing the input tensor data as input to a machine learning discretization level reduction model, the machine learning discretization level reduction model being configured to receive tensor data having a plurality of discretization levels and, in response to receiving the tensor data, generate level-reduced tensor data having a reduced number of discretization levels; and obtaining the level-reduced tensor data from the machine learning discretization level reduction model. The machine learning discretization level reduction model is trained using reconstructed input tensor data generated using outputs of the machine learning discretization level reduction model. The machine learning discretization level reduction model may include one or more level reduction layers configured to receive input having a first number of discretization levels and provide layer outputs having a reduced number of discretization levels.

Description

Technical Field

[0001] The present disclosure generally relates to systems and methods for binarization and / or other bit reduction of tensor data, such as images, and more specifically, to machine learning models that produce output tensor data with a reduced number of discretization levels (e.g., preserving and matching color information when compressing a color image to a black and white image). Background Art

[0002] Tensors can hold structured data. The data within a tensor can have multiple levels of discretization associated with it. As an example, an image can be represented as a discrete tensor with different intensity levels. As an example, an image can be represented by a combination of channels. For example, an image can be represented as a combination of various channels, each corresponding to a color, hue, intensity, etc. For example, some images can be represented as a tensor with a red channel, a blue channel, and a green channel, with varying intensity levels at each channel, corresponding to the intensity of the corresponding color at a point in the tensor. Display screens and other systems can display information, such as images, based on tensors. Summary of the Invention

[0003] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.

[0004] One example aspect of the present disclosure relates to a computer-implemented method for providing level-reduced tensor data with improved information representation. The computer-implemented method may include obtaining input tensor data. The computer-implemented method may include providing the input tensor data as input to a machine learning discretization level reduction model, the model configured to receive tensor data having a plurality of discretization levels and, in response to receiving the tensor data, generate level-reduced tensor data having a reduced number of discretization levels. The machine learning discretization level reduction model may include at least one input layer configured to receive tensor data and one or more level reduction layers connected to the at least one input layer, the one or more level reduction layers configured to receive input having a first number of discretization levels and provide layer output having a reduced number of discretization levels, wherein each level reduction layer is associated with a corresponding number of discretization levels, and the discretization levels are reduced at each of the one or more level reduction layers based at least in part on a discretization activation function having a corresponding number of discretization levels associated with the level reduction layer. The computer-implemented method may include obtaining the level-reduced tensor data from the machine learning discretization level reduction model. A machine learning discretization level reduction model is trained using reconstructed input tensor data generated using the output of the machine learning discretization level reduction model.

[0005] Another example aspect of the present disclosure relates to a computer-implemented method for training a discretization level reduction model to provide level-reduced tensor data with an improved representation of information. The computer-implemented method may include obtaining, by a computing system comprising one or more computing devices, training data, the training data comprising input tensor data. The computer-implemented method may include providing, by the computing system, the training data to the discretization level reduction model, the discretization level reduction model configured to receive tensor data comprising a plurality of discretization levels and, in response to receiving the tensor data, generate level-reduced tensor data having a reduced number of discretization levels. The computer-implemented method may include determining, by the computing system and based at least in part on the discretization level reduction model, the level-reduced tensor data. The computer-implemented method may include determining, by the computing system, reconstructing input tensor data based at least in part on the discretization level reduction model and based at least in part on the level-reduced tensor data. The computer-implemented method may include determining, by the computing system, a loss based at least in part on the input tensor data and the reconstructed input tensor data. The computer-implemented method may include adjusting, by the computing system, one or more parameters of the discretization level reduction model based at least in part on the loss.

[0006] Another example aspect of the present disclosure relates to one or more non-transitory computer-readable media storing a machine learning discretization level reduction model, the machine learning discretization level reduction model configured to receive tensor data comprising a plurality of discretization levels and, in response to receiving the tensor data, generate level-reduced tensor data comprising a reduced number of discretization levels. The machine learning discretization level reduction model may include at least one input layer configured to receive the tensor data and a plurality of level reduction layers connected to the at least one input layer, the plurality of level reduction layers configured to progressively and monotonically reduce the number of discretization levels of each of the plurality of level reduction layers.

[0007] In examples described herein, a machine learning discretization level reduction model is provided. The machine learning discretization level reduction model is configured to receive tensor data having a plurality of discretization levels and, in response to receiving the tensor data, generate level-reduced tensor data having a reduced number of discretization levels.

[0008] The machine learning discretization level reduction model can be trained using reconstructed input tensor data generated using outputs of the machine learning discretization level reduction model. The machine learning discretization level reduction model can be stored on one or more non-transitory computer-readable storage media.

[0009] A machine learning discretization level reduction model may include at least one input layer configured to receive tensor data; and one or more level reduction layers connected to the at least one input layer, the one or more level reduction layers configured to receive an input having a first number of discretization levels and to provide a layer output having a reduced number of discretization levels.

[0010] Each level reduction layer can be associated with a corresponding number of discretization levels, and the discretization levels can be reduced at each of the one or more level reduction layers based at least in part on a discretization activation function having the corresponding number of discretization levels associated with the level reduction layer. The discretization activation function can be a hyperbolic tangent function.

[0011] The one or more level reduction layers may each be configured to reduce the number of discretization levels based at least in part on a scaling factor. For example, the scaling factor may be half.

[0012] The one or more level reduction layers may progressively and monotonically reduce the number of discretization levels of each of the one or more level reduction layers.

[0013] The discretization level reduction model may include at least one feature representation layer configured to map input tensor data from an input layer to a feature representation of the input tensor data.

[0014] The discretization level reduction model may include at least one channel reduction layer configured to reduce input to at least one channel reduction layer input data having a first number of channels to output of at least one channel reduction layer having a reduced number of channels.

[0015] A machine learning discretization level reduction model may include an output layer configured to provide level-reduced tensor data.

[0016] A machine learning discretization level reduction model may include one or more reconstruction layers configured to reconstruct input tensor data from level-reduced tensor data.

[0017] The discretization level reduction model includes a color bypass network. The color bypass network can include one or more fully connected hidden units. For example, the color bypass network can include one to ten fully connected hidden units.

[0018] According to examples described herein, there is a computer-implemented method for providing level-reduced tensor data with improved information representation using a machine learning discretization level reduction model. The method includes obtaining input tensor data; providing the input tensor data as input to the machine learning discretization level reduction model; and obtaining level-reduced tensor data from the machine learning discretization level reduction model.

[0019] According to another example described herein, there is a computer-implemented method for training a discretized level reduction model to provide level-reduced tensor data having an improved information representation. The method includes obtaining training data, the training data including input tensor data; providing the training data to the discretized level reduction model; determining the level-reduced tensor data based at least in part on the discretized level reduction model; determining reconstructed input tensor data based at least in part on the discretized level reduction model and at least in part on the level-reduced tensor data; determining a loss based at least in part on the input tensor data and the reconstructed input tensor data; and adjusting one or more parameters of the discretized level reduction model based at least in part on the loss.

[0020] The loss may include pixel differences between the input tensor data and the reconstructed input tensor data.

[0021] In a case where the discretization level reduction model includes a color bypass network, determining the reconstructed input tensor data based at least in part on the level-reduced tensor data may include: obtaining a first reconstructed input tensor data component from one or more reconstruction layers, the first reconstructed input tensor data component being based at least in part on the level-reduced tensor data; obtaining a second reconstructed input tensor data component from the color bypass network, the second reconstructed input tensor data component being based at least in part on the input tensor data; and determining the reconstructed input tensor data based at least in part on the first reconstructed input tensor data component and the second reconstructed input data component.

[0022] The first reconstruction input tensor data component may include a reconstructed image, and the second reconstruction input tensor data component may include a hue of the reconstructed image.

[0023] In a method of using a machine learning discretized level reduction model or a method of training a machine learning discretized level reduction model, input tensor data includes image data, and wherein the level-reduced tensor data includes binarized image data.

[0024] The reduced number of discretization levels of the level-reduced tensor data is two discretization levels.

[0025] According to another example described herein, a system includes one or more processors and one or more computer-readable memory devices storing instructions that, when implemented, cause the one or more processors to perform any of the methods set forth above or below.

[0026] Other aspects of the present disclosure are directed to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0027] These and other features, aspects and advantages of various embodiments of the present disclosure will be better understood with reference to the following description and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] A detailed discussion of embodiments for those of ordinary skill in the art is set forth in the specification with reference to the accompanying drawings, in which:

[0029] Figure 1A Depicted is a block diagram of an example computing system that performs discretization level reduction according to example implementations of the present disclosure.

[0030] Figure 1B Depicted is a block diagram of an example computing device that performs discretization level reduction according to example implementations of the present disclosure.

[0031] Figure 1C Depicted is a block diagram of an example computing device that performs discretization level reduction according to example implementations of the present disclosure.

[0032] Figure 2 Depicted is a block diagram of an example discretization level reduction system according to an example implementation of the present disclosure.

[0033] Figure 3 Depicted is a block diagram of an example discretized level reduction model according to an example implementation of the present disclosure.

[0034] Figure 4 Depicted is a block diagram of an example discretized level reduction model according to an example implementation of the present disclosure.

[0035] Figure 5 Depicted is a block diagram of an example discretized level reduction model according to an example implementation of the present disclosure.

[0036] Figure 6A 、 6B , 6C, and 6D depict example discretized activation functions according to example implementations of the present disclosure.

[0037] Figure 7Depicted is a flow diagram of an example computer-implemented method for providing level-reduced tensor data with improved representation of visual information according to an example implementation of the present disclosure.

[0038] Figure 8 Depicted is a flow diagram of an example computer-implemented method for training a discretized level reduction model to provide level-reduced tensor data with improved representation of visual information, according to an example implementation of the present disclosure.

[0039] Reference numerals repeated across the various figures are intended to identify like features in the various embodiments. DETAILED DESCRIPTION

[0040] In general, the present disclosure relates to systems and methods for binarization and / or other bit reduction of tensor data, such as visual or otherwise displayable tensor data, such as images (e.g., two-dimensional images). Binarization refers to converting discretized tensor data having multiple discretization levels (e.g., 24 bits per level) into tensor data having only two discretization levels (e.g., 0 and 1, such as black and white or two-tone images). In addition, tensor data can be converted from having multiple channels (e.g., color channels) to tensor data having a single channel. As an example, the original (e.g., input) tensor data can be or include RGB image data having 256 (e.g., 8 bits) discretization levels, and the level-reduced (e.g., output) tensor data can be two-tone (e.g., black and white) image data having a single channel with two discretization levels, corresponding to two-tone pixel levels (e.g., black and white, shadow and non-shadow, etc.). The example aspects of the present disclosure can be generalized to any suitable level reduction, such as reducing tensor data to four discretization levels (e.g., two bits), eight discretization levels (e.g., three bits), etc.

[0041] As an example, bit reduction of tensor data may be useful where a medium intended to display or otherwise utilize the tensor data is unable (e.g., due to structural and / or other limitations) to convey sufficient information to accurately represent the tensor data. For example, a two-color display screen, such as included in an e-reader or e-ink system, may not be able to display an RGB image because the display's pixels may be limited to two colors (e.g., shaded / lit and non-shaded / lit). As another example, a printer may be configured to print black and white images, such as for newspaper printing, batch printing, photocopying, etc. As another example, a subtractive construction system, such as a CNC machine, laser etching, etc., may be able to perform subtractive construction based on an image, but may be limited to two levels (e.g., etched and not etched, cut or not cut, etc.) or a reduced number of discretization levels compared to the original number of discretization levels of the image. Example aspects of the present disclosure may find particular benefit in these and other scenarios where limited display media are intended to display full-color images and where it is desired to maintain the visual integrity (e.g., comprehensibility) of the image. Those skilled in the art will appreciate that the systems and methods described herein are discussed with respect to image data for purposes of illustration and may be extended to any suitable tensor data having multiple levels of discretization and / or one or more channels.

[0042] Some existing image binarization methods fail to preserve the visual information available in the original image in the binarized image. For example, one example method for image binarization is thresholding, in which each pixel of an image is converted to one of two colors (e.g., black and white) based on the intensity at that pixel. This intensity is typically the cross-channel intensity, such as the average intensity of each color. While this method can produce a binarized image, it may lose detail compared to the original image. For example, thresholding may fail to reproduce the distinction between differently colored areas in the binarized image, and instead produces uninterpretable shadow areas for many images, especially those with many different colors of similar intensity. Another example method is dithering. Dithering, like thresholding, often fails to capture the distinction between colors and, in addition, adds darkening or other noise to the output image. Furthermore, dithering often loses detail. Another example method is edge representation. Edge representation often exacerbates noise (e.g., JPEG compression noise) and may not represent colors, merely defining edges between colors. Furthermore, edge representation can become difficult to interpret for detailed images. Therefore, many (if not all) existing image binarization methods fail to maintain the visual integrity of the original image and often do not sufficiently resemble the original image. Furthermore, these images may be unpleasant to the viewer in addition to failing to convey the information available in the original image.

[0043] Another challenge in image binarization involves the lack of suitable training data for machine learning. For example, traditional generative machine learning techniques require existing example output data, such as a set of input and output data that represents the desired performance of the machine learning model. Manually creating a sufficient amount of adequately binarized or level-reduced output data may be difficult or impossible. Furthermore, creating binarized output data for training by existing methods produces output data that includes the aforementioned problems of existing methods. Using such training data may not allow the machine learning model to provide any improvement over existing methods. Therefore, challenges are encountered when using machine learning to binarize images.

[0044] Systems and methods according to example aspects of the present disclosure can provide solutions to these and other problems. For example, systems and methods according to example aspects of the present disclosure can provide level-reduced tensor data with improved representation of visual information. For example, if the level-reduced tensor data is image data, the reduced discretized level image data can better capture information available in the original image, such as channel (e.g., color) boundaries, shapes and regions, the subject matter of the image, etc., compared to level-reduced images produced by existing methods such as thresholding, dithering, edge representation, etc.

[0045] As used herein, a discretization level refers to one of a plurality of discrete values ​​that can be maintained by the value of a tensor within a particular channel. For example, an image having 256 discretization levels for each channel may include pixel values ​​having intensities between 0 and 255 for each channel and each pixel. Typically, multiple discretization levels may correspond to the number of bits used to store each item of tensor data and / or the output capacity of a medium that interprets the tensor data. For example, a data item in tensor data having 256 discretization levels may require 8 bits to store and / or may be used to drive the color of a pixel in a display screen to one of 256 discrete intensities. As another example, a data item in tensor data having two discretization levels may be used to turn a pixel on or off, print or not print a dot, etc. Although a greater number of channels and / or discretization levels can convey more information, this can also result in increased memory requirements for storage and / or increased display costs and / or (over) computational requirements.

[0046] According to example aspects of the present disclosure, level-reduced tensor data can be generated from input tensor data by a machine learning discretization level reduction model. The machine learning discretization level reduction model can be configured to receive input tensor data comprising at least one channel and, in response to receiving the input tensor data, generate level-reduced tensor data. The level-reduced tensor data can include a reduced number of discretization levels (e.g., compared to the input tensor data). The level-reduced tensor data can approximate (e.g., visually approximate) the input tensor data. For example, the reduced discretization level image can be a binary image with two discretization levels. A two-tone image can approximate a full-color image with more discretization levels (accounting for 256 discretization levels). Additionally and / or alternatively, in some embodiments, the level-reduced tensor data can include fewer channels than the input tensor data. For example, the level-reduced tensor data can include a single channel, while the input tensor data can include more than one channel (e.g., three channels, four channels, etc.).

[0047] In some embodiments, a machine learning discretized level reduction model may include multiple layers. For example, these layers may form a network that converts input tensor data into output tensor data. In addition, in some embodiments, these layers may reconstruct the input tensor data from the output tensor data. Reconstructing the input data may be an attempt to reconstruct the input tensor data using the level-reduced tensor data and, in some implementations, information from the color bypass network. For example, reconstructing the input tensor data may be used to determine the loss relative to the original input tensor data. The loss may be back-propagated through each layer to train the model. In some embodiments, reconstructed input data may be generated using only the level-reduced tensor data and / or the color bypass network information, which may intuitively provide the information required for the model to be trained to include the information required to reconstruct the input tensor data in the level-reduced tensor data. Reconstructing the input tensor data may be used to train the model.

[0048] The discretization level reduction model may include at least one input layer configured to receive tensor data. For example, the input layer may receive tensor data, such as pixel data (e.g., an M×N image). The input layer may serve as an entry point for the tensor data.

[0049] In some embodiments, the discretization level reduction model may include at least one feature representation layer. For example, in some implementations, the at least one feature representation layer may be or may include a convolutional layer, such as a 3×3, 6×6, or other convolutional layer. The feature representation layer(s) may map (e.g., via convolution) the input tensor data from the input layer to a feature representation of the input tensor data, such as a feature map. In some embodiments, the feature representation layer(s) may be (multiple) stride-1 convolutional layers, such as (multiple) 3×3, stride-1 convolutional layers.

[0050] For example, a convolutional layer can operate by applying a convolution kernel (such as a weight kernel) to data in a previous layer. The kernel can be applied at a center, such as at a corresponding position in a previous layer. The stride of the layer can refer to the number of positions that the kernel moves for each value in the convolutional layer. A value can be calculated by applying the convolution kernel. The value can be provided as an input to an activation function, and the output of the activation function can be a value at the convolutional layer (e.g., at a cell of the convolutional layer). According to example aspects of the present disclosure, it can be beneficial to use convolutional layers in a discretized level reduction model (e.g., at (multiple) level reduction layers). For example, convolutional layers can intuitively prevent binary representations (e.g., level-reduced tensor data) from becoming uninterpretable because these representations can be formed solely from the data specified by the kernel of the convolutional layer.

[0051] While convolutional layers are provided as one example implementation, it will be appreciated that other implementations may alternatively be used. By way of example only, a self-attention based model, such as a transformer, may be used alone or in combination with a convolutional layer to provide a feature representation layer.

[0052] In some implementations, the machine learning discretization level reduction model may be or may include a channel reduction layer. For example, the channel reduction layer may be configured to receive input data from a previous layer (e.g., (multiple) input layers and / or (multiple) feature representation layers). The input data from the previous layer may have a first number of channels, such as, for example, three channels, four channels, etc. The channel reduction layer may reduce the input data having the first number of channels to output data having a second (e.g., reduced) number of channels, such as, for example, a single channel. For example, the channel reduction layer may combine data from multiple channels into a reduced plurality of channels and / or a single channel. As an example, the channel reduction layer may intuitively convert data indicating a full-color image into data indicating a grayscale image corresponding to the full-color image. In some implementations, the channel reduction layer may retain multiple discretization levels. For example, the input data and / or output data of the channel reduction layer may have the same number of discretization levels.

[0053] According to example aspects of the present disclosure, a machine learning discretized level reduction model may include one or more level reduction layers connecting at least one input layer to an output layer. For example, the level reduction layer(s) may receive input data from the previous layer(s) (e.g., input layer(s), feature reduction layer(s), channel reduction layer(s), previous level reduction layer(s), etc.). In some implementations, the level reduction layer(s) may be or may include convolutional layers, such as 3×3, 6×6, etc. convolutional layers. In some implementations, the level reduction layer(s) may be stride-1 convolutional layers.

[0054] Each of the one or more level reduction layers can be configured to reduce the number of discretization levels based at least in part on a scaling factor. In some embodiments, the scaling factor can be half. For example, in some implementations, each of the (multiple) level reduction layers can reduce the discretization level at the output of the layer to half of the discretization level at the input of the layer. For example, if the input of a layer has a channel with 128 discretization levels, the output of the channel may have 64 discretization levels. According to example aspects of the present disclosure, other suitable scaling factors can be used to reduce the discretization levels. In some implementations, each level reduction layer can have the same scaling factor (e.g., half). Additionally and / or alternatively, in some implementations, a first level reduction layer can have a first scaling factor and a second level reduction layer can have a second scaling factor that is different from the first scaling factor.

[0055] As an example, the discretization levels can be reduced at each level reduction layer by a discretization activation function having a plurality of activation levels corresponding to the desired number of discretization levels for the layer. For example, in some embodiments, the (multiple) level reduction layers can each include a discretization activation function having a plurality of activation levels corresponding to the reduced number of discretization levels from the previous layer. In some embodiments, each level reduction layer can have a discretization activation function having half the number of activation levels of the previous layer (e.g., the immediately previous layer). In some embodiments, the discretization activation function can be a discretized hyperbolic tangent function. For example, for any given input, the discretized hyperbolic tangent function can be discretized into a discrete plurality of outputs.

[0056] In some embodiments, during training (e.g., backpropagation) of a discretized level-reduction model, at least the activation functions in the level-reduction layer(s) may be ignored. For example, the activation function may be used during forward propagation and / or inference, but not affected during the backpropagation step. For example, the activation function may not be modified during training.

[0057] According to example aspects of the present disclosure, a discretization level reduction model may include any suitable number of level reduction layers. For example, the number of level reduction layers may be based at least in part on the desired number of discretization levels at the output layer and / or the scaling factor by which each level reduction layer reduces the number of discretization levels. For example, one example implementation includes seven level reduction layers, each of which reduces the number of discretization levels at the output to half that at the input. For example, an example implementation may be configured to reduce input data having 256 discretization levels to binary output data having two discretization levels. As another example, if the desired output data has four discretization levels, only five level reduction layers may be included, each of which reduces the number of discretization levels to half that of the input. For example, in some implementations, the level reduction layer(s) may progressively and / or monotonically reduce the number of discretization levels of each level reduction layer in one or more level reduction layers. For example, each subsequent level reduction layer may have fewer discretization levels than the previous level reduction layer.

[0058] According to example aspects of the present disclosure, a discretization level reduction model may include an output layer configured to provide level-reduced tensor data. For example, the output layer may provide level-reduced tensor data as an output (e.g., an image) of the discretization level reduction model. In some embodiments, the output layer may also be a final level reduction layer. For example, in addition to providing output data as the output of the discretization level reduction model, the output layer may also reduce the input from the penultimate level reduction layer to output data having a desired number of discretization levels. As another example, the output layer may be a final level reduction layer configured to reduce the number of discretization levels of the input to the output layer to a reduced number of discretization levels of the level-reduced tensor data. In some embodiments, the reduced number of discretization levels of the level-reduced tensor data may be two discretization levels (e.g., 0 and 1). In some implementations, the output layer includes a spatial component (e.g., an image of M×N binary pixels) such that the representation found in the output layer can be returned directly as an image, such as without any further transformation or other modification. In some embodiments, the intermediate level reduction layer(s) may be omitted such that the model goes directly from the input resolution to the desired output resolution. Ignore

[0059] Additionally and / or alternatively, the machine learning discretization level reduction model may include one or more reconstruction layers. The reconstruction layer(s) may be after the output layer. For example, the reconstruction layer(s) may attempt to reconstruct the input tensor data from the level-reduced tensor data. In some embodiments, the reconstruction layer(s) may be structurally similar to and / or equivalent to the feature representation layer(s). For example, in some implementations, the reconstruction layer(s) may be or may include convolutional layers, such as 3×3, 6×6, etc. convolutional layers and / or stride-1 convolutional layers. The reconstruction layer(s) may be used at least during training and / or may not be used during inference. For example, the reconstruction layer(s) may be omitted from the deployed model and / or included in the deployed model, such as for adjusting the model after deployment. For example, the reconstructed input data may not be used or provided as an output of the model.

[0060] Intuitively, including at least reconstruction layers for training can ensure that the model learns to produce output tensor data that includes sufficient channel (e.g., color) and / or spatial information to accurately reconstruct the original tensor data (e.g., image). For example, this can result in sufficient color information being included in a binary image (e.g., as a learned binary pattern) so that the color information can be perceived in the binary image itself. Therefore, while the reconstruction layers may not be used to generate the final output of the machine learning discretization level reduction model, when used in the training step, they can provide improved generative capabilities of the model. This can be beneficial in cases where supervised training data is not readily available (e.g., suitable binarized images), as the model can be trained in an unsupervised manner solely on readily available input data (e.g., any suitable image).

[0061] In some embodiments, the dimensionality of the tensor data can be maintained by the machine learning discretization level reduction model. For example, some or all dimensions (e.g., length, width, height, etc.) can be the same as the corresponding dimensions of the level-reduced tensor data. For example, the binarized image produced by the machine learning discretization level reduction model can have the same visual dimensions (e.g., width×height) as the input image.

[0062] In some embodiments, the discretized level reduction model may further include a color bypass network. The color bypass network can pass image range information (e.g., color information) through some or all layers of the discretized level reduction model. For example, the color bypass network can pass image range information, such as hue and / or color information, to provide a color bypass reconstruction that is separate from the reconstruction generated by the reconstruction layer(s). The color bypass network can include one or more hidden units. In some embodiments, the color bypass network can be fully connected to a layer of the discretized level reduction model, such as an input layer. For example, the color bypass network can include one or more fully connected hidden units that are fully connected to the layer. For example, including fully connected hidden units can allow the hidden units to capture image range information. In implementations where the layers of the discretized level reduction model are convolutional layers, this can provide these layers (e.g., feature representation layer(s), level reduction layer(s), etc.) with the ability to capture local spatial information, while the color bypass network can capture image range information, such as color, hue, etc.

[0063] Intuitively, including a color bypass network allows image range information, such as color, hue, etc., to be passed to the color bypass reconstruction. This provides that this information does not have to be captured at the level-reduced tensor data, which may be useless in the level-reduced representation (e.g., because the level-reduced representation may lack, for example, color channels). Instead, this information is passed through a supplementary color bypass network, providing the level-reduced tensor data to include (e.g., in some implementations, with the aid of a convolutional layer) increased local spatial / boundary information, which is useful for providing the level-reduced tensor data with improved visual information. However, by passing this information through the color bypass network, it can be used to train the model. For example, as a first reconstruction input tensor data component, the reconstruction from the reconstruction layer can be combined with the color bypass reconstruction as a second reconstruction input tensor data component to reconstruct the reconstruction input tensor data. The model can then be trained on this reconstructed input tensor data (e.g., as opposed to reconstructing directly from the reconstruction layer).

[0064] In general, it is desirable that the color bypass network include enough hidden units to capture the desired image-wide information, but not so large that the color bypass network will capture local information, which can prevent that information from being included in the level-reduced tensor data. Thus, in some embodiments, the color bypass network can include 1 to 10 hidden units, such as 1 to 10 fully connected hidden units. For example, in some embodiments, the color bypass network can include two hidden units. Intuitively, these two hidden units can capture information related to the dimensions of the image, such as a color gradient in the width direction and / or a color gradient in the height direction, although this is described for illustrative purposes only and the hidden units can capture any suitable image-wide information.

[0065] In some embodiments, a computing system can be configured to provide level-reduced tensor data having an improved representation of visual information. According to example aspects of the present disclosure, a computing system can include (e.g., stored in a memory) a machine learning discretization level reduction model. For example, the discretization level reduction model can be configured to receive tensor data comprising at least one channel and, in response to receiving the tensor data, generate level-reduced tensor data comprising a reduced number of discretization levels for the at least one channel.

[0066] The computing system may include one or more processors and one or more computer-readable memory devices storing instructions that, when implemented, cause the one or more processors to perform operations. For example, the operations may implement a computer-implemented method for providing level-reduced tensor data with an improved representation of visual information. As an example, the operations may include obtaining tensor data. Additionally and / or alternatively, the operations may include providing the tensor data as input to a machine learning discretized level reduction model. Additionally and / or alternatively, the operations may include obtaining level-reduced tensor data from the machine learning discretized level reduction model.

[0067] In some embodiments, the machine learning discretization level reduction model can be stored in a computer-readable memory. For example, according to example aspects of the present disclosure, one or more non-transitory computer-readable media can store the machine learning discretization level reduction model. For example, the discretization level reduction model can be configured to receive tensor data comprising at least one channel and, in response to receiving the tensor data, generate level-reduced tensor data comprising a reduced number of discretization levels for the at least one channel.

[0068] In some embodiments, a computing system can be configured to implement a computer-implemented method for training a discretized level reduction model to provide level-reduced tensor data with an improved representation of visual information. For example, the computing system can include one or more computing devices. As an example, the computing system can be a training computing system configured to train and / or distribute the discretized level reduction model. As another example, the computing system can be a local computing system, such as a client computing system and / or a server computing system, configured to train and / or perform inference using the discretized level reduction model.

[0069] The computer-implemented method may include obtaining (e.g., via a computing system comprising one or more computing devices) training data. The training data may be any suitable training data for training a discretized level reduction model. For example, the training data may include input tensor data. In many cases, preparing supervised training data (e.g., pairs of input and desired output data) may be difficult or impossible, and therefore, the systems and methods described herein may provide unsupervised training. For example, the training data may include only input data, such as a library of images.

[0070] The computer-implemented method may include providing (e.g., via a computing system) training data to a discretization level reduction model. The discretization level reduction model may be configured to receive tensor data having a plurality of discretization levels and, in response to receiving the tensor data, generate level-reduced tensor data having a reduced number of discretization levels.

[0071] According to example aspects of the present disclosure, the discretization level reduction model can be any suitable discretization level reduction model. For example, in some embodiments, the discretization level reduction model may include at least one input layer configured to receive input tensor data. Additionally and / or alternatively, in some embodiments, the discretization level reduction model may include an output layer configured to provide level-reduced tensor data. Additionally and / or alternatively, in some implementations, the discretization level reduction model may include one or more level reduction layers connecting at least one input layer to the output layer. The one or more level reduction layers may be configured to reduce the number of discretization levels of each level reduction layer in the one or more level reduction layers. For example, in some implementations, the (multiple) level reduction layers may progressively and / or monotonically reduce the number of discretization levels of each level reduction layer in the one or more level reduction layers. For example, each subsequent level reduction layer may have fewer discretization levels than the previous level reduction layer.

[0072] Additionally and / or alternatively, in some embodiments, the discretized level reduction model may include one or more reconstruction layers configured to reconstruct the input tensor data from the level-reduced tensor data. Furthermore, in some embodiments, the discretized level reduction model may include a color bypass network, such as a color bypass network including one or more fully connected hidden units, such as from one to ten hidden units, such as two hidden units.

[0073] The computer-implemented method may include determining (e.g., by a computing system) level-reduced tensor data based at least in part on a discretized level reduction model. For example, the level-reduced tensor data may be determined by the discretized level reduction model, such as by an output layer of the discretized level reduction model. The level-reduced tensor data may be provided as an output and / or may be propagated for use in training a model (e.g., provided as an output or not). In some embodiments, the level-reduced tensor data may be or may include a binarized image generated from an input image of the training data. For example, in some embodiments, the input tensor data may be or may include image data, and / or the level-reduced tensor data may be or may include binarized image data. As an example, the level-reduced tensor data may be determined by providing the input tensor data to a discretized level reduction model that includes, for example, input layer(s), feature representation layer(s), channel reduction layer(s), level reduction layer(s), and / or output layer(s).

[0074] The computer-implemented method may include determining (e.g., by a computing system) to reconstruct input tensor data based at least in part on the discretized level reduction model and at least in part on the level-reduced tensor data. For example, the reconstructed input tensor data may be reconstructed from at least the level-reduced tensor data, such as by (multiple) reconstruction layers of the discretized level reduction model. The reconstructed input tensor data may be similar to the input tensor data. For example, the reconstructed input tensor data may include a greater amount of information than the level-reduced tensor data and / or information in a more readily perceptible manner, including information extrapolated from the level-reduced tensor data to reconstruct the input tensor data. Generally, it is desirable to reconstruct the input tensor data as close to the input tensor data as possible while conforming to the structure of the discretized level reduction model. In this way, the model can learn to provide sufficient spatial information at the level-reduced tensor data to closely reconstruct the input tensor data.

[0075] In some embodiments, such as embodiments in which the discretized level reduction model includes a color bypass network, determining the reconstructed input tensor data can be based at least in part on the level-reduced tensor data and the color bypass reconstruction. For example, in some embodiments, determining the reconstructed input tensor data can include obtaining (e.g., by a computing system) a first reconstructed input tensor data component. The first reconstructed input tensor data component can be obtained from one or more reconstruction layers. The first reconstructed input tensor data component can be based at least in part on the level-reduced tensor data. For example, the first reconstructed input tensor data component can be (e.g., intermediate) reconstructed input tensor data that is generated by the reconstruction layer from the level-reduced tensor data. As an example, the first reconstructed input tensor data component can be a reconstructed image (e.g., a full-color image) that approximates the input image. For example, the reconstructed image can have the same number of channels and / or discretization levels as the input image. According to example aspects of the present disclosure, by including information from the color bypass network, the image can be made to more closely approximate the input image.

[0076] Additionally and / or alternatively, in some embodiments, determining the reconstructed input tensor data may include obtaining (e.g., by a computing system) a second reconstructed input tensor data component. The second reconstructed input tensor data component may be obtained from a color bypass network. For example, in some embodiments, the second reconstructed input tensor data may be a color bypass reconstruction. For example, the second reconstructed input tensor data component may be obtained from a color bypass reconstruction layer that is included in and / or otherwise connected to the color bypass network. The second reconstructed input tensor data component may be based at least in part on the input tensor data. For example, in some embodiments, the second reconstructed input tensor data component may be obtained based at least in part on a color bypass network that is connected (e.g., fully connected, such as by including at least one fully connected hidden unit) to an input layer that includes the input tensor data. In some embodiments, the second reconstructed input tensor data component may be a reconstructed image based on the input image. The second reconstructed input tensor data component may be a reconstructed image that includes less local spatial information than the reconstructed image of the first reconstructed input tensor data component. For example, the second reconstructed input tensor data component may be a hue of the reconstructed image, such as one or more gradients.

[0077] Additionally and / or alternatively, in some embodiments, determining the reconstructed input tensor data may include determining (e.g., by a computing system) the reconstructed input tensor data based at least in part on the first reconstructed input tensor data component and the second reconstructed input data component. For example, in some embodiments, the reconstructed input tensor data may be determined based at least in part on a pixel-by-pixel combination of the first reconstructed input tensor data component and the second reconstructed input data component.

[0078] The computer-implemented method may include determining (e.g., by a computing system) a loss based at least in part on the input tensor data and the reconstructed input tensor data. For example, in some embodiments, the loss may be or may include a pixel difference between the input tensor data and the reconstructed input tensor data. For example, the loss may convey the difference between the input tensor data and the reconstructed input data. The loss may include or define one or more gradients, such as gradients of parameters of a reduced model with respect to a discretization level. For example, in some embodiments, the model may be trained using a backpropagation / optimization algorithm such as Adam.

[0079] The computer-implemented method may include adjusting (e.g., by a computing system) one or more parameters of a discretized level reduction model based at least in part on the loss. The discretized level reduction model may include one or more parameters (such as, for example, node and / or link weights, kernel weights, activation values ​​or levels, etc.) of layer(s) (such as input layer(s), feature representation layer(s), channel reduction layer(s), level reduction layer(s), output layer(s), reconstruction layer(s), etc.) and / or a color bypass network, and / or other parts of the discretized level reduction model. These parameters may be adjusted based on the loss, such as based on the gradient of the loss. For example, the loss (e.g., the gradient of the loss) may be backpropagated through the discretized level reduction model to adjust the parameters of the model, thereby training the model. In some embodiments, the activation value or level of a discretized activation function (such as a discretized hyperbolic tangent activation function) may not change during training. For example, because the discretized activation function is defined as a discretized input, it may not be necessary to move, scale, or otherwise modify the activation function during training. Therefore, during the backpropagation step, the activation levels of the discretized activation function can be ignored, which helps to simplify the training of the model.

[0080] In at least this way, a discretized level reduction model can be trained to produce level-reduced tensor data that includes sufficient information to reconstruct sufficiently accurate reconstructed input tensor data. This can provide level-reduced tensor data that includes a sufficient amount of spatial information, which can translate into, for example, improved visibility and / or usability of images of the level-reduced tensor data, as well as various other uses. Furthermore, the systems and methods described herein can provide for training discretized level reduction models even in situations where it is difficult and / or impossible to generate sufficient amounts of supervised training data. For example, the model can be trained using (e.g., only) readily available images with little or no modification required to the images.

[0081] Intuitively, a machine learning discretized level reduction model can learn to map the colors of a full-color image into different binary or other level-reduced hashes or textures. The model can also intuitively learn a "texture map" that visually reflects their source colors by being similar in the case of similar colors. This behavior is not well-defined and is, in fact, an unexpected consequence of configuring the machine learning model in the manner described according to the example aspects of the present disclosure. This behavior can provide for the generation of level-reduced images that can better capture visual information, thereby improving the usability of the image.

[0082] The systems and methods according to example aspects of the present disclosure may find application in a variety of applications. As an example, the systems and methods described herein may be used for two-tone printing. For example, two-tone printing may be performed faster and / or at a lower cost than, for example, grayscale and / or color printing. Two-tone printing may be suitable for batch printing of, for example, worksheets, newspapers, or other suitable media. For example, according to example aspects of the present disclosure, the systems and methods described herein may be used to convert grayscale and / or full-color images into two-tone images suitable for two-tone printing. As an example, the systems and methods described herein may be incorporated into driver software or other software associated with printer hardware. As another example, the systems and methods described herein may be used to prepare documents for printing.

[0083] As another example, the systems and methods described herein can be used as a network service or other image processing service. For example, a user can upload a (e.g., full-color) image to an image processing service and receive a binarized or otherwise bit-reduced image as output from the service. The service can be a local service, such as a service stored on a memory of a computing device operated by the user, and / or a network service, such as a service stored on a computing device remote from the user and / or accessed via the Internet or other network. As an example, the systems and methods described herein can be incorporated into an image filter that converts a full-color image into a binary or other bit-reduced image.

[0084] As another example, the systems and methods described herein can be used to generate images and / or schematics for some construction applications, such as subtractive construction (e.g., laser etching, CNC machines, robotic cutting machines, etc.). For example, the systems and methods described herein can be incorporated into driver software or other software associated with a subtractive construction system. As another example, the systems and methods described herein can be used to generate images or other (e.g., binary) schematics that are provided to a subtractive construction system(s).

[0085] As another example, the systems and methods described herein can be used to generate two-tone or other bit-reduced images for display on two-tone or other constrained displays. For example, the systems and methods described herein can be used to generate images for two-tone displays (e.g., two-tone pixel displays), such as e-readers, e-ink displays, calculators, etc. As an example, the systems and methods described herein can be included as software on a device that includes a two-tone display.

[0086] As another example, the systems and methods described herein can be used as a lossy compression scheme. For example, a discretized level reduction model can be used to generate level-reduced tensor data from input tensor data. The simplified discretized tensor data can require fewer computing resources (e.g., fewer bits in memory, less bandwidth, etc.) to store and / or transmit and / or interpret the input tensor data. The (multiple) reconstruction layers can then be used to reconstruct the input tensor data, such as at a later point in time and / or at a computing system other than the computing system that generated the level-reduced tensor data.

[0087] The systems and methods described herein can provide many technical effects and benefits, including but not limited to improvements to computing technology. As an example, the systems and methods described herein can generate level-reduced tensor data with improved spatial information preservation from input tensor data. This improved spatial information preservation can contribute to improved usability of the tensor data, such as, for example, improved visibility and / or information transmission capabilities of a binarized or otherwise level-reduced image. This can provide level-reduced tensor data that is more reflective of the input tensor data, which can improve usability as a lossy compression scheme, display on a limited-capability display, and the like.

[0088] As another example, improved spatial information retention can provide level-reduced tensor data with improved spatial information retention for use in applications for which higher level tensor data have been required because level-reduced tensor data according to conventional methods cannot convey enough information to be useful in those applications. For example, an application that previously required full-color images because traditional binarized images cannot convey enough spatial information may find it advantageous to use binarized images produced according to example aspects of the present disclosure that can convey sufficient spatial information. This can provide computational resource savings in at least these applications because the binarized images and / or other level-reduced tensor data produced according to example aspects of the present disclosure can have reduced computational resource requirements for storage, transmission, and / or interpretation (e.g., fewer bits per pixel).

[0089] Referring now to the drawings, example implementations of the present disclosure will be discussed in greater detail.

[0090] Figure 1A A block diagram of an example computing system 100 that performs discretization level reduction according to an example implementation of the present disclosure is depicted. System 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 communicatively coupled via a network 180.

[0091] The user computing device 102 may be any type of computing device, such as a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0092] The user computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one or more processors operatively connected. The memory 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 can store data 116 and instructions 118 that are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0093] In some implementations, the user computing device 102 may store or include one or more discretization level reduction models 120. For example, the discretization level reduction model 120 may be or may otherwise include various machine learning models, such as neural networks (e.g., deep neural networks) or other types of machine learning models, including nonlinear models and / or linear models. The neural network may include a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks. Figures 2 to 5 An example discretization level reduction model 120 is discussed.

[0094] In some implementations, one or more discretization level reduction models 120 may be received from the server computing system 130 via the network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing device 102 may implement multiple parallel instances of a single discretization level reduction model 120 (e.g., perform parallel discretization level reduction across multiple instances of the discretization level reduction model).

[0095] More specifically, according to example aspects of the present disclosure, level-reduced tensor data can be generated from input tensor data by a machine learning discretization level reduction model 120. The machine learning discretization level reduction model 120 can be configured to receive input tensor data comprising at least one channel and, in response to receiving the input tensor data, generate level-reduced tensor data. The level-reduced tensor data can include a reduced number of discretization levels (e.g., compared to the input tensor data). The level-reduced tensor data can approximate (e.g., visually approximate) the input tensor data. For example, a reduced discretization level image can be a binary image having two discretization levels. A two-tone image can approximate a full-color image having more discretization levels (such as 256 discretization levels). Additionally and / or alternatively, in some embodiments, the level-reduced tensor data can include fewer channels than the input tensor data. For example, the level-reduced tensor data can include a single channel, while the input tensor data can include more than one channel (e.g., three channels, four channels, etc.).

[0096] Additionally or alternatively, one or more discretization level reduction models 140 may be included in, or stored and implemented by, a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the discretization level reduction models 140 may be implemented by the server computing system 140 as part of a web service (e.g., a discretization level reduction service). Thus, one or more models 120 may be stored and implemented at the user computing device 102, and / or one or more models 140 may be stored and implemented at the server computing system 130.

[0097] The user computing device 102 may also include one or more user input components 122 for receiving user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user may provide user input.

[0098] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one or more processors operatively connected. The memory 134 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 can store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0099] In some implementations, server computing system 130 includes or is implemented by one or more server computing devices. Where server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0100] As described above, the server computing system 130 may store or otherwise include one or more machine learning discretization level reduction models 140. For example, the model 140 may be or may include various machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Figures 2 to 5 Discuss example model 140.

[0101] User computing device 102 and / or server computing system 130 may train models 120 and / or 140 by interacting with training computing system 150 communicatively coupled via network 180. Training computing system 150 may be separate from server computing system 130 or may be part of server computing system 130.

[0102] The training computing system 150 includes one or more processors 152 and a memory 154. The one or more processors 152 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one or more processors operatively connected. The memory 154 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158 executed by the processor 152 to cause the training computing system 150 to perform operations. In some embodiments, the training computing system 150 includes or is implemented by one or more server computing devices.

[0103] The training computing system 150 may include a model trainer 160 that trains the machine learning models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130 using various training or learning techniques, such as, for example, backpropagation of errors. For example, a loss function may be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques may be used to iteratively update parameters over multiple training iterations. In some embodiments, for example, the model(s) 120 and / or 140 may be stored at the model trainer 160 during training and subsequently transmitted to the user computing device 120 and / or the server computing system 130. The model trainer 160 may provide (multiple) layers and / or other components of a discretized model for inference (e.g., an input layer, an output layer, and (multiple) layers connected therebetween), and may provide and / or retain (multiple) layers and / or other components of a model for training (e.g., (multiple) reconstruction layers, (multiple) reconstruction output layers, color bypass networks, etc.).

[0104] In some embodiments, performing backpropagation of errors may include performing truncated backpropagation over time. The model trainer 160 may perform various generalization techniques (eg, weight decay, dropout, etc.) to improve the generalization ability of the trained model.

[0105] In particular, the model trainer 160 can train the discretization level reduction models 120 and / or 140 based on a set of training data 162. The training data 162 can include, for example, any suitable training data 162 for training (multiple) discretization level reduction models 120, 140. For example, the training data 162 can include input tensor data, such as image data (e.g., full-color image data). The image data can be provided in any suitable (e.g., digital) image format, such as, for example, BMP, JPEG / JPG, PNG, TIFF, or any other suitable format. In many cases, preparing supervised training data (e.g., pairs of input and desired output data) can be difficult or impossible, and therefore, the systems and methods described herein can provide unsupervised training. For example, the training data 162 can include only input data, such as an image library.

[0106] In some implementations, if the user has provided consent, the training examples may be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 may be trained by the training computing system 150 based on user-specific data received from the user computing device 102. In some cases, this process may be referred to as personalized modeling.

[0107] The model trainer 160 includes computer logic for providing the required functionality. The model trainer 160 can be implemented using hardware, firmware, and / or software that controls a general-purpose processor. For example, in some embodiments, the model trainer 160 includes a program file stored on a storage device, loaded into memory, and executed by one or more processors. In other embodiments, the model trainer 160 includes one or more sets of computer-executable instructions stored in a tangible computer-readable storage medium, such as a RAM hard disk or optical or magnetic media.

[0108] The network 180 can be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and can include any number of wired or wireless links. In general, communications on the network 180 can be carried via any type of wired and / or wireless connection using various communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0109] In some embodiments, the input of the (multiple) machine learning models of the present disclosure may be image data. The (multiple) machine learning models may process the image data to generate an output. As an example, the (multiple) machine learning models may process the image data to generate an image recognition output (e.g., recognition of the image data, potential embedding of the image data, an encoded representation of the image data, a hash of the image data, etc.). As another example, the (multiple) machine learning models may process the image data to generate an image segmentation output. As another example, the (multiple) machine learning models may process the image data to generate an image classification output. As another example, the (multiple) machine learning models may process the image data to generate an image data modification output (e.g., a change to the image data, etc.). As another example, the (multiple) machine learning models may process the image data to generate an encoded image data output (e.g., an encoded and / or compressed representation of the image data, etc.). As another example, the (multiple) machine learning models may process the image data to generate an amplified image data output. As another example, the (multiple) machine learning models may process the image data to generate a prediction output.

[0110] In some embodiments, the input of the (multiple) machine learning models of the present disclosure can be latent coded data (e.g., a latent space representation of the input, etc.). The (multiple) machine learning models can process the latent coded data to generate an output. For example, the (multiple) machine learning models can process the latent coded data to generate a recognition output. As another example, the (multiple) machine learning models can process the latent coded data to generate a reconstruction output. As another example, the (multiple) machine learning models can process the latent coded data to generate a search output. As another example, the (multiple) machine learning models can process the latent coded data to generate a re-clustering output. As another example, the (multiple) machine learning models can process the latent coded data to generate a prediction output.

[0111] In some embodiments, the input to the (multiple) machine learning models of the present disclosure may be sensor data. The (multiple) machine learning models may process the sensor data to generate an output. For example, the (multiple) machine learning models may process the sensor data to generate a recognition output. As another example, the (multiple) machine learning models may process the sensor data to generate a prediction output. As another example, the (multiple) machine learning models may process the sensor data to generate a classification output. As another example, the (multiple) machine learning models may process the sensor data to generate a segmentation output. As another example, the (multiple) machine learning models may process the sensor data to generate a segmentation output. As another example, the (multiple) machine learning models may process the sensor data to generate a visualization output. As another example, the (multiple) machine learning models may process the sensor data to generate a diagnostic output. As another example, the (multiple) machine learning models may process the sensor data to generate a detection output.

[0112] In some cases, (multiple) machine learning models can be configured to perform a task that includes encoding input data for reliable and / or efficient transmission or storage (and / or corresponding decoding). For example, the task can be an audio compression task. The input can include audio data, and the output can include compressed audio data. In another example, the input includes visual data (e.g., one or more images or videos), the output includes compressed visual data, and the task is a visual data compression task. In another example, the task can include generating an embedding for the input data (e.g., input audio or video data).

[0113] In some cases, the input includes visual data, and the task is a computer vision task. In some cases, the input includes pixel data from one or more images, and the task is an image processing task. For example, the image processing task may be image classification, where the output is a set of scores, each corresponding to a different object class, and representing the likelihood that one or more images depict an object belonging to that class. The image processing task may be object detection, where the image processing output identifies one or more regions in one or more images and, for each region, identifies the likelihood that the region depicts an object of interest. As another example, the image processing task may be image segmentation, where the image processing output defines, for each pixel in one or more images, the corresponding likelihood of belonging to each of a predetermined set of classes. For example, the set of classes may be foreground and background. As another example, the set of classes may be object classes. As another example, the image processing task may be depth estimation, where the image processing output defines a corresponding depth value for each pixel in one or more images. As another example, the image processing task may be motion estimation, where the network input includes multiple images, and the image processing output defines, for each pixel of one of the input images, the motion of the scene depicted at that pixel between the images in the network input.

[0114] Figure 1A An example computing system that can be used to implement the present disclosure is shown. Other computing systems may also be used. For example, in some implementations, the user computing device 102 may include a model trainer 160 and training data 162. In such implementations, the model 120 may be trained and used locally on the user computing device 102. In some such implementations, the user computing device 102 may implement the model trainer 160 to personalize the model 120 based on user-specific data.

[0115] Figure 1B Depicted is a block diagram of an example computing device 10 that performs discretization level reduction according to an example implementation of the present disclosure. Computing device 10 may be a user computing device or a server computing device.

[0116] Computing device 10 includes multiple applications (e.g., applications 1 to N). Each application includes its own machine learning library and (multiple) machine learning models. For example, each application can include a machine learning model. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, and the like.

[0117] like Figure 1BAs shown, each application can communicate with multiple other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some embodiments, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0118] Figure 1C Depicted is a block diagram of an example computing device 50 that performs discretization level reduction according to an example implementation of the present disclosure. The computing device 50 may be a user computing device or a server computing device.

[0119] The computing device 50 includes a plurality of applications (e.g., applications 1 through N). Each application communicates with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, and the like. In some embodiments, each application can communicate with the central intelligence layer (and the model(s) stored therein) using an API (e.g., a public API across all applications).

[0120] The central intelligence layer includes many machine learning models. For example, Figure 1C As shown, a corresponding machine learning model (e.g., model) can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine learning model. For example, in some embodiments, the central intelligence layer can provide a single model (e.g., a single model) for all applications. In some embodiments, the central intelligence layer is included in the operating system of the computing device 50, or is implemented by the operating system of the computing device 50.

[0121] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized data warehouse of the computing device 50. Figure 1C As shown, the central device data layer can communicate with multiple other components of the computing device, such as one or more sensors, context managers, device state components, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0122] Figure 2 A block diagram of an example discretization level reduction system 200 according to an example implementation of the present disclosure is depicted. The discretization level reduction system 200 can include a machine learning discretization level reduction model 202. In some embodiments, the discretization level reduction model 202 is trained to receive a set of input data 204 describing input tensor data and, as a result of receiving the input data 204, provide output data 206 describing level-reduced tensor data.

[0123] Figure 3 A block diagram of an example discretization level reduction model 300 according to an example implementation of the present disclosure is depicted. The discretization level reduction model 300 includes (multiple) discretization level reduction layers 302. The discretization level reduction layer(s) may include layers configured to reduce input data 204 (e.g., input tensor data) to output data 206 (e.g., level-reduced tensor data). For example, the discretization level reduction layer(s) 302 may be layers that produce the overall output of the discretization level reduction model 300. As examples, the discretization level reduction layer(s) 302 may be or may include (multiple) input layers, (multiple) feature representation layers, (multiple) channel reduction layers, (multiple) level reduction layers, and / or (multiple) output layers.

[0124] Additionally and / or alternatively, the discretized level reduction model 300 can include a reconstruction layer(s) 304. The reconstruction layer(s) 304 can generate reconstruction input data 306 (e.g., reconstruction input tensor data) from at least the output data 206 (e.g., level-reduced tensor data). For example, the reconstruction layer(s) 304 can include a reconstruction output layer that provides the reconstruction input data 306. The reconstruction input data 306 may or may not be provided as an output of the discretized level reduction model 300. Typically, the reconstruction input data 306 is used to train the model 300 to improve predictions of the output data 206, as described herein.

[0125] Figure 4 A block diagram of an example discretization level reduction model 400 according to an example implementation of the present disclosure is depicted. According to example aspects of the present disclosure, output data 206 (e.g., level-reduced tensor data) can be generated from input data 204 (e.g., input tensor data) by the discretization level reduction model 400. The machine learning discretization level reduction model 400 can be configured to receive input tensor data comprising at least one channel and, in response to receiving the input tensor data, generate level-reduced tensor data. The level-reduced tensor data can include a reduced number of discretization levels (e.g., compared to the input tensor data). The level-reduced tensor data can approximate (e.g., visually approximate) the input tensor data. For example, the reduced discretization level image can be a binary image having two discretization levels. A two-tone image can approximate a full-color image having more discretization levels (e.g., 256 discretization levels). Additionally and / or alternatively, in some embodiments, the level-reduced tensor data can include fewer channels than the input tensor data. For example, the level-reduced tensor data may include a single channel, while the input tensor data may include more than one channel (e.g., three channels, four channels, etc.).

[0126] In some embodiments, the machine learning discretized level reduction model 400 may include multiple layers. For example, these layers may form a network that converts input data 204 (e.g., input tensor data) into output data 206 (e.g., level-reduced tensor data). In addition, in some embodiments, these layers may reconstruct the input data 204 (e.g., input tensor data) from the output data 206 (e.g., level-reduced tensor data). The reconstructed input tensor data may be used to train the model 400. For example, the reconstructed input tensor data may be used to determine the loss relative to the original input tensor data. The loss may be back-propagated through each layer to train the model 400.

[0127] The discretization level reduction model 400 may include at least one input layer 402 configured to receive tensor data. For example, the input layer 402 may receive tensor data, such as pixel data (e.g., an M×N image). The input layer 402 may serve as an entry point for the tensor data.

[0128] In some embodiments, the discretization level reduction model 400 may include at least one feature representation layer 404. For example, in some implementations, the at least one feature representation layer 404 may be or may include a convolutional layer, such as a 3×3, 6×6, or other convolutional layer. The feature representation layer(s) 404 may map (e.g., via convolution) the input tensor data from the input layer 402 to a feature representation of the input tensor data, such as a feature map. In some embodiments, the feature representation layer(s) 404 may be a stride-1 convolutional layer(s), such as a 3×3 stride-1 convolutional layer(s).

[0129] For example, a convolutional layer can operate by applying a convolution kernel (such as a weight kernel) to data in a previous layer. The kernel can be applied at a center, such as a corresponding position in a previous layer. The stride of the layer can refer to the number of positions that the kernel moves for each value in the convolutional layer. A value can be calculated by applying the convolution kernel. The value can be provided as an input to an activation function, and the output of the activation function can be a value at the convolutional layer (e.g., a cell of the convolutional layer). According to example aspects of the present disclosure, it can be beneficial to use convolutional layers in a discretized level reduction model 400 (e.g., at (multiple) level reduction layers 408). For example, convolutional layers can intuitively prevent binary representations (e.g., level-reduced tensor data) from becoming uninterpretable because these representations can be formed solely from the data specified by the kernel of the convolutional layer.

[0130] In some embodiments, the machine learning discretization level reduction model 400 may be or include a channel reduction layer 406. For example, the channel reduction layer 406 may be configured to receive input data from a previous layer (e.g., input layer(s) 402 and / or feature representation layer(s) 404). The input data from the previous layer may have a first number of channels, such as, for example, three channels, four channels, etc. The channel reduction layer 406 may reduce the input data having the first number of channels to output data having a second (e.g., reduced) number of channels (e.g., a single channel). For example, the channel reduction layer 406 may combine data from multiple channels into a reduced number of channels and / or a single channel. As an example, the channel reduction layer 406 may intuitively convert data indicating a full-color image into data indicating a grayscale image corresponding to the full-color image. In some implementations, the channel reduction layer 406 may retain multiple discretization levels. For example, the input data and / or output data of the channel reduction layer 406 may have the same number of discretization levels.

[0131] According to example aspects of the present disclosure, the machine learning discretized level reduction model 400 may include one or more level reduction layers 408 connecting at least one input layer 402 to an output layer 410. For example, the level reduction layer(s) 408 may receive input data from previous layers (e.g., input layer(s) 402, feature reduction layer(s), channel reduction layer(s) 406, previous level reduction layer(s) 408, etc.). In some implementations, the level reduction layer(s) 408 may be or may include convolutional layers, such as 3×3, 6×6, etc. convolutional layers. In some implementations, the level reduction layer(s) 408 may be stride-1 convolutional layers.

[0132] One or more level reduction layers 408 can each be configured to reduce the number of discretization levels based at least in part on a scaling factor. In some embodiments, the scaling factor can be half. For example, in some implementations, each of the (multiple) level reduction layers 408 can reduce the discretization level at the output of the layer to half of the discretization level at the input of the layer. For example, if the input of the layer has a channel with 128 discretization levels, the output of the channel may have 64 discretization levels. According to example aspects of the present disclosure, other suitable scaling factors can be used to reduce the discretization levels. In some embodiments, each level reduction layer 408 can have the same scaling factor (e.g., half). Additionally and / or alternatively, in some implementations, the first level reduction layer 408 can have a first scaling factor, and the second level reduction layer 408 can have a second scaling factor different from the first scaling factor.

[0133] In some implementations, the level reduction layer(s) 408 may progressively and / or monotonically reduce the number of discretization levels in each of the one or more level reduction layers. For example, each subsequent level reduction layer 408 may have fewer discretization levels than the previous level reduction layer 408. As an example, the discretization levels may be reduced at each level reduction layer 408 by a discretized activation function having a number of activation levels corresponding to the desired number of discretization levels at that layer. For example, in some implementations, the level reduction layer(s) 408 may each include a discretized activation function having a number of activation levels corresponding to the reduced number of discretization levels from the previous layer. In some embodiments, each level reduction layer 408 may have a discretized activation function having half the number of activation levels of the previous layer (e.g., the immediately previous layer). In some embodiments, the discretized activation function may be a discretized hyperbolic tangent function. For example, for any given input, the discretized hyperbolic tangent function may be discretized into a discrete plurality of outputs.

[0134] In some embodiments, at least the activation functions in the level reduction layer(s) 408 can be ignored during backpropagation of the discretized level reduction model 400. For example, the activation function can be used during forward propagation and / or inference, but not affected during the backpropagation step. For example, the activation function may not be modified during training.

[0135] According to example aspects of the present disclosure, the discretization level reduction model 400 may include any suitable number of level reduction layers 408. For example, the number of level reduction layers 408 may be based at least in part on the desired number of discretization levels at the output layer 410 and / or a scaling factor by which each level reduction layer 408 reduces the number of discretization levels. For example, one example implementation includes seven level reduction layers 408, each of which reduces the number of discretization levels at the output to half that at the input. For example, an example implementation may be configured to reduce input data having 256 discretization levels to binary output data having two discretization levels. As another example, if the desired output data has four discretization levels, only five level reduction layers 408 may be included, each of which reduces the number of discretization levels to half that of the input.

[0136] According to an example aspect of the present disclosure, the discretization level reduction model 400 may include an output layer 410 that is configured to provide level-reduced tensor data. For example, the output layer 410 may provide level-reduced tensor data as an output (e.g., an image) of the discretization level reduction model 400. In some embodiments, the output layer 410 may additionally be the final level reduction layer 408. For example, in addition to providing output data as the output of the discretization level reduction model 400, the output layer 410 may also reduce the input from the penultimate level reduction layer 408 to output data having a desired number of discretization levels. As another example, the output layer 410 may be the final level reduction layer 408 that is configured to reduce the number of discretization levels of the input to the output layer 410 to a reduced number of discretization levels of the level-reduced tensor data. In some embodiments, the reduced number of discretization levels of the level-reduced tensor data may be two discretization levels (e.g., 0 and 1). In some implementations, the output layer 410 includes a spatial component (e.g., an image of M×N binary pixels), such that the representation found in the output layer 410 can be returned directly as an image, such as without any further transformation or other modification.

[0137] Additionally and / or alternatively, the machine learning discretized level reduction model 400 may include one or more reconstruction layers 412. The reconstruction layer(s) 412 may follow the output layer 410. For example, the reconstruction layer(s) 412 may attempt to reconstruct the input tensor data from the level-reduced tensor data. As an example, the reconstruction output layer 414 (e.g., the final reconstruction layer) may provide the reconstructed input tensor data. In some embodiments, the reconstruction layer(s) 412 may be structurally similar and / or identical to the feature representation layer(s) 404. For example, in some implementations, the reconstruction layer(s) 412 may be or may include convolutional layer(s), such as 3×3, 6×6, etc. convolutional layers and / or stride-1 convolutional layers. The reconstruction layer(s) 412 may be used at least during training and / or may not be used during inference. For example, the reconstruction layer(s) 412 may be omitted from the deployed model 400 and / or included in the deployed model 400, such as for tuning the model 400 after deployment. For example, reconstruction input data may not be used or provided as an output of the model 400 .

[0138] Intuitively, including the reconstruction layer(s) 412 for at least training can ensure that the model 400 learns to produce output tensor data that includes sufficient channel (e.g., color) and / or spatial information to accurately reconstruct the original tensor data (e.g., image). For example, this can result in sufficient color information being included in a binary image (e.g., as a learned binary pattern) so that the color information can be perceived in the binary image itself. Thus, while the reconstruction layer(s) 412 may not be used to generate the final output of the machine learning discretized level reduction model 400, when used in the training step, they can provide improved generative capabilities of the model 400. This can be beneficial in situations where supervised training data (e.g., suitable binarized images) is not readily available, as the model 400 can be trained in an unsupervised manner solely on readily available input data (e.g., any suitable image).

[0139] In some embodiments, the dimensionality of the tensor data can be maintained by the machine learning discretization level reduction model 400. For example, some or all dimensions (e.g., length, width, height, etc.) can be the same as the corresponding dimensions of the level-reduced tensor data. For example, the binarized image produced by the machine learning discretization level reduction model 400 can have the same visual dimensions (e.g., width×height) as the input image.

[0140] Figure 5 A block diagram of an example discretization level reduction model 500 according to an example implementation of the present disclosure is depicted. According to example aspects of the present disclosure, output data 206 (e.g., level-reduced tensor data) can be generated from input data 204 (e.g., input tensor data) by the discretization level reduction model 500. The machine learning discretization level reduction model 500 can be configured to receive input tensor data comprising at least one channel and, in response to receiving the input tensor data, generate level-reduced tensor data. The level-reduced tensor data can include a reduced number of discretization levels (e.g., compared to the input tensor data). The level-reduced tensor data can approximate (e.g., visually approximate) the input tensor data. For example, the reduced discretization level image can be a binary image having two discretization levels. A two-tone image can approximate a full-color image having more discretization levels (e.g., 256 discretization levels). Additionally and / or alternatively, in some embodiments, the level-reduced tensor data can include fewer channels than the input tensor data. For example, the level-reduced tensor data may include a single channel, while the input tensor data may include more than one channel (e.g., three channels, four channels, etc.).

[0141] In some embodiments, the machine learning discretized level reduction model 500 may include multiple layers. For example, these layers may form a network that converts input data 204 (e.g., input tensor data) into output data 206 (e.g., level-reduced tensor data). In addition, in some embodiments, these layers may reconstruct the input data 204 (e.g., input tensor data) from the output data 206 (e.g., level-reduced tensor data). The reconstructed input tensor data may be used to train the model 500. For example, the reconstructed input tensor data may be used to determine the loss relative to the original input tensor data. The loss may be back-propagated through each layer to train the model 500.

[0142] The discretization level reduction model 500 may include at least one input layer 502 configured to receive tensor data. For example, the input layer 502 may receive tensor data, such as pixel data (e.g., an M×N image). The input layer 502 may serve as an entry point for the tensor data.

[0143] In some embodiments, the discretization level reduction model 500 may include at least one feature representation layer 504. For example, in some implementations, the at least one feature representation layer 504 may be or may include a convolutional layer, such as a 3×3, 6×6, or other convolutional layer. The feature representation layer(s) 504 may map (e.g., via convolution) the input tensor data from the input layer 502 to a feature representation of the input tensor data, such as a feature map. In some embodiments, the feature representation layer(s) 504 may be a stride-1 convolutional layer(s), such as a 3×3, stride-1 convolutional layer(s).

[0144] For example, a convolutional layer can operate by applying a convolution kernel (such as a weight kernel) to data in a previous layer. The kernel can be applied at a center, such as at a corresponding position in a previous layer. The stride of the layer can refer to the number of positions that the kernel moves for each value in the convolutional layer. A value can be calculated by applying the convolution kernel. The value can be provided as an input to an activation function, and the output of the activation function can be a value at the convolutional layer (e.g., a cell of the convolutional layer). According to example aspects of the present disclosure, it can be beneficial to use convolutional layers in the discretized level reduction model 500 (e.g., at (multiple) level reduction layers 508). For example, convolutional layers can intuitively prevent binary representations (e.g., level-reduced tensor data) from becoming uninterpretable because these representations can be formed solely from the data specified by the kernel of the convolutional layer.

[0145] In some implementations, the machine learning discretization level reduction model 500 may be or include a channel reduction layer 506. For example, the channel reduction layer 506 may be configured to receive input data from a previous layer (e.g., input layer(s) 502 and / or feature representation layer(s) 504). The input data from the previous layer may have a first number of channels, such as three channels, four channels, etc. The channel reduction layer 506 may reduce the input data having the first number of channels to output data having a second (e.g., reduced) number of channels (such as a single channel). For example, the channel reduction layer 506 may combine data from multiple channels into a reduced number of channels and / or a single channel. As an example, the channel reduction layer 506 may intuitively convert data indicating a full-color image into data indicating a grayscale image corresponding to the full-color image. In some implementations, the channel reduction layer 506 may retain multiple discretization levels. For example, the input data and / or output data of the channel reduction layer 506 may have the same number of discretization levels.

[0146] According to example aspects of the present disclosure, the machine learning discretized level reduction model 500 may include one or more level reduction layers 508 connecting at least one input layer 502 to an output layer 510. For example, the level reduction layer(s) 508 may receive input data from previous layers (e.g., input layer(s) 502, feature reduction layer(s), channel reduction layer(s) 506, previous level reduction layer(s) 508, etc.). In some implementations, the level reduction layer(s) 508 may be or may include convolutional layers, such as 3×3, 6×6, etc. convolutional layers. In some implementations, the level reduction layer(s) 508 may be stride-1 convolutional layers.

[0147] One or more level reduction layers 508 can each be configured to reduce the number of discretization levels based at least in part on a scaling factor. In some embodiments, the scaling factor can be half. For example, in some implementations, each of the (multiple) level reduction layers 508 can reduce the discretization level at the output of the layer to half of the discretization level at the input of the layer. For example, if the input of the layer has a channel with 128 discretization levels, the output of the channel may have 64 discretization levels. According to example aspects of the present disclosure, other suitable scaling factors can be used to reduce the discretization levels. In some embodiments, each level reduction layer 508 can have the same scaling factor (e.g., half). Additionally and / or alternatively, in some implementations, the first level reduction layer 508 can have a first scaling factor, and the second level reduction layer 508 can have a second scaling factor different from the first scaling factor.

[0148] As an example, the discretization levels can be reduced in each level reduction layer 508 by a discretization activation function having a plurality of activation levels corresponding to the desired number of discretization levels for that layer. For example, in some embodiments, the (multiple) level reduction layers 508 can each include a discretization activation function having a plurality of activation levels corresponding to the reduced number of discretization levels from the previous layer. In some embodiments, each level reduction layer 508 can have a discretization activation function having half the number of activation levels of the previous layer (e.g., the immediately previous layer). In some embodiments, the discretization activation function can be a discretized hyperbolic tangent function. For example, for any given input, the discretized hyperbolic tangent function can be discretized into a discrete plurality of outputs.

[0149] In some embodiments, at least the activation functions in the level reduction layer(s) 508 may be ignored during backpropagation of the discretized level reduction model 500. For example, the activation functions may be used during forward propagation and / or inference, but not affected during the backpropagation step. For example, the activation functions may not be modified during training.

[0150] According to example aspects of the present disclosure, the discretization level reduction model 500 may include any suitable number of level reduction layer(s) 508. For example, the number of level reduction layer(s) 508 may be based at least in part on the desired number of discretization levels at the output layer 510 and / or a scaling factor by which each level reduction layer 508 reduces the number of discretization levels. For example, one example implementation includes seven level reduction layers 508, each of which reduces the number of discretization levels at the output to half that at the input. For example, an example implementation may be configured to reduce input data having 256 discretization levels to binary output data having two discretization levels. As another example, if the desired output data has four discretization levels, only five level reduction layers 508 may be included, each of which reduces the number of discretization levels to half that of the input.

[0151] According to an example aspect of the present disclosure, the discretization level reduction model 500 may include an output layer 510 that is configured to provide level-reduced tensor data. For example, the output layer 510 may provide level-reduced tensor data as an output (e.g., an image) of the discretization level reduction model 500. In some embodiments, the output layer 510 may additionally be a final level reduction layer 508. For example, in addition to providing output data as an output of the discretization level reduction model 500, the output layer 510 may also reduce the input from the penultimate level reduction layer 508 to output data having a desired number of discretization levels. As another example, the output layer 510 may be a final level reduction layer 508 that is configured to reduce the number of discretization levels of the input to the output layer 510 to a reduced number of discretization levels of the level-reduced tensor data. In some embodiments, the reduced number of discretization levels of the level-reduced tensor data may be two discretization levels (e.g., 0 and 1). In some implementations, the output layer 510 includes a spatial component (e.g., an image of M×N binary pixels), such that the representation found in the output layer 510 can be returned directly as an image, such as without any further transformation or other modification.

[0152] Additionally and / or alternatively, the machine learning discretized level reduction model 500 may include one or more reconstruction layers 512. The reconstruction layer(s) 512 may follow the output layer 510. For example, the reconstruction layer(s) 512 may attempt to reconstruct the input tensor data from the level-reduced tensor data. As an example, the reconstruction output layer 514 (e.g., the final reconstruction layer) may provide the reconstructed input tensor data. In some embodiments, the reconstruction layer(s) 512 may be structurally similar and / or identical to the feature representation layer(s) 504. For example, in some implementations, the reconstruction layer(s) 512 may be or may include convolutional layer(s), such as 3×3, 6×6, etc. convolutional layers and / or stride-1 convolutional layers. The reconstruction layer(s) 512 may be used at least during training and / or may not be used during inference. For example, the reconstruction layer(s) 512 may be omitted from the deployed model 500 and / or included in the deployed model 500, such as for tuning the model 500 after deployment. For example, reconstruction input data may not be used or provided as an output of the model 500 .

[0153] Intuitively, including the reconstruction layer(s) 512 for at least training can ensure that the model 500 learns to produce output tensor data that includes sufficient channel (e.g., color) and / or spatial information to accurately reconstruct the original tensor data (e.g., image). For example, this can result in sufficient color information being included in a binary image (e.g., as a learned binary pattern) so that the color information can be perceived in the binary image itself. Thus, while the reconstruction layer(s) 512 may not be used to generate the final output of the machine learning discretization level reduction model 500, when used in the training step, they can provide improved generative capabilities of the model 500. This can be beneficial in situations where supervised training data (e.g., suitable binarized images) is not readily available, as the model 500 can be trained in an unsupervised manner solely on readily available input data (e.g., any suitable image).

[0154] In some embodiments, the dimensionality of the tensor data can be maintained by the machine learning discretization level reduction model 500. For example, some or all dimensions (e.g., length, width, height, etc.) can be the same as the corresponding dimensions of the level-reduced tensor data. For example, the binarized image produced by the machine learning discretization level reduction model 500 can have the same visual dimensions (e.g., width×height) as the input image.

[0155] The discretized level reduction model 500 can further include a color bypass network 522. The color bypass network 522 can pass image range information (e.g., color information) through some or all layers of the discretized level reduction model 500. For example, the color bypass network 522 can pass image range information, such as hue and / or color information, to provide a color bypass reconstruction at the color bypass reconstruction output layer 524 that is separate from the reconstruction generated by the reconstruction layer(s) 512 (e.g., at the reconstruction output layer 514). The color bypass network 522 can include one or more hidden units. In some embodiments, the color bypass network 522 can be fully connected to a layer of the discretized level reduction model 500, such as, for example, the input layer 502. For example, the color bypass network 522 can include one or more fully connected hidden units that are fully connected to the input layer 502. For example, including fully connected hidden units can allow the hidden units to capture image range information. In an implementation where the layers of the discretized level reduction model 500 are convolutional layers, this can provide that these layers (e.g., feature representation layer(s), level reduction layer(s), etc.) can capture local spatial information, while the color bypass network 522 can capture image range information, such as color, hue, etc. As an example, the color bypass network can typically capture the overall brightness or overall shadow effect, such as capturing that the upper right corner is the brightest and the lower left corner is the darkest.

[0156] Intuitively, the inclusion of the color bypass network 522 can provide image range information, such as color, hue, etc., to be passed to the color bypass reconstruction at the color bypass reconstruction output layer 524. This eliminates the need to capture this information at the level-reduced tensor data at the output data 206, which may be useless in the level-reduced representation (e.g., because the level-reduced representation may lack, for example, color channels). Instead, this information is passed through the supplementary color bypass network 522 so that the level-reduced tensor data includes (e.g., in some implementations, with the aid of a convolutional layer) increased local spatial / boundary information, which is useful for providing the level-reduced tensor data with improved spatial (e.g., visual) information. However, by passing this information through the color bypass network 522, it can be used to train the model 500. For example, the reconstruction from the reconstruction layers 512 and / or 514 as a first reconstruction input tensor data component can be combined with the color bypass reconstruction from the color bypass reconstruction output layer 524 as a second reconstruction input tensor data component to reconstruct the reconstruction input tensor data. Model 500 can then be trained on this reconstructed input tensor data (e.g., as opposed to reconstructing directly from reconstruction layers 512 and / or 514). As an example, components 514 and 524 can be combined by pixel-wise addition, such as by adding them together pixel by pixel.

[0157] In general, it is desirable that the color bypass network 522 include enough hidden units to capture the desired image range information, but not so large that the color bypass network 522 will capture local information, which can prevent that information from being included in the level-reduced tensor data. Thus, in some embodiments, the color bypass network 522 can include one to ten hidden units, such as one to ten fully connected hidden units. For example, in some embodiments, the color bypass network 522 can include two hidden units. Intuitively, these two hidden units can capture information related to the dimensions of the image, such as a color gradient in the width direction and / or a color gradient in the height direction, although this is described for illustrative purposes only and the hidden units can capture any suitable image range information.

[0158] Figure 6A 、 6B, 6C and 6D depict example discretized activation functions 600, 620, 640 and 660 according to example implementations of the present disclosure. For example, the discretized activation functions 600, 620, 640, 660 are discretized hyperbolic tangent functions with a decreasing number of activation levels (e.g., corresponding to decreasing discretization levels). As an example, the discretized activation function 600 includes 256 activation levels and can produce a layer output with 256 discretization levels. For example, the input to a layer including the function 600 will be used as the input of the function 600 and mapped to the output of the function as a value retained at the layer. For example, if the input tensor data has 256 discretization levels, such as input image data having 8 bits per pixel per channel, the activation function 600 can be included in the first level reduction layer. Similarly, Figure 6B A discretized activation function 620 having 64 activation levels is depicted, corresponding to 64 discretization levels. The activation function 620 can be included in a level reduction layer that produces an output having 64 discretization levels, such as a third level reduction layer (e.g., in an implementation where each level reduction layer reduces the number of discretization levels to half the number of inputs). Similarly, Figure 6C A discretized activation function 640 with 16 activation levels is depicted, corresponding to the 16 discretization levels. Figures 6A to 6C As shown, a reduction in the number of activation levels generally corresponds to a reduction in the granularity of the output data, which can provide less information conveyed by the data while reducing the requirements for storing, transmitting and / or interpreting the data. Figure 6D An activation function 660 is depicted having only two discretization levels, 0 and 1. For example, the activation function 660 may be included as an activation function of an output layer and / or a final level reduction layer to provide a binarized output.

[0159] Figure 7 A flow diagram of an example computer-implemented method 700 for providing level-reduced tensor data with improved (e.g., spatial) information representation is depicted in accordance with an example implementation of the present disclosure. Although for purposes of illustration and discussion, Figure 7 The steps are depicted as being performed in a particular order, but the method of the present disclosure is not limited to the particular illustrated order or arrangement. The steps of method 700 may be omitted, rearranged, combined, and / or modified in various ways without departing from the scope of the present disclosure.

[0160] The computer-implemented method 700 may include, at 702, obtaining (e.g., by a computing system) input tensor data. For example, the input tensor data may be obtained from a user, such as in response to the user performing a file upload or file transfer action. As another example, the input tensor data may be received from a separate computing system. In some embodiments, the input tensor data may be or may include image data, such as a full-color image.

[0161] The computer-implemented method 700 may include providing (e.g., by a computing system) input tensor data as input to a machine learning discretization level reduction model, at 704. The discretization level reduction model may be configured to receive tensor data having a plurality of discretization levels and, in response to receiving the tensor data, generate level-reduced tensor data having a reduced number of discretization levels.

[0162] The computer-implemented method 700 may include obtaining (e.g., via a computing system) level-reduced tensor data from a machine learning discretization level-reduction model at 706. For example, the model may provide the level-reduced tensor data as an output of the model. The level-reduced tensor data may have a reduced number of discretization levels from the input tensor data.

[0163] The computer-implemented method 700 may include displaying the level-reduced tensor data (e.g., by a computing system), at 708. For example, the level-reduced tensor data may be displayed (e.g., as an image), provided to a printer, construction machine, or other suitable device, and / or otherwise presented to a user.

[0164] Figure 8 A flow chart of an example computer-implemented method 800 for training a discretized level-reduction model to provide level-reduced tensor data with improved (e.g., spatial) information representation is depicted in accordance with an example implementation of the present disclosure. Although for purposes of illustration and discussion, Figure 8 The steps are depicted as being performed in a particular order, but the method of the present disclosure is not limited to the particular illustrated order or arrangement. The steps of method 800 may be omitted, rearranged, combined, and / or modified in various ways without departing from the scope of the present disclosure.

[0165] The computer-implemented method 800 may include, at 802, obtaining (e.g., via a computing system including one or more computing devices) training data. The training data may be any suitable training data for training a discretized level reduction model. For example, the training data may include input tensor data. In many cases, preparing supervised training data (e.g., pairs of input and desired output data) may be difficult or impossible, and therefore, the systems and methods described herein may provide unsupervised training. For example, the training data may include only input data, such as a library of images.

[0166] The computer-implemented method 800 may include providing (e.g., by a computing system) the training data to a discretization level reduction model at 804. The discretization level reduction model may be configured to receive tensor data having a plurality of discretization levels and, in response to receiving the tensor data, generate level-reduced tensor data having a reduced number of discretization levels.

[0167] According to example aspects of the present disclosure, the discretization level reduction model can be any suitable discretization level reduction model. For example, in some embodiments, the discretization level reduction model may include at least one input layer configured to receive input tensor data. Additionally and / or alternatively, in some embodiments, the discretization level reduction model may include an output layer configured to provide level-reduced tensor data. Additionally and / or alternatively, in some implementations, the discretization level reduction model may include one or more level reduction layers connecting at least one input layer to the output layer. The one or more level reduction layers may be configured to reduce the number of discretization levels of each level reduction layer in the one or more level reduction layers. Additionally and / or alternatively, in some embodiments, the discretization level reduction model may include one or more reconstruction layers configured to reconstruct the input tensor data from the level-reduced tensor data. Furthermore, in some embodiments, the discretization level reduction model may include a color bypass network, such as a color bypass network including one or more fully connected hidden units (such as from one to ten hidden units, such as two hidden units).

[0168] The computer-implemented method 800 may include, at 806, determining (e.g., by a computing system) level-reduced tensor data based, at least in part, on the discretized level reduction model. For example, the level-reduced tensor data may be determined by the discretized level reduction model, such as by an output layer of the discretized level reduction model. The level-reduced tensor data may be provided as an output and / or may be propagated for use in training the model (e.g., provided as an output or not). In some embodiments, the level-reduced tensor data may be or may include a binarized image generated from an input image of the training data. For example, in some embodiments, the input tensor data may be or may include image data. Additionally and / or alternatively, the level-reduced tensor data may be or may include binarized image data. As an example, the level-reduced tensor data may be determined by providing the input tensor data to a discretized level reduction model comprising, for example, input layer(s), feature representation layer(s), channel reduction layer(s), level reduction layer(s), and / or output layer(s).

[0169] The computer-implemented method 800 may include, at 808, determining (e.g., by a computing system) to reconstruct input tensor data based at least in part on the discretized level reduction model and at least in part on the level-reduced tensor data. For example, the reconstructed input tensor data may be reconstructed from at least the level-reduced tensor data, such as by (multiple) reconstruction layers of the discretized level reduction model. The reconstructed input tensor data may be similar to the input tensor data. For example, the reconstructed input tensor data may include a greater amount of information than the level-reduced tensor data and / or information in a more easily perceptible manner, including information extrapolated from the level-reduced tensor data to reconstruct the input tensor data. Generally, it is desirable to reconstruct the input tensor data as close to the input tensor data as possible while conforming to the structure of the discretized level reduction model. In this way, the model can learn to provide sufficient spatial information at the level-reduced tensor data to closely reconstruct the input tensor data.

[0170] In some embodiments, such as embodiments in which the discretized level reduction model includes a color bypass network, determining the reconstructed input tensor data can be based at least in part on the level-reduced tensor data and the color bypass reconstruction. For example, in some embodiments, determining the reconstructed input tensor data can include obtaining (e.g., by a computing system) a first reconstructed input tensor data component. The first reconstructed input tensor data component can be obtained from one or more reconstruction layers. The first reconstructed input tensor data component can be based at least in part on the level-reduced tensor data. For example, the first reconstructed input tensor data component can be (e.g., intermediate) reconstructed input tensor data that is generated by a reconstruction layer from the level-reduced tensor data. As an example, the first reconstructed input tensor data component can be a reconstructed image (e.g., a full-color image) that approximates the input image. For example, the reconstructed image can have the same number of channels and / or discretization levels as the input image. According to example aspects of the present disclosure, by including information from the color bypass network, the image can be made to more closely approximate the input image.

[0171] Additionally and / or alternatively, in some embodiments, determining the reconstructed input tensor data may include obtaining (e.g., by a computing system) a second reconstructed input tensor data component. The second reconstructed input tensor data component may be obtained from a color bypass network. For example, in some embodiments, the second reconstructed input tensor data may be a color bypass reconstruction. For example, the second reconstructed input tensor data component may be obtained from a color bypass reconstruction layer that is included in and / or otherwise connected to the color bypass network. The second reconstructed input tensor data component may be based at least in part on the input tensor data. For example, in some embodiments, the second reconstructed input tensor data component may be obtained based at least in part on a color bypass network that is connected (e.g., fully connected, such as by including at least one fully connected hidden unit) to an input layer that includes the input tensor data. In some embodiments, the second reconstructed input tensor data component may be a reconstructed image based on the input image. The second reconstructed input tensor data component may be a reconstructed image that includes less local spatial information than the reconstructed image of the first reconstructed input tensor data component. For example, the second reconstructed input tensor data component may be a hue of the reconstructed image, such as one or more gradients.

[0172] Additionally and / or alternatively, in some embodiments, determining the reconstructed input tensor data may include determining (e.g., by a computing system) the reconstructed input tensor data based at least in part on the first reconstructed input tensor data component and the second reconstructed input data component. For example, in some embodiments, the reconstructed input tensor data may be determined based at least in part on a pixel-by-pixel combination of the first reconstructed input tensor data component and the second reconstructed input data component.

[0173] The computer-implemented method 800 may include, at 810, determining (e.g., by a computing system) a loss based at least in part on the input tensor data and the reconstructed input tensor data. For example, in some embodiments, the loss may be or may include a pixel difference between the input tensor data and the reconstructed input tensor data. For example, the loss may convey the difference between the input tensor data and the reconstructed input data. The loss may include or define one or more gradients, such as gradients with respect to parameters of a discretized level reduction model. In some embodiments, the reconstructed input data may be generated using only the level-reduced tensor data and / or (in some embodiments) color bypass network information, which may intuitively provide the information required for the model to be trained to include in the level-reduced tensor data to reconstruct the input tensor data.

[0174] The computer-implemented method 800 may include, at 812, adjusting (e.g., by a computing system) one or more parameters of a discretized level reduction model based at least in part on the loss. The discretized level reduction model may include one or more parameters (e.g., node and / or link weights, kernel weights, activation values ​​or levels, etc.) of layer(s) (e.g., input layer(s), feature representation layer(s), channel reduction layer(s), level reduction layer(s), output layer(s), reconstruction layer(s), etc.) and / or a color bypass network, and / or other components of the discretized level reduction model. These parameters may be adjusted based on the loss, such as based on the gradient of the loss. For example, the loss (e.g., the gradient of the loss) may be backpropagated through the discretized level reduction model to adjust the parameters of the model, thereby training the model. In some embodiments, the activation value or level of a discretized activation function (e.g., a discretized hyperbolic tangent activation function) may not change during training. For example, because the discretized activation function is defined as a discretized input, it may not be necessary to shift, scale, or otherwise modify the activation function during training. Therefore, during the backpropagation step, the activation levels of the discretized activation function can be ignored, which helps to simplify the training of the model.

[0175] In at least this way, a discretized level reduction model can be trained to produce level-reduced tensor data that includes sufficient information to reconstruct sufficiently accurate reconstructed input tensor data. This can provide level-reduced tensor data that includes a sufficient amount of spatial information, which can translate into, for example, improved visibility and / or usability of images of the level-reduced tensor data, as well as various other uses. Furthermore, the systems and methods described herein can provide for training discretized level reduction models even in situations where it is difficult and / or impossible to generate sufficient amounts of supervised training data. For example, the model can be trained using (e.g., only) readily available images with little or no modification required to the images.

[0176] Intuitively, a machine learning discretized level reduction model can learn to map the colors of a full-color image into different binary or other level-reduced hashes or textures. The model can also intuitively learn a "texture map" that visually reflects their source colors by being similar in the case of similar colors. This behavior is not well-defined and is in fact an unexpected result of configuring the machine learning model in the manner described according to the example aspects of the present disclosure. This behavior can provide for the generation of level-reduced images that can better capture visual or other spatial information, thereby improving the usability of the image.

[0177] The techniques discussed here involve servers, databases, software applications, and other computer-based systems, as well as the actions taken and information sent to and received from these systems. The inherent flexibility of computer-based systems allows for many possible configurations, combinations, and divisions of tasks and functions among components. For example, the processes discussed here can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0178] Although the present subject matter has been described in detail with respect to various specific example implementations thereof, each example is provided by way of explanation, not limitation of the present disclosure. Those skilled in the art, after understanding the foregoing, can easily make changes, modifications, and equivalents to such embodiments. Therefore, the present subject matter disclosure does not exclude such modifications, variations, and / or additions to the present subject matter, which will be apparent to those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment can be used together with another embodiment to produce a further embodiment. Therefore, the present disclosure is intended to cover such changes, variations, and equivalents.

Claims

1. A computer-implemented method for providing level-reduced tensor data with improved information representation, the method comprising: Obtaining input tensor data, the input tensor data comprising image data, audio data, sensor data, or a latent encoding of the image data, audio data, or sensor data; providing the input tensor data as input to a machine learning discretization level reduction model, the machine learning discretization level reduction model configured to receive tensor data comprising a plurality of discretization levels and, in response to receiving the tensor data, generate level-reduced tensor data comprising a reduced number of discretization levels, wherein the machine learning discretization level reduction model comprises: at least one input layer configured to receive the tensor data; and one or more level reduction layers connected to the at least one input layer, the one or more level reduction layers configured to receive an input having a first number of discretization levels and to provide a layer output having a reduced number of discretization levels; wherein each level reduction layer is associated with a respective number of discretization levels, and the discretization levels are reduced at each of the one or more level reduction layers based at least in part on a discretization activation function having a respective number of discretization levels associated with the level reduction layer; obtaining the level-reduced tensor data from the machine learning discretized level-reduction model; Wherein, the machine learning discretization level reduction model is trained using reconstructed input tensor data, and the reconstructed input tensor data is generated using the output of the machine learning discretization level reduction model.

2. The method according to claim 1, wherein The discretization level reduction model further includes at least one feature representation layer configured to map input tensor data from the input layer to feature representations of the input tensor data.

3. The method of claim 1 , wherein the discretization level reduction model further comprises at least one channel reduction layer, wherein the at least one channel reduction layer is configured to reduce input to at least one channel reduction layer input data having a first number of channels to output of at least one channel reduction layer having a reduced number of channels. 4 . The method of claim 1 , wherein the one or more level reduction layers are each configured to reduce a number of discretization levels based at least in part on a scaling factor. The method of claim 4 , wherein the scaling factor is half.

6. The method according to claim 1, wherein The one or more level reduction layers progressively and monotonically reduce the number of discretization levels at each of the one or more level reduction layers.

7. The method of claim 1, wherein the discretized activation function is a discretized hyperbolic tangent function.

8. The method according to claim 1, wherein The machine learning discretization level reduction model includes an output layer configured to provide the level-reduced tensor data.

9. The method according to claim 1, wherein The reduced number of discretization levels of the level-reduced tensor data is two discretization levels.

10. The method according to claim 1, wherein The discretized level reduction model includes one or more reconstruction layers configured to reconstruct the reconstructed input tensor data from the level-reduced tensor data.

11. The method of claim 1 , wherein the discretization level reduction model comprises a color bypass network comprising one or more fully connected hidden units.

12. The method of claim 11, wherein the color bypass network comprises one to ten fully connected hidden units.

13. A computer-implemented method for training a discretized level-reduction model to provide level-reduced tensor data with improved information representation, the computer-implemented method comprising: obtaining, by a computing system comprising one or more computing devices, training data comprising input tensor data, the input tensor data comprising image data, audio data, sensor data, or a latent encoding of the image data, audio data, or sensor data; providing, by the computing system, the training data to a discretization level reduction model, the discretization level reduction model configured to receive tensor data comprising a plurality of discretization levels and, in response to receiving the tensor data, generate level-reduced tensor data comprising a reduced number of discretization levels; determining, by the computing system and based at least in part on the discretized level reduction model, level-reduced tensor data; determining, by the computing system, reconstruction input tensor data based at least in part on the discretized level reduction model and at least in part on the level-reduced tensor data; determining, by the computing system, a loss based at least in part on the input tensor data and the reconstructed input tensor data; as well as One or more parameters of the discretization level reduction model are adjusted by the computing system based at least in part on the loss.

14. The computer-implemented method of claim 13, wherein the loss comprises a pixel-by-pixel difference between the input tensor data and the reconstructed input tensor data.

15. The computer-implemented method of claim 13 or 14, wherein the machine learning discretized level reduction model comprises: at least one input layer configured to receive the input tensor data; one or more level reduction layers connected to the at least one input layer, the one or more level reduction layers configured to reduce a number of discretization levels at each of the one or more level reduction layers; and One or more reconstruction layers are configured to reconstruct input tensor data from the level-reduced tensor data.

16. The computer-implemented method of claim 15, wherein: The discretized level reduction model includes a color bypass network including one or more fully connected hidden units, and wherein determining, by the computing system and based at least in part on the discretized level reduction model, to reconstruct input tensor data based at least in part on the level-reduced tensor data includes: obtaining, by the computing system, a first reconstructed input tensor data component from the one or more reconstruction layers, the first reconstructed input tensor data component being based at least in part on the level-reduced tensor data; obtaining, by the computing system, a second reconstructed input tensor data component from the color bypass network, the second reconstructed input tensor data component being based at least in part on the input tensor data; and The reconstructed input tensor data is determined by the computing system based at least in part on the first reconstructed input tensor data component and the second reconstructed input data component.

17. The computer-implemented method of claim 16, wherein the first reconstruction input tensor data component comprises a reconstructed image, and wherein the second reconstruction input tensor data component comprises a hue of the reconstructed image.

18. One or more non-transitory computer-readable media storing a machine learning discretization level reduction model, the machine learning discretization level reduction model configured to receive tensor data comprising a plurality of discretization levels, the tensor data comprising image data, audio data, sensor data, or a potential encoding of the image data, audio data, or sensor data, and in response to receiving the tensor data, generate level-reduced tensor data comprising a reduced number of discretization levels, wherein the machine learning discretization level reduction model comprises: at least one input layer configured to receive the tensor data; and a plurality of level reduction layers connected to the at least one input layer, the plurality of level reduction layers configured to progressively and monotonically reduce the number of discretization levels at each level reduction layer of the plurality of level reduction layers; wherein the plurality of level reduction layers are configured to receive an input having a first number of discretization levels and provide a layer output having a reduced number of discretization levels; wherein each level reduction layer is associated with a respective number of discretization levels, and the discretization levels are reduced at each of the plurality of level reduction layers based at least in part on a discretization activation function having the respective number of discretization levels associated with the level reduction layer; and Wherein, the machine learning discretization level reduction model is trained using reconstructed input tensor data, and the reconstructed input tensor data is generated using the output of the machine learning discretization level reduction model.

Images