Deep palette prediction

Abstract

Example embodiments allow for training an encoder (e.g., an artificial neural network (ANN)) to generate a color palette based on an input image. The color palette can then be used to generate a quantized, reduced color depth image corresponding to the input image using the input image. The differences between a plurality of such input images and the corresponding quantized images are used to train the encoder. An encoder trained in this manner is particularly suitable for generating color palettes for converting images into different reduced color depth image file formats. Such an encoder also offers advantages in terms of memory usage and computational time or cost relative to median cut algorithms or other methods for generating reduced color depth color palettes for images.

Description

Technical Field

[0001] The present disclosure generally relates to methods, computer-readable media, and systems for generating or predicting color palettes. Background Art

[0002] Different image formats can have different numbers of distinct colors they can support, which can be indicated by how many bits they use to represent each pixel. For example, the Portable Network Graphics (PNG) format can store colors represented by 8, 24, or 48 bits per pixel, while the Graphics Interchange Format (GIF) supports 8 bits per pixel or less (up to 256 colors in the palette). Therefore, when an image file is converted to a format with a smaller color range, a new color palette may be generated for the image. For large images, this can be a slow process and may reduce the perceived quality of the image. Summary of the Invention

[0003] To convert an image file from one format to another, a new color palette may be generated. This is because some image formats limit the number of colors that can be represented in an image in order to reduce the number of bits per pixel used to store the image's color data. For example, the Portable Network Graphics (PNG) format can represent pixel color data using 8, 24, or 48 bits per pixel, while the Graphics Interchange Format (GIF) uses 8 bits per pixel or less to represent color information.

[0004] Such conversion between formats can be a slow process, especially for image files that encode many pixels, have a deep color depth, or are otherwise large in size. In addition, conversion to a format with a lower number of colors or color depth can result in a loss of subjective quality of the image.

[0005] In order to effectively reduce the number of colors represented in an image, it may be beneficial to identify a set of colors for a reduced color palette. The characteristics of each color in such a reduced color palette (e.g., the relative levels of red, blue, and green) can be selected to maintain image quality and / or minimize the degradation of image quality caused by reducing the number of colors in the image. In some examples, the colors in such a reduced color palette can be pre-selected, for example, as defined according to an image compression standard. Alternatively, a reduced color palette for generating an image with a reduced number of colors can be determined based on an input image to improve the quality of the output image. For example, an input image that primarily includes red hues can be represented by a color palette that primarily includes relatively fewer colors than red or near-red colors, blue, green, or other non-red hues, so as to effectively generate an aesthetically pleasing output image (e.g., subjectively pleasing to an average viewer).

[0006] Various methods can be applied to determine a color palette for an input image that will produce a subjectively pleasing or otherwise "good" output image, where pixels are constrained to be represented by colors in the determined color palette. An example of such a method is the median cut algorithm that is part of the GIF standard for compressed images.

[0007] Embodiments described herein provide methods for applying an artificial neural network (ANN), k-means or other machine learning algorithm and / or encoder to generate a color palette to compress an image and / or convert an image from one format to another. The ANN or other algorithm for generating a color palette from an input image can be trained or otherwise developed according to various processes. The present disclosure provides methods for training an encoder, which may include an artificial neural network to generate a predicted color palette based on an input image. The generated color palette contains color values ​​that can then be used to convert the applied image into an image that contains fewer colors than the applied input image. The difference between the reduced color image and the input image can then be used to train the ANN (e.g., by using the difference as an error signal to update the ANN via backpropagation). The subjective quality of the generated image can additionally or alternatively be used to train the ANN.

[0008] Such an updating process can be performed multiple times and / or based on multiple different training images to generate an ANN. Such a trained ANN can then be used (e.g., by a processor of a computer) to determine a color palette for additional images, which can then be used to convert the additional images into an image format different from their original image format.

[0009] The methods disclosed herein can advantageously enable a training set of images to be used to train a color palette-predicting artificial neural network or other algorithm without requiring information about the "ground truth" color palette of any image in the training set. Thus, the images in such a training set can be obtained from any suitable image source, without the need to determine or otherwise obtain a "ground truth" color palette with a reduced number of colors.

[0010] In a first aspect, a method for training an encoder to generate a color palette from an input image includes: (i) obtaining training data, wherein the training data includes one or more images, wherein each of the one or more images has multiple color channels; (ii) applying a first image from the training data to an encoder to generate a first color palette for the first image, wherein the first color palette includes a first number of colors, each color of the first color palette includes a set of values ​​corresponding to a respective color channel of the first image; (iii) quantizing the first image using the first color palette to generate a first quantized image; (iv) comparing the first image to the first quantized image; and (v) updating the encoder based on the comparison. Updating the encoder based on the comparison can include, for example, updating the encoder to reduce a difference between the first image and the first quantized image.

[0011] The above method may include one or more of the following optional features. The first image from the training data may be an RGB image having a red channel, a blue channel, and a green channel. The first image may be modified by data augmentation before being applied to the encoder. The encoder may include one or more convolutional layers and one or more output layers, wherein at least one output layer receives as input an output of at least one convolutional layer, wherein applying the first image to the encoder to generate the first color palette comprises applying the first image to the at least one convolutional layer, wherein a final layer of the one or more output layers comprises a first number of output units, and wherein each output unit of the final layer corresponds to a respective color of the first color palette. One or more such output layers may be referred to as a "hidden layer."

[0012] The method may further include: (i) generating a reduced palette encoder by modifying the updated encoder, wherein modifying the updated encoder includes: (a) discarding at least a final layer from the updated encoder; and (b) replacing the discarded final layer with a reduced palette final layer, wherein the reduced palette final layer includes a second number of output units, wherein the second number of output units is less than the first number of output units; (ii) applying a second image from the training data to the reduced palette encoder to generate a second palette for the second image, wherein the second palette includes a second number of colors, each color of the second palette includes a set of values ​​corresponding to a corresponding color channel of the second image, wherein the second number of colors is less than the first number of colors, and wherein each output unit of the reduced palette final layer corresponds to a corresponding color of the second palette; (iii) quantizing the second image using the second palette to generate a second quantized image; (iv) comparing the second image to the second quantized image; and (v) updating the reduced palette encoder based on the comparison of the second image to the second quantized image. The first number of colors may be greater than 255, and the second number of colors may be less than 65. Updating the reduced palette encoder based on the comparison may include, for example, updating the reduced palette encoder to reduce a difference between the second image and the second quantized image.

[0013] Updating the reduced palette encoder may include updating at least one parameter of the reduced palette final layer. Updating the reduced palette encoder may also include not modifying any parameters of one or more convolutional layers. Not modifying any parameters of one or more convolutional layers may include fixing one or more convolutional layers and / or their parameters, for example, by setting a learning rate parameter of the convolutional layer to zero, by setting a flag to indicate that the convolutional layer may be fixed or otherwise not updated, or by taking some other action.

[0014] Comparing the first image to the first quantized image may include determining a difference between a particular pixel of the first input image and a corresponding pixel of the first quantized image. Comparing the first image to the first quantized image may include applying a convolutional neural network to the first quantized image to generate a subjective quality score; an example of this is using a Neural Image Assessment (NIMA) score. The encoder may be differentiable end-to-end, such that updating the encoder may include performing backpropagation through the encoder. Comparing the first image to the first quantized image may include performing the comparison process as a differentiable end-to-end, such that updating the encoder may include performing backpropagation through the comparison process.

[0015] In another aspect, a method for predicting a color palette of an image includes: (i) obtaining an image; and (ii) applying the image to an encoder to generate at least one color palette for the image, wherein the encoder has been trained according to the method of the previous aspect. The image may be an RGB image file;

[0016] In another aspect, a system includes a controller and a computer readable medium having program instructions stored thereon, the program instructions, when executed by the controller, causing the controller to perform any of the above methods.Such a computer readable medium may be non-transitory.

[0017] In another aspect, a system includes a computer-readable medium having program instructions stored thereon, the program instructions, when executed by a controller, causing the controller to perform any of the above methods. Such a computer-readable medium may be non-transitory and may be incorporated into an article of manufacture.

[0018] These and other aspects, advantages and alternatives will become apparent to those skilled in the art by reading the following detailed description with appropriate reference to the accompanying drawings. In addition, it should be understood that the description provided in this summary section and elsewhere in this document is intended to illustrate the claimed subject matter by way of example and not limitation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 An example method is illustrated.

[0020] Figure 2 is a simplified block diagram illustrating some components of an example computing system.

[0021] Figure 3 It is a flowchart of a method.

[0022] Figure 4 It is a flowchart of a method. DETAILED DESCRIPTION

[0023] Examples of methods and systems are described herein. It should be understood that the words "exemplary," "example," and "illustrative" as used herein mean "serving as an example, instance, or illustration." Any embodiment or feature described herein as "exemplary," "example," or "illustrative" is not necessarily to be construed as preferred or advantageous over other embodiments or features. Furthermore, the exemplary embodiments described herein are not meant to be limiting. It will be readily understood that certain aspects of the disclosed systems and methods may be arranged and combined in a variety of different configurations.

[0024] I. Sample Application and Image Format Conversion

[0025] Generally, imaging can refer to capturing and storing the color and brightness characteristics of a real-world environment or scene in a digital, chemical, or other format (e.g., in a photograph and / or motion video). A variety of image capture devices (e.g., CCDs, photodetector arrays, active pixel sensors) exist to facilitate capturing images in many different formats (e.g., color imaging, black and white imaging).

[0026] Different image formats have different numbers of distinct colors they can support. This may be related to how many bits are used to represent each pixel. For example, the Portable Network Graphics (PNG) format can store colors represented by 8, 24, or 48 bits per pixel, while the Graphics Interchange Format (GIF) only supports 8 bits per pixel. Therefore, if an image is converted from one format to another, the color data can be converted so that it is supported by the new format. The format of an image file may change due to website image embedding, file compression, file transfer, or other reasons. Such conversion can be a slow process, especially for large image files. In addition, the conversion can result in a loss of perceived quality of the image. In various applications, it may be advantageous to accelerate the process of image format conversion while maintaining the perceived quality of the original image. One solution is to develop an artificial neural network (ANN) or other algorithm to generate a predicted color palette from an image. This predicted color palette can then be used to generate an image in a specified file format.

[0027] II. Example Color Palette Prediction

[0028] In order to generate an image from an input image that includes fewer colors than the input image, a reduced color palette can first be obtained. Each pixel of the input image can then be mapped to one of the colors in the palette based on the color of the pixel in the input image. Such a palette can be obtained as a default color palette, for example, the target image format can specify a default color palette that must be used when encoding an image according to the target image format. Alternatively, the color palette can be determined based on the input image. In such an example, the determined color palette can be adapted to the colors appearing in the input image so as to improve the subjective quality of the encoded image while using a specified number of colors in the color palette to represent the image. Various methods can be used to determine the color palette from the input image. One such method is the median cut algorithm.

[0029] The image is converted to GIF image format using the median cut algorithm. The median cut algorithm involves sorting the color values ​​of the pixels of the input image relative to the color channel that exhibits the largest range of values ​​in the pixel. For example, if the blue channel has the largest range in an RGB image, the color values ​​will be sorted first based on the blue value. The sorted pixels are then divided into two groups or "buckets". This sorting and dividing process is then repeated in each "bucket" until there are as many "buckets" as the number of colors required for the palette. For each bucket, the color channel used to sort the pixels in the bucket is selected based on which color channel exhibits the largest range of values ​​in the pixels in the bucket. The color values ​​of each "bucket" are then averaged into a single color value for that bucket. This results in determining that the number of colors of the output color palette is equal to the number of buckets.

[0030] While straightforward, the process can be computationally expensive when used to determine large color palettes; the larger the color palette, the longer the process takes. Furthermore, averaging the colors within a bucket does not always accurately reflect the range of colors within the bucket, resulting in a loss of subjective quality in the output image. Furthermore, the median cut algorithm is iterative and includes multiple sorting and segmentation steps, and thus can require significant and variable memory, computation time / cycles, and other computational resources, especially for large images and / or output color palettes with many colors.

[0031] The embodiments provided herein represent improvements to the median cut algorithm and other methods previously employed for determining a color palette for encoding reduced color depth images. These embodiments employ various machine learning techniques or other algorithms to identify improved color palettes that can correct for discoloration, efficiently predict the color palette of an image within a more constrained time period, or provide other benefits. The embodiments described in the present disclosure avoid the iterative processing steps required by the median cut algorithm (or similar algorithms) and, therefore, allow the color palette to be determined more quickly than using the median cut algorithm. Furthermore, experiments have shown that reduced color depth images created according to the present disclosure are of higher quality than images generated using the median cut algorithm.

[0032] In some examples, an artificial neural network (ANN) can be applied to generate a color palette for an image. Such an ANN can be configured in various ways. For example, the ANN can include two or more layers, can include units with linear, logarithmic, or other specified output functions, can include fully or other connected neurons, can include recurrent and / or feedforward connections between neurons in different layers, can include filters or other elements to process input information and / or information passed between layers, or can be configured in other ways to facilitate generating a predicted color palette based on an input image.

[0033] An ANN for generating a predicted color palette based on an input image may include one or more filters that can be applied to the input image, and the outputs of these filters can then be applied to the inputs of one or more neurons of the ANN. For example, such an ANN can be or can include a convolutional neural network (CNN). Convolutional neural networks are a variety of ANNs that are configured to facilitate ANN-based classification or other processing based on images or other large dimensional inputs that organize their elements in two or more dimensions. The organization of the ANN along these dimensions can be related to some structure in the input structure (for example, the relative position of an image in a two-dimensional space can be related to the similarity between the pixels of the image).

[0034] In an example embodiment, a CNN includes at least one two-dimensional (or higher dimensional) filter applied to an input image; the filtered input image is then applied to neurons of the CNN (e.g., neurons of a convolutional layer of the CNN). The convolution of such a filter and the input image can represent a color value of a pixel or a group of pixels from the input image. A group of neurons of the CNN can receive corresponding inputs determined by applying the same filter to the input image. Additionally or alternatively, a group of neurons of the CNN can be associated with corresponding different filters and can receive corresponding inputs determined by applying the corresponding filters to the input image. Such filters can be trained during the training of the CNN or can be pre-specified. For example, such filters can represent wavelet filters, center-surround filters, biologically inspired filter kernels (e.g., from studies of receptive fields of animal visual processing), or some other pre-specified filter patterns.

[0035] A CNN or other type of ANN can include multiple convolutional layers (e.g., corresponding to different filters and / or features), pooling layers, rectification layers, fully connected layers, or other types of layers. The convolutional layers of a CNN represent the convolution of an input image or some other input (e.g., a filtered, downsampled, or otherwise processed version of the input image) with a filter. The pooling layers of a CNN apply nonlinear downsampling to the higher layers of the CNN, for example, by applying a maximum, average, L2 norm, or other pooling function to a subset of neurons, outputs, or other features of the higher layer(s) of the CNN. The rectification layers of a CNN apply a rectification nonlinear function (e.g., an unsaturated activation function, a sigmoid function) to the outputs of the higher layers. The fully connected layers of a CNN receive inputs from many or all neurons in one or more higher layers of the CNN. The outputs of the neurons of one or more fully connected layers (e.g., the final layer of an ANN or CNN) can be used to determine a region of the input image (e.g., for each pixel of the input image) or to determine a predicted color palette for the image as a whole.

[0036] The neurons in a CNN can be organized according to the corresponding dimensions of the input. For example, where the input is an image (a two-dimensional input, or a three-dimensional input in which the color channels of the image are arranged along a third dimension), the neurons of a CNN (e.g., an input layer of a CNN, a pooling layer of a CNN) can correspond to positions in the two-dimensional input image. The connections between neurons and / or filters in different layers of a CNN may be associated with these positions. For example, a neuron in a convolutional layer of a CNN may receive an input based on a convolution of a filter with a portion of the input image, or with a portion of some other layer of the CNN that is located close to the position of the convolutional layer neurons. In another example, a neuron in a pooling layer of a CNN may receive input from a neuron in a layer higher than the pooling layer (e.g., in a convolutional layer, in a higher pooling layer) whose position is close to the position of the pooling layer neurons.

[0037] The ANN can generate a color palette based on the input image, and the colors of the color palette can be used to represent a relatively larger number of colors present in the input image using fewer colors. Such a color palette can be used to reduce the color depth of the input image, or quantize the input image in other ways, thereby allowing the input image to be represented with fewer bits (e.g., to achieve compression of the input image) or provide some other benefits (e.g., allowing the image to be displayed on a low color depth display). Such a color palette can be used to convert the input image into a specified image format whose color range corresponds to the size of the generated color palette. In some examples, this can include the ANN generating a color palette having a number of colors corresponding to the number of colors allowed by a particular image format, such as a GIF or PNG image format.

[0038] Various methods can be used to apply the generated color palette to convert an input image into a reduced color depth image that includes only colors from the color palette. Such processing can be performed to convert the input image into a format that supports a limited number of colors from the generated color palette. This can be accomplished by selecting a color from the generated color palette for each pixel of the input image to represent the corresponding pixel in the converted image. For example, selecting a color from the color palette for a particular pixel can include determining the distance between the color of the particular pixel and each color in the color palette in a three-dimensional color space. The color from the color palette having the closest distance can then be selected for the particular pixel.

[0039] III. Example Training Method

[0040] An encoder (e.g., part or all of a CNN and / or other type of ANN) or other algorithm for generating a color palette from an image can be trained to improve the quality of the color palette generated thereby. Such training can include iteratively or otherwise updating the encoder based on a set of training images. Such training can include applying particular training images to the encoder, determining an output color palette, generating a quantized version of the input using the color palette, and then determining an error or loss based on the degree of correspondence between the generated quantized image and the input image (e.g., the mean squared difference between the pixel colors of the input image and the corresponding pixel colors of the quantized image). The encoder can then be updated (e.g., by backpropagation) based on the determined degree of correspondence.

[0041] Such "unsupervised" training methods can provide benefits over "supervised" methods that train an encoder based on the correspondence between a generated color palette and a known "true" color palette. In practice, accessing such "true" color palette information can be difficult or expensive (in terms of manpower). Furthermore, where such a "true" color palette is generated by a human, the "true" palette itself may be suboptimal in ways that may be difficult to correct (e.g., biased by a human's subjective perception of the image).

[0042] The method described herein can improve the above disadvantages by facilitating the training of an encoder based on the differences (e.g., pixel differences) between a source image and a quantized image generated therefrom according to a color palette generated from the source image. This allows the encoder to be trained using a large number of different images. Furthermore, the training can cause the encoder to attempt to create a color palette for generating a quantized image that corresponds to the source image, rather than including some deviations within the color space of the color palette.

[0043] As described herein, multiple sets of images or other image information can be acquired for multiple different scenes, perspectives, and / or colors thereof in order to generate multiple images for training the encoder. The multiple sets of images or other image information for training the encoder can be obtained in various ways. In some examples, one or more images of the training set can be enhanced to increase the diversity of images and image types that appear in the set of training images. This can include intentionally adding noise, changing the color balance, introducing simulated scratches, streaks, or other defects, or introducing some other anomaly into one or more training images. This can be done in many ways, such as, causing a pixel to have an incorrect color (e.g., a black colored pixel in a blue sky image), causing multiple pixels to have an incorrect color (e.g., a row of white pixels spanning columns of different colors), out-of-focus or "blurred" images, cropped images (e.g., rows and / or columns of pixels removed from an image), and / or monochrome images (e.g., grayscale images).

[0044] Figure 1 An example flow chart illustrating elements of such an encoder training method is shown.A plurality of images ("training image set") includes example source images ("source images").

[0045] An encoder ("encoder"), such as an ANN, CNN, or some other encoder algorithm, is used to determine a color palette ("color palette") based on a source image. The encoder can be configured in various ways (e.g., as described above) to provide a color palette based on an input image. For example, the encoder can include a convolutional neural network. An algorithm that converts the source image into a quantized image with reduced color depth ("quantizer") using the generated color palette is then used to generate a quantized image ("quantized image") based on the color palette. The quantized image thus represents a version of the source image whose pixels are restricted to the colors represented in the color palette output by the encoder.

[0046] A difference ("error signal") between the quantized image and the source image can be determined and used to update the encoder. The difference so determined can include a difference image determined by, for example, subtracting the values ​​of corresponding pixels between the quantized image and the source image. Updating the encoder can include applying a loss function to the determined difference between the quantized image and the source image, performing backpropagation, determining the gradient of the difference with respect to the encoder coefficients, or updating the encoder according to some other method based on the determined difference. In some examples, the loss function can include one or more items related to the subjective quality of the quantized image. Such a subjective quality value can be provided, for example, by a human who has viewed the quantized image. Additionally or alternatively, a convolutional neural network or other algorithm or model can be applied to the quantized image to generate such a subjective quality score. For example, a neural image assessment (NIMA) can be applied to the quantized image to generate the subjective quality score.

[0047] The above process for updating the encoder may be performed multiple times (e.g., in combination with Figure 1 ) to train the encoder. For example, for a specified number of iterations, or according to some other conditions, these processes can be iteratively performed until the encoder converges (e.g., until the degree of variation of the encoder coefficients in each iteration decreases below a specified absolute or relative level), until a measure of the magnitude of the error signal determined in each iteration (e.g., a sum of squared errors, a corrected sum of errors, a subjective quality score) decreases below a specified absolute or relative level.

[0048] In addition, combined Figure 1The described embodiments are intended to be illustrative example embodiments and not restrictive. For example, the plurality of images used to train the encoder can include multiple sets of images corresponding to respective different objects. Each set of such images can include one or more fully focused images, shallow depth of field images, or other images that can be used to perform one or more elements of the encoder training methods described herein. In such an example, the encoder can be trained so that the color palette of each image in such a set of images is similar.

[0049] The encoder can be trained by a server, cloud computing service, or other computing system according to the techniques described herein. The system for training the encoder can then apply the trained encoder to determine color palettes for more images. For example, the system can receive additional images (e.g., from a mobile phone, digital camera, or other image capture system) and apply the trained encoder to these images to generate color palettes for these images, use these generated color palettes to generate quantized versions of these images with reduced color depth (e.g., as part of a process of encoding images according to GIF, PNG, or some other image format), or perform other operations related to the images and / or color palettes generated therefrom. The system can then transmit the determined color palettes, quantized versions of the images, or other generated information to any device or system that provides the additional image(s).

[0050] Additionally or alternatively, a server or other system may train an encoder as described herein and then transmit an indication of the trained encoder (e.g., an indication of the coefficients, internal structure, or other details of the trained encoder) to some other system. The other system may then be operable to apply the trained encoder to additional images. For example, a server may train an encoder as described herein and then transmit an indication of the trained encoder to a cell phone (e.g., via the Internet and / or a cellular data network). The cell phone may then be operable to apply the trained encoder to an image (e.g., an image captured via the cell phone's camera) to generate a color palette, a quantized version of the image using such a color palette, or other processing using the trained encoder. Transmitting the indication of the trained encoder may include transmitting information to an already running system (e.g., to a cell phone, digital camera, or other device that communicates with the Internet) via wired or wireless components. Additionally or alternatively, transmitting the indication of the trained encoder to a system may occur during assembly and / or programming of the system. For example, transmitting an indication of the trained encoder may include programming a controller or memory (e.g., flash memory or some other non-transitory computer-readable medium) with the trained encoder and subsequently integrating the controller or memory into a device or other system.

[0051] Note that while the images used to train the encoder as described herein may be characterized as "training data" or "training images" used to train the encoder for later application to other images of interest (e.g., images captured by a cell phone), these characterizations are intended as non-limiting examples of the methods described herein. An ANN may be trained using one or more images and then, after having been trained using images, used again to generate a color palette for one or more images.

[0052] IV. Illustrative Systems

[0053] Figure 2 An example computing device 200 that can be used to implement the methods described herein is illustrated. By way of example and not limitation, the computing device 200 can be a cellular mobile phone (e.g., a smartphone), a camera, a camcorder, a fax machine, a computer (e.g., a desktop, laptop, tablet, or handheld computer), a personal digital assistant (PDA), a home automation component, a digital video recorder (DVR), a digital television, a remote control, a wearable computing device, a robot, a drone, an autonomous vehicle, or some other type of device. Such a device can be equipped with an image capture device to generate one or more images, which can then be compressed and / or quantized (e.g., to generate an output image with reduced color depth or otherwise quantized) according to the methods described herein. It should be understood that the computing device 200 can represent a physical camera device, such as a digital camera, a specific physical hardware platform on which an image processing application runs in software, or other combinations of hardware and software configured to perform image processing and / or encoder training functions.

[0054] like Figure 2 As shown, computing device 200 may include a communication interface 202 , a user interface 204 , a processor 206 , and data storage 208 , all of which may be communicatively linked together via a system bus, network, or other connection mechanism 210 .

[0055] The communication interface 202 may function to allow the computing device 200 to communicate with other devices, access networks, and / or transport networks using analog or digital modulation of electrical, magnetic, electromagnetic, optical, or other signals. Thus, the communication interface 202 may facilitate circuit-switched and / or packet-switched communications, such as plain old telephone service (POTS) communications and / or Internet Protocol (IP) or other packetized communications. For example, the communication interface 202 may include a chipset and antenna arranged to wirelessly communicate with a radio access network or access point. Furthermore, the communication interface 202 may take the form of or include a wired interface, such as an Ethernet, Universal Serial Bus (USB), or High Definition Multimedia Interface (HDMI) port. The communication interface 202 may also take the form of or include a wireless interface, such as Wi-Fi, Global Positioning System (GPS) or wide area wireless interface (e.g., WiMAX or 3GPP Long Term Evolution (LTE)). However, other forms of physical layer interfaces and other types of standard or proprietary communication protocols may be used on the communication interface 202. In addition, the communication interface 202 may include multiple physical communication interfaces (e.g., Wifi interface, interface and wide area wireless interface).

[0056] In some embodiments, the communication interface 202 can function to allow the computing device 200 to communicate with other devices, remote servers, access networks, and / or transport networks. For example, the communication interface 202 can function to access a trained encoder via communication with a remote server or other remote device or system to allow the computing device 200 to use the trained encoder to generate a color palette based on a captured or otherwise obtained image and / or to quantize such an image using such a color palette. For example, the computing system 200 can be a mobile phone, a digital camera, or other image capture device, and the remote system can be a server containing a memory that contains such a trained encoder.

[0057] The user interface 204 can work to allow computing device 200 to interact with the user, for example, receives input from the user and / or provides output to the user.Therefore, the user interface 204 can include an input component, such as a keypad, a keyboard, a touch-sensitive panel or there is a sensitive panel, a computer mouse, a trackball, a joystick, a microphone or the like. The user interface 204 can also include one or more output components, such as a display screen, which for example can be combined with the presence of a sensitive panel. The display screen can be based on CRT, LCD and / or LED technology, or other technologies known or developed later. The user interface 204 can also be configured to generate (a plurality of) audible output via loudspeaker, speaker jack, audio output port, audio output device, earphones and / or other similar devices.

[0058] The processor 206 may include one or more general-purpose processors (e.g., microprocessors) and / or one or more special-purpose processors (e.g., digital signal processors (DSPs), graphics processing units (GPUs), floating-point units (FPUs), network processors, tensor processing units (TPUs), or application-specific integrated circuits (ASICs). In some cases, the special-purpose processors may be capable of image processing, image alignment, and merging images, among other applications or functions. The data storage device 208 may include one or more volatile and / or non-volatile storage components, such as magnetic, optical, flash, or organic memory, and may be integrated with the processor 206 in whole or in part. The data storage device 208 may include removable and / or non-removable components.

[0059] The processor 206 may be capable of executing program instructions 218 (e.g., compiled or non-compiled program logic and / or machine code) stored in the data storage 208 to perform the various functions described herein. Thus, the data storage 208 may include a non-transitory computer-readable medium having program instructions stored thereon that, when executed by the computing device 200, cause the computing device 200 to perform any of the methods, processes, or functions disclosed in this specification and / or the accompanying figures. The execution of the program instructions 218 by the processor 206 may cause the processor 206 to use the data 212.

[0060] By way of example, program instructions 218 may include an operating system 222 (e.g., an operating system kernel, device driver(s), and / or other modules) and one or more applications 220 (e.g., camera functionality, encoder and / or ANN training, image-based color palette generation) installed on computing device 200. Data 212 may include training images 214 and / or one or more trained encoders 216. Training images 214 may be used to train an ANN or other encoder or an algorithm containing an encoder, as described herein (e.g., to generate and / or update trained encoders 216). Trained encoders 216 may be applied to generate a color palette based on an input image (e.g., an image captured using a camera component of device 200 and / or accessed via communication interface 202).

[0061] The application 220 can communicate with the operating system 222 through one or more application programming interfaces (APIs). These APIs can facilitate, for example, the application 220 reading and / or writing the trained encoder 216, transmitting or receiving information via the communication interface 202, receiving and / or displaying information on the user interface 204, capturing images using the camera component 224, and the like.

[0062] Applications 220 may take the form of “apps” that can be downloaded to computing device 200 through one or more online application stores or application marketplaces (e.g., via communication interface 202). However, applications may also be installed on computing device 200 in other ways, such as via a web browser or through a physical interface of computing device 200 (e.g., a USB port).

[0063] V. Example Method

[0064] Figure 3 3 is a flow chart of a method 300 for training an encoder to generate a color palette from an input image. The method 300 includes obtaining, by a system, training data comprising one or more images (310), wherein each of the one or more images comprises a plurality of color channels. The method 300 further includes applying a first image from the training data to the encoder to generate a first color palette for the first image (320). The first color palette comprises a first number of colors, and each color of the first color palette comprises a set of values ​​corresponding to a respective color channel of the first image. The method 300 further includes quantizing the first image using the first color palette to generate a first quantized image (330). The method 300 further includes comparing the first image to the first quantized image (340). The method 300 further includes updating the encoder based on the comparison result (350).

[0065] These embodiments may include additional elements or features.In some embodiments, the first image is an RGB image having a red channel, a blue channel, and a green channel.

[0066] In some embodiments, the first image is modified by data augmentation before being applied to the encoder. For example, the first image can be enhanced by introducing anomalies into the image, such as, but not limited to, adding noise, changing the color balance, and applying simulated scratches or other simulated defects. In some examples, the first image can be enhanced by applying out-of-focus objects and / or by changing the blur or focus of portions of the image, adding glare, and creating corrupted pixels.

[0067] In some embodiments, the encoder includes one or more convolutional layers and one or more output layers. The at least one output layer receives as input the output of the at least one convolutional layer. Alternatively, one or more such output layers may be referred to as "hidden layers." Applying the first image to the encoder to generate the first color palette includes applying the first image to the at least one convolutional layer. A final layer of the one or more output layers includes a first number of output units, and each output unit of the final layer corresponds to a respective color of the first color palette.

[0068] In some embodiments, the method further comprises generating a reduced palette encoder by modifying the updated encoder. Modifying the updated encoder comprises discarding at least a final layer from the updated encoder and replacing the discarded final layer with a final layer of the reduced palette. The reduced palette final layer comprises a second number of output units, wherein the second number of output units is less than the first number of output units. Modifying the updated encoder further comprises applying a second image from the training data to the reduced palette encoder to generate a second palette for the second image, wherein the second palette comprises a second number of colors, each color of the second palette comprises a set of values ​​corresponding to a corresponding color channel of the second image, wherein the second number of colors is less than the first number of colors, and wherein each output unit of the reduced palette final layer corresponds to a corresponding color of the second palette. Modifying the updated encoder further comprises quantizing the second image using the second palette to generate a second quantized image. Modifying the updated encoder further comprises comparing the second image to the second quantized image. Modifying the encoder further comprises updating the reduced palette encoder based on the comparison of the second image to the second quantized image.

[0069] In some embodiments, the first number of colors is greater than 255 and the second number of colors is less than 65. In some embodiments, the first number of colors is greater than 1023 and the second number of colors is less than 255. In some embodiments, the first number of colors is greater than 127 and the second number of colors is less than 32.

[0070] In some embodiments, updating the reduced palette encoder comprises updating at least one parameter of the reduced palette final layer.

[0071] In some embodiments, updating the reduced palette encoder includes not modifying any parameters of one or more convolutional layers. That is, updating the reduced palette encoder may include causing the parameters of the convolutional layers to be unchanged, and only changing the parameters, number of nodes, connection structure, or other properties of the reduced palette final layer. This may include taking some proactive action so as not to modify any parameters of one or more convolutional layers, for example, by setting a learning rate parameter of a convolutional layer to zero, by setting a flag to indicate that a convolutional layer may be fixed or otherwise not updated, or by fixing one or more convolutional layers and / or their parameters by taking some other action to prevent the parameters of one or more convolutional layers from being modified when being updated to the reduced palette encoder.

[0072] In some embodiments, comparing the first image and the first quantized image includes determining a difference between a particular pixel of the first input image and a corresponding pixel of the first quantized image. For example, the sum of squared differences between each pair of pixels in the first image and the first quantized image may be determined.

[0073] In some embodiments, comparing the first image to the first quantized image includes applying a convolutional neural network to the first quantized image to generate a subjective quality score. This may include using a neural image assessment (NIMA) deep convolutional neural network to determine a subjective quality score that predicts whether a typical human viewer of the first quantized image would find the first quantized image subjectively pleasing.

[0074] In some embodiments, the encoder is differential end-to-end and updating the encoder includes performing backpropagation through the encoder. For example, a calculated difference between a pixel of the first image and a corresponding pixel of the first quantized image can be backpropagated through the encoder and used to update one or more parameters of the encoder, possibly in combination with one or more other calculated differences that have been backpropagated through the encoder. This is an improvement over other quantization methods (e.g., the median cut algorithm) that may be iterative or may have other properties that prevent end-to-end differentiation. Therefore, it may be difficult or impossible to train such other algorithms via efficient, computationally tractable methods such as backpropagation or other training algorithms that rely on end-to-end differentiability of encoder elements or other algorithms to be trained. Furthermore, because the encoder is end-to-end differentiable, it can be easily connected to other learning systems, and in particular, it can be included as a modular component in any differentiable learning system.

[0075] In some embodiments, comparing the first image to the first quantized image includes performing a comparison process that is end-to-end differentiable, and updating the encoder includes performing backpropagation through the comparison process.

[0076] Figure 4 4 is a flow chart of a method 400 for predicting a color palette of an image. The method 400 includes obtaining an image (410). The method 400 further includes applying the image to an encoder trained according to the above method 300 to generate a color palette of the image (420).

[0077] These embodiments may include additional elements or features. In some embodiments, the image is an RGB image file. In some embodiments, method 400 includes determining an image with reduced color depth or an image quantized in another manner from the obtained image using the generated palette. The quantization process can be performed based on the image format, such as the GIF image format.

[0078] VI. Conclusion

[0079] The above detailed description describes the various features and functions of the disclosed systems, devices, and methods with reference to the accompanying drawings. In the drawings, similar symbols generally identify similar components unless the context dictates otherwise. The illustrative embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the scope of the subject matter presented herein. It will be readily understood that the various aspects of the present disclosure, as generally described herein and illustrated in the accompanying drawings, may be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are expressly contemplated herein.

[0080] With respect to any or all of the message flow diagrams, scenarios, and process diagrams in the accompanying drawings, and as discussed herein, each step, block, and / or communication may represent information processing and / or information transmission in accordance with example embodiments. Alternative embodiments are included within the scope of these example embodiments. In these alternative embodiments, for example, the functions described as steps, blocks, transmissions, communications, requests, responses, and / or messages may be performed out of the order shown or discussed, including substantially simultaneously or in reverse order, depending on the functionality involved. In addition, more or fewer steps, blocks, and / or functions may be used with any of the message flow diagrams, scenarios, and process diagrams discussed herein. Figure 1 The message flow diagrams, scenarios and flowcharts can be used together, and these message flow diagrams, scenarios and flowcharts can be combined with each other in part or in whole.

[0081] The steps or blocks representing information processing may correspond to circuits that can be configured to perform the specific logical functions of the methods or techniques described herein. Alternatively or additionally, the steps or blocks representing information processing may correspond to modules, segments, or portions of program code (including associated data). The program code may include one or more instructions executable by a processor for implementing the specific logical functions or actions in the method or technique. The program code and / or associated data may be stored on any type of computer-readable medium, such as a storage device, including a disk drive, hard drive, or other storage medium.

[0082] Computer-readable media may also include non-transitory computer-readable media, such as computer-readable media that store data for a short period of time, such as register memory, processor cache, and / or random access memory (RAM). Computer-readable media may also include non-transitory computer-readable media that store program code and / or data for a longer period of time, such as secondary or permanent long-term memory, such as read-only memory (ROM), optical or magnetic disks, and / or compact disk read-only memory (CD-ROM). Computer-readable media may also be any other volatile or non-volatile storage system. For example, a computer-readable medium may be considered to be a computer-readable storage medium, or a tangible storage device.

[0083] In addition, steps or blocks representing one or more information transfers may correspond to information transfers between software and / or hardware modules in the same physical device. However, other information transfers may be performed between software and / or hardware modules in different physical devices.

[0084] Although various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for illustrative purposes only and are not intended to be limiting, with the true scope being indicated by the appended claims.

[0085] Although exemplary embodiments include throughout the use of an ANN to generate a color palette based on an input image, it is foreseeable that alternative algorithms may be applied to generate a color palette based on an image. For example, according to the methods described herein, kernel methods, Bayesian inference algorithms, reinforcement learning algorithms, cellular automata, decision trees, boosted decision trees, or other alternative algorithms and / or combinations thereof may be applied to generate a color palette based on an input image. Accordingly, the methods described herein for training an ANN may be applied to training such an alternative algorithm, for example, by providing error information (e.g., the difference between an input image and an image generated by applying a generated color palette with reduced color depth thereto) that may be applied to update such an alternative algorithm (e.g., using backpropagation, reinforcement learning, or some other technique appropriate to the selected alternative algorithm).

Claims

1. A method for training an encoder to generate a color palette from an input image, the method comprising: Get training data, where The training data includes one or more images, wherein each of the one or more images has a plurality of color channels; applying a first image from the training data to an encoder to generate a first color palette for the first image, wherein the first color palette includes a first number of colors, each color of the first color palette includes a set of values ​​corresponding to a corresponding color channel of the first image, wherein the encoder includes one or more output layers, wherein a final layer of the one or more output layers includes a first number of output units, and wherein each output unit of the final layer corresponds to a corresponding color of the first color palette; quantizing the first image using the first color palette to generate a first quantized image; comparing the first image to the first quantized image; updating the encoder based on the comparison; and The reduced palette encoder is generated by modifying the updated encoder, Among them, modifications to the updated encoder include: discarding at least the final layer from the updated encoder; and The discarded final layer is replaced with a reduced palette final layer, wherein the reduced palette final layer includes a second number of output units, wherein the second number of output units is less than the first number of output units.

2. The method according to claim 1, wherein The first image is an RGB image having a red channel, a blue channel, and a green channel.

3. The method according to claim 1 or 2, wherein: The first image is modified by data augmentation before being applied to the encoder.

4. The method according to claim 1 or 2, wherein The encoder also includes one or more convolutional layers, wherein at least one output layer receives as input an output of the at least one convolutional layer, wherein applying the first image to the encoder to generate the first color palette comprises applying the first image to the at least one convolutional layer.

5. The method according to claim 4, further comprising: Applying a second image from the training data to a reduced palette encoder to generate a second palette for the second image, wherein a second palette comprising a second number of colors, each color of the second palette comprising a set of values ​​corresponding to a respective color channel of the second image, wherein the second number of colors is less than the first number of colors, and wherein each output cell of the reduced palette final layer corresponds to a respective color of the second palette; quantizing the second image using the second color palette to generate a second quantized image; comparing the second image to the second quantized image; and The reduced palette encoder is updated based on a comparison of the second image and the second quantized image.

6. The method according to claim 5, wherein: The first number of colors is greater than 255, and wherein the second number of colors is less than 65.

7. The method according to claim 5, wherein: Updating the reduced palette encoder includes updating at least one parameter of the reduced palette final layer.

8. The method according to claim 5, wherein During the update of the reduced palette encoder, the parameters of one or more convolutional layers are fixed.

9. The method according to any one of claims 1, 2, 5-8, wherein Comparing the first image to the first quantized image includes determining a difference between a particular pixel of the first input image and a corresponding pixel of the first quantized image.

10. The method according to any one of claims 1, 2, 5-8, wherein Comparing the first image to the first quantized image includes applying a convolutional neural network to the first quantized image to generate a subjective quality score.

11. The method according to claim 10, wherein: Applying a convolutional neural network to the first quantized image to generate a subjective quality score includes using Neural Image Assessment (NIMA).

12. The method according to any one of claims 1, 2, 5-8, and 11, wherein: The encoder is differentiable end-to-end, and wherein updating the encoder comprises performing backpropagation through the encoder.

13. The method according to claim 12, wherein: Comparing the first image to the first quantized image includes performing the comparison process as a differentiable end-to-end process, and wherein updating the encoder includes performing backpropagation through the comparison process.

14. A method for predicting a color palette of an image, the method comprising: Obtain images; Applying the image to an encoder to generate at least one color palette for the image, wherein The encoder has been trained according to the method of any one of claims 1-13.

15. The method according to claim 14, wherein Image is an RGB image file.

16. The method of claim 14 or 15, further comprising converting the image into a reduced color depth image by: For each pixel of the image, selecting a color from the generated palette to represent the corresponding pixel in the reduced color depth image; and Each color selected is stored in association with a corresponding pixel.

17. A computer-readable medium having program instructions stored thereon, the program instructions, when executed by a computing device, causing the computing device to perform the method of any one of claims 1 to 16.

18. A system for generating a color palette, comprising: Controller; and A non-transitory computer readable medium having program instructions stored thereon, the program instructions, when executed by a controller, causing the controller to perform the method of any one of claims 1-16.

Images