Method for encoding, encoder, and image compression and storage system

Abstract

The present invention provides a method for encoding, an encoder, and an image compression and storage system. The method includes: receiving an image frame including pixels, wherein each pixel includes n color channels; generating a first sample group of quantized indexes based on the color value of the residual of a first specific pixel in the image frame by a separate color channel quantization branch configured to ensure non-negative reconstruction error for each color channel in the n color channels, wherein the first sample group corresponds to a first n-tuple; and generating entropy coding for the first specific pixel by an entropy encoder based on data mapping the n-tuple of the quantized index to an entropy coding index, wherein the first n-tuple is mapped to a first entropy coding index in the entropy coding index.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 151,344, filed on February 19, 2021, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present application relates to image processing systems and methods. More specifically, the present application relates to systems and methods for joint color channel entropy coding with positive reconstruction error. Background Art

[0004] Flat panel displays such as liquid crystal displays (LCDs) and light emitting diode (LED) displays are becoming very popular due to their small size, excellent image quality, and lower power consumption.

[0005] The pixels in a display panel have a specific response time. The data provided to the pixel for display may change faster than the pixel's ability to respond. As a result, undesirable effects such as motion blur or ghosting effects may occur. In order to make the displayed image more aesthetically pleasing, image compensation techniques are needed. Overdrive algorithms have been developed that can compensate image data to reduce motion blur exhibited by display pixels.

[0006] The above information disclosed in this Background section is only for enhancement of understanding of the present disclosure and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0007] According to an embodiment, a method for encoding is described. The method may include: receiving an image frame including pixels, wherein each pixel includes n color channels; generating a first sample group of quantized indexes based on the color value of the residual of a first specific pixel in the image frame by a separate color channel quantization branch configured to ensure non-negative reconstruction error for each color channel in the n color channels, wherein the first sample group corresponds to a first n-tuple; and generating entropy encoding for the first specific pixel by an entropy encoder based on data mapping the n-tuple of quantized indexes to entropy encoding indexes, wherein the first n-tuple is mapped to a first entropy encoding index in the entropy encoding indexes.

[0008] The generation of the first sample group of quantized indices may include calculating, by each color channel quantization branch, a residual by subtracting a prediction based on previously reconstructed pixels from the received image frame.

[0009] The generation of the first sample group of the quantized index may further include: quantizing, by a quantizer corresponding to each color channel quantization branch, the calculated residual corresponding to the same color channel quantization branch to generate the quantized index.

[0010] Each n-tuple of quantized indices may be associated with an occurrence frequency of the corresponding n-tuple in a set of quantized training images, and each n-tuple may include quantized indices of n color channels, where n is an integer greater than 1.

[0011] The method may further include: generating, by a separate color channel quantization branch, a second sample group of quantized indexes based on the color value of the residual of a second specific pixel in the image frame, wherein the second sample group corresponds to a second n-tuple; and generating, by an entropy encoder, entropy coding for the second specific pixel based on the data, wherein the second n-tuple is mapped to a second entropy coding index in the entropy coding index.

[0012] The second specific pixel may be a neighboring pixel relative to the first specific pixel in the image frame.

[0013] Entropy coding may include Exponential-Golomb coding, Golomb-Rice coding, Huffman coding, or a combination of Exponential-Golomb coding and Golomb-Rice coding.

[0014] Data can be stored as arrays.

[0015] The generation of the entropy coding may include: calculating a first mode value based on the first n-tuple; searching the calculated first mode value in an array including data to determine a corresponding first entropy coding index; and encoding the first entropy coding index.

[0016] The generation of the entropy coding may include: calculating a first mode value based on the first n-tuple; obtaining a first entropy coding index from the array according to a position of the array indicated by the first mode value; and encoding the first entropy coding index.

[0017] Data mapping an n-tuple of quantized indices to an entropy coding index may be generated by the following steps: receiving a first training image frame; calculating a training reconstruction error value corresponding to the first training image frame; generating a quantized index based on the calculated training reconstruction error value, the training reconstruction error value including an n-tuple of quantized indices; identifying an occurrence frequency of the n-tuple of quantized indices; and assigning an entropy coding index to each n-tuple of the n-tuple of quantized indices based on the occurrence frequency of the n-tuple of quantized indices.

[0018] The method may further include: sorting the n-tuples of quantized indices according to occurrence frequency.

[0019] Data mapping the n-tuple of quantized indices to entropy coded indices may be generated during an offline process.

[0020] According to another embodiment, an encoder is described. The encoder may include: n separate color channel quantization branches configured to generate n-tuples of quantized indices, each color channel quantization branch corresponding to one of the n color channels of a pixel of an image frame; an entropy encoder configured to receive the n-tuples of quantized indices and to entropy encode the n-tuples of quantized indices based on data mapping the n-tuples of quantized indices to entropy encoding indices, wherein each of the n separate color channel quantization branches is configured to ensure a non-negative reconstruction error, wherein n is an integer greater than 1.

[0021] Each n-tuple of the quantized index may be associated with a frequency of occurrence of the corresponding n-tuple in a set of quantized training images.

[0022] Each n-tuple may include quantized indices of n color channels.

[0023] According to another embodiment, an image compression and storage system is described. The system may include: a memory storing computer executable instructions; and a processor configured to execute instructions that cause the system to perform operations including the following steps: receiving an image frame including pixels, wherein each pixel includes n color channels; generating a first sample group of quantized indexes based on the color value of the residual of a first specific pixel in the image frame by a separate color channel quantization branch configured to ensure non-negative reconstruction error for each color channel in the n color channels, wherein the first sample group corresponds to a first n-tuple; and generating an entropy code for the first specific pixel by an entropy encoder based on data mapping the n-tuple of quantized indexes to entropy coding indexes, wherein the first n-tuple is mapped to a first entropy coding index in the entropy coding indexes, wherein each n-tuple of the quantized indexes is associated with the frequency of occurrence of the corresponding n-tuple in a set of quantized training images, and wherein each n-tuple includes quantized indexes of n color channels, wherein n is an integer greater than 1.

[0024] The instructions may further cause the system to perform operations including the following steps: generating, by a separate color channel quantization branch, a second sample group of quantized indexes based on a color value of a residual of a second specific pixel in the image frame, wherein the second sample group corresponds to a second n-tuple; and generating, by an entropy encoder, entropy coding for the second specific pixel based on the data, wherein the second n-tuple is mapped to a second entropy coding index in the entropy coding index.

[0025] The scope of the present invention is defined by the claims incorporated into this section by reference. Those skilled in the art will more fully understand the embodiments of the present invention and realize the additional advantages of the present invention by considering the following detailed description of one or more embodiments. Reference will be made to the accompanying drawings which will first be briefly described. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a schematic diagram of a frame compensation system utilizing an image compression and storage system according to an embodiment of the present disclosure.

[0027] Figure 2 is a schematic diagram of an encoder of an image compression and storage system according to an embodiment of the present disclosure.

[0028] Figure 3 is a schematic diagram of a decoder of an image compression and storage system according to an embodiment of the present disclosure.

[0029] Figure 4A is a schematic diagram showing a portion of an encoder together with a portion of a decoder according to an embodiment of the present disclosure.

[0030] Figure 4B According to the embodiment of the present disclosure Figure 4A Simplified diagram of , which illustrates the identity operation of the entropy encoder and entropy decoder.

[0031] Figure 5A Quantization tables utilized by a quantizer and an inverse quantizer according to an embodiment of the present disclosure are shown.

[0032] Figure 5B The operation of a quantizer and an inverse quantizer according to examples of the present disclosure is shown.

[0033] Figure 5C and Figure 5D A uniform quantization table and a non-uniform quantization table utilized by a quantizer according to an embodiment of the present disclosure are respectively shown.

[0034] Figure 6 is a schematic diagram of a color transform positive error reconstruction encoder of an image compression and storage system according to an embodiment of the present disclosure.

[0035] Figure 7 is a schematic diagram of a joint color channel entropy coding positive error reconstruction encoder of an image compression and storage system according to an embodiment of the present disclosure.

[0036] Figure 8 is an example of an unsorted histogram and a sorted histogram showing the frequency of occurrence of a specific pattern in an image frame residual according to an embodiment of the present disclosure.

[0037] Fig. 9is a table showing numerical examples of triplet color values ​​and their corresponding pattern values ​​according to an embodiment of the present disclosure.

[0038] Fig.10 is a table showing a numerical example of a lookup table according to an embodiment of the present disclosure.

[0039] Fig.11 is a table showing a numerical example of another lookup table according to an embodiment of the present disclosure.

[0040] Fig.12 is a table showing a numerical example of a codebook stored in a lookup table according to an embodiment of the present disclosure.

[0041] Fig.13 : is a table showing a numerical example of another codebook stored in a lookup table for pixels having an RGBG format according to an embodiment of the present disclosure.

[0042] Figures 14 to 16 is a flow chart illustrating an example method for performing joint color channel entropy encoding according to various embodiments of the present disclosure.

[0043] Fig.17 is a flow chart illustrating an example method for training an encoder according to an embodiment of the present disclosure.

[0044] Fig.18 is a flow chart illustrating an example method for decoding an entropy coded index according to various embodiments of the present disclosure.

[0045] The embodiments of the present disclosure and their advantages can be best understood by referring to the following detailed description. Unless otherwise specified, the same reference numerals represent the same elements throughout the drawings and written description, and therefore, their description will not be repeated. In the drawings, the relative sizes of elements, layers and regions may be exaggerated for clarity. DETAILED DESCRIPTION

[0046] Hereinafter, example embodiments will be described in more detail with reference to the accompanying drawings. However, the present invention may be implemented in various different forms and should not be construed as being limited to the embodiments shown herein. On the contrary, these embodiments are provided as examples so that this disclosure will be thorough and complete, and will fully convey the aspects and features of the present invention to those skilled in the art. Accordingly, processes, elements, and techniques that are not necessary for a person of ordinary skill in the art to fully understand the aspects and features of the present invention may not be described.

[0047] Figure 1 is a schematic diagram of a frame compensation system 100 utilizing an image compression and storage system 110 according to an embodiment of the present disclosure.

[0048] In some embodiments, the image compression and storage system 110 includes an encoder 200 for encoding (e.g., compressing) data to reduce its size, a storage medium 140 for storing the encoded data, and a decoder 160 for decoding (e.g., decompressing) the stored data to reconstruct the data. The storage of certain data such as 4K image frames consumes a large amount of storage space. Here, the encoder 200 and the decoder 160 allow the use of a smaller storage medium 140, which can result in space and cost savings.

[0049] like Figure 1 As shown in , in some examples, the image compression and storage system 110 can form part of a frame compensation system 100 in which the overdrive system 120 uses a previous frame to compensate for a current frame (e.g., to reduce ghosting effects or blurring). In some examples, the truncation and delay circuit 130 receives a frame (e.g., an 8-bit per color (8bpc) frame), truncates the frame (i.e., reduces the number of bpc of the pixel values ​​of the frame to, for example, 3bpc) and delays the frame to produce a truncated previous frame. The image compression and storage system 110 then compresses the previous frame and stores it for later consumption by the overdrive system 120. Here, the truncation and compression can make the stored frame significantly smaller in size than the original (previous) frame. For example, in the case of an RGB input frame in which each color value is represented by 8 bits (or 24 bits per pixel (24bpp)), truncation can result in a 3bpc / 9bpp frame representation, and the 1.5:1 compression performed by the encoder 200 can further reduce the frame representation to 2bpc / 6bpp. However, embodiments of the present disclosure are not limited to RGB input frames, and may use RGBG (for The encoder 200 may be used to decode any suitable input frame format (sub-pixel arrangement). For example, in the case of an RGBG input frame, truncation may reduce the frame size from 16bpp to 6bpp, and the 1.5:1 compression performed by the encoder 200 may further reduce the frame size to 4bpp. Furthermore, embodiments of the present disclosure are not limited to the use of the truncation operation described above, and in some examples, the truncation and delay circuit 130 may be replaced with a delay block and the truncation operation may be omitted from the process described herein.

[0050] If the reconstruction error (i.e., the error introduced by the image compression and storage system 110) is only non-negative values ​​(i.e., zero or positive values), the performance of the overdrive system 120 can be improved. The encoding schemes of the related art usually optimize the mean square error (MSE) or the peak signal-to-noise ratio (PSNR), which means that the polarity of the error in the reconstructed image can be both positive and negative. According to some embodiments, by ensuring that the reconstruction error (which is defined as the difference between the reconstructed frame and the original frame) (mainly introduced due to quantization) is always a non-negative value (i.e., a value greater than or equal to zero) and is also limited to a set maximum value, the image compression and storage system 110 improves the performance of the overdrive system 120 relative to the related art.

[0051] Throughout this disclosure, a frame refers to data of an image frame including a plurality of pixels. Each pixel of the frame can be represented by a plurality of color values ​​(eg, red, green, and blue).

[0052] Figure 2 is a schematic diagram of an encoder 200 of an image compression and storage system 110 according to an embodiment of the present disclosure. According to an embodiment, an original sample X is provided to a subtractor 220. A previously reconstructed sample is also provided to the subtractor 220, in which the difference between the original sample and the previously reconstructed sample is obtained. The resulting error R is provided to the input of a modulo adder 230 to add a bias value. The output (e.g., biased residual) e from the modulo adder 230 is then provided to the input of a quantizer 240 for quantization to generate a quantized index. The quantized index is provided to an entropy encoder 250 for encoding, and the quantized index is also provided to an inverse quantizer 260 to generate a dequantized biased residual e'. The dequantized biased residual e' is provided to an adder 270 where it is combined with the previous reconstructed sample and the resulting sum is provided to a modular subtractor 280 to subtract the bias value added by the modular adder 230. The output from the modular subtractor 280 is the reconstructed sample X', and the output from the modular subtractor 280 is also provided to the sample predictor 210. Further details of each of these blocks will be described in more detail below.

[0053] Reference Figure 2According to some embodiments, the encoder 200 utilizes a predictive coding scheme in which the residual of the original sample (e.g., the truncated frame) is encoded instead of the original sample itself. In some embodiments, the encoder 200 is configured to encode all color components of the residual (e.g., red color component, green color component, and blue color component in RGB format) at the same time. The residual (also referred to as error R) is defined as the difference between the original sample and the prediction based on one or more adjacent samples. In some embodiments, the sample predictor 210 generates a predicted sample based on one or more of the previous samples that have been encoded, and the subtractor 220 calculates the residual R based on the original sample and the predicted sample. In one example, the predicted sample is set to be the same as the previous reconstructed sample in the same row (i.e., in the horizontal direction). That is, if (i, j) is the current sample position, (i-1, j) is the previous sample position. The calculated residual R can be a positive or negative value. The modulo adder 230 adds a bias value to the residual R to generate a biased residual e to reduce the signaling cost of 1-bit symbol information. The quantizer 240 quantizes the biased residual e to generate a quantized biased residual and passes the corresponding quantized index to the entropy encoder 250 (eg, Huffman encoder, Exponential-Golomb entropy encoder). The compressed data (or encoded value) is stored in the storage medium 140.

[0054] In some embodiments, to generate a prediction sample, the inverse quantizer 260 performs an inverse operation of the operation performed by the quantizer 240 on the quantized biased residual to generate a dequantized biased residual e', the adder 270 adds the prediction sample to the dequantized biased residual e' to generate a reconstructed biased sample, and the modulo subtractor 280 subtracts the bias value added by the modulo adder 230 to generate a reconstructed sample X'. In some examples, a clipping function may be applied to the reconstructed sample to ensure that the reconstructed sample is within a desired dynamic range, which may be expressed as [0,2 bitDepth -1]. The sample predictor 210 uses the reconstructed sample X' (eg, the cropped reconstructed sample) to predict the next sample.

[0055] In some embodiments, if the error in the residual space is maintained as a non-negative value, the error in the signal space will also be non-negative. Therefore, the modular adder 230 and the quantizer 240 are configured so that the error in the residual space remains positive and bounded, which ensures that this is also the case in the signal space. In some embodiments, the quantizer 240 and the inverse quantizer 260 ensure positive bounded errors by appropriately adjusting the quantization band size and the reconstruction points in each band. Accordingly, the image compression and storage system 110 ensures that the reconstruction error (i.e., the quantization error) is always non-negative.

[0056] Reference Figure 3 In some embodiments, the decoder 160 includes an entropy decoder 510 that decompresses the stored samples received from the storage medium 140, and the same inverse quantizer 260, adder 270, modular subtractor 280, and sample predictor 210 as described above. Except for the entropy decoder 510, the remaining components of the decoder 160 are the same as those in the encoder 200 and are configured in the same manner. Therefore, for the sake of brevity, their detailed description will not be repeated here.

[0057] Because the decoder 160 cannot access the original samples but only the reconstructed samples, the sample predictor 210 of the decoder 160 uses the reconstructed samples to generate predictions of previous samples. Further, because the operations of the encoder 200 and the decoder 160 mirror each other, even if the encoder 200 does have access to the original samples, the sample predictor 210 of the encoder 200 uses the same reconstructed samples to generate predictions of previous samples, so that the encoder 200 can be synchronized with the decoder 160.

[0058] In some embodiments, the encoder 200 and the decoder 160 operate on a single color value of a pixel. Accordingly, multiple encoders 200 and decoders 160 can operate in parallel (eg, concurrently / simultaneously) to process various colors of a pixel, which will be described in more detail later.

[0059] Figure 4A is a schematic diagram illustrating a portion of the encoder 200 together with a portion of the decoder 160 according to some embodiments of the present disclosure. Figure 4B According to some embodiments of the present disclosure Figure 4A , which illustrates the identical operation of the entropy encoder 250 and the entropy decoder 510. According to some embodiments, except for the quantizer 240, the operations performed by the components of the encoder 200 and the decoder 160 are lossless operations. For example, Figure 4A The sequential operations of the entropy encoder 250 and the entropy decoder 510 (such as block A) shown in are identical operations and can therefore be omitted in effect, resulting in Figure 4B Therefore, the error in the signal space (which is defined as the difference between the reconstructed samples at the output of the decoder 160 and the original samples at the input of the encoder 200) is equivalent to the error in the residual space.

[0060] Therefore, if the error in the residual space is maintained as a non-negative value, the error in the signal space will also be non-negative. In addition, if the error in the residual space is bounded, that is, if e'-e≤δ (where e' is the output of the inverse quantizer 260, e is the input of the quantizer 240, and where δ is a positive integer representing the maximum allowable error), the error in the signal space will also be limited to δ. This relationship holds true even if there is a modular adder 230 and a modular subtractor 280. In contrast, transform coding used in related arts (e.g., High Efficiency Video Coding (HEVC), JPEG-2000, or VESA Display Compression-M (VDC-M)) may not be suitable for ensuring non-negative reconstruction errors because positive errors in the transform domain do not ensure non-negative reconstruction errors in the signal space. This may be due to the fact that transforms such as discrete cosine transforms (DCT) retain the L2 norm (or energy) between the original coefficients and the transformed coefficients rather than the L1 norm.

[0061] Accordingly, in some embodiments, the modulo adder 230 and the quantizer 240 are configured in such a way as to ensure that the error in the residual space remains non-negative and is bounded (ie, limited to δ), which ensures that the error in the signal space is the same.

[0062] The quantizer 240 and the inverse quantizer 260 ensure non-negative bounded errors by appropriately adjusting the quantization band size and the reconstruction points in each band. In some embodiments, given a maximum allowed error δ, the quantizer 240 utilizes uniform or non-uniform quantization bands with a maximum quantization step size δ+1, and uses reconstruction points at the right edge (i.e., maximum value) of the band. Further, in some embodiments, the zero value is at the right edge of the band (e.g., the first quantization band of index 1), so that zero appears in the reconstructed value. This may be due to the fact that the residual R is peaked at zero.

[0063] Figure 5A Quantization tables utilized by the quantizer 240 and the inverse quantizer 260 are shown according to some embodiments of the present disclosure. Figure 5B The operation of the quantizer 240 and the inverse quantizer 260 according to some examples of the present disclosure is shown (the input is e and the output is e'). Figure 5C and Figure 5D A uniform quantization table and a non-uniform quantization table utilized by the quantizer 240 according to some embodiments of the present disclosure are respectively shown.

[0064] Reference Figure 5A, the quantizer 240 uses a uniform band with a reconstruction point at the right edge of each band and a quantization step size of δ+1. In some embodiments, the right edge of the first quantization band is zero (i.e., the reconstruction point is zero). Here, the biased residual e is an integer. In some embodiments, when the modulus addition and subtraction parameters defined in Equation 5 and Equation 7 (also referred to as optimized modulus addition and subtraction parameters) are used, the number of bands NB can be expressed as:

[0065]

[0066] Where δ represents the maximum allowable error, and Ceiling(.) represents the ceiling function. Here, δ+1 can be the maximum allowable step size, that is, a smaller step size can be selected. In some other embodiments, when unoptimized modulus addition and subtraction parameters (i.e., parameters not defined by Equation 5 and Equation 7) are used, the number of bands NB can be expressed as:

[0067]

[0068] In some embodiments, when quantizing the biased residual, the quantizer 240 calculates the quantization index of the biased residual as:

[0069]

[0070] When the optimized modulo addition and subtraction parameters of Equation 5 and Equation 7 are used, -δ≤e≤2 bitdepth -1, where I[e] represents the quantization index, Ceiling(.) represents the ceiling function, e represents the residual after bias, and δ represents the maximum allowable error.

[0071] Similarly, performing inverse quantization on the quantized biased residual may include calculating the dequantized biased residual as:

[0072] Q[e]=(δ+1)I[e] Equation (4)

[0073] Where Q(e) represents the dequantized biased residual, I[e] represents the quantization index, and e represents the unquantized biased residual. Equations 3 and 4 are applicable to embodiments utilizing uniform quantization. When non-uniform quantization is used, the maximum step size may be δ+1.

[0074] Figure 5BAn example is shown where δ=2 and the bit depth (i.e., the number of bits per color) is 3. Here, the bit depth refers to the color information stored in the image, and affects the dynamic range of the image. The higher the bit depth of the image, the more colors can be stored. The simplest image (a 1-bit image) can only display two colors (black and white). This is because 1 bit can only store one of two values ​​(0 (white) and 1 (black)). As the bit depth increases, the dynamic range also increases, which increases the number of quantization bands.

[0075] like Figure 5B As shown in the example of , the biased residual e is 1, resulting in a dequantized biased residual e' of 3.

[0076] According to some embodiments, the modulo adder 230 calculates the biased residual e as:

[0077]

[0078] This means that the residual error e after bias is within the range that can be expressed as:

[0079] -δ≤e≤2 bitdepth -1 Equation (6)

[0080] The upper limit of the biased residual e allows for non-uniform quantization bands which may reduce some of the errors in the biased residual e and thus improve performance.

[0081] For example, when δ=2 and bit depth=3, −2≤e≤7, and quantizer 240 may utilize Figure 5C The uniform quantization table shown in Figure 5D The non-uniform quantization table shown in . When using a uniform band size of 2 Figure 5C In , the quantization errors of all four bands are bounded by δ (i.e., 2). However, embodiments of the present disclosure also allow Figure 5D The non-uniform quantization bands of the quantization table shown in Figure 5D The third and fourth bands have smaller quantization errors bounded by 1 (rather than 2). In some examples, quantizer 240 reserves narrower bands for those bands that are more statistically significant (e.g., bands in which more post-bias residuals exist, or bands whose values ​​have a greater impact on the performance of overdrive system 120). Therefore, the smaller quantization error achieved by the operation of modulo adder 230 can improve the performance of overdrive system 120.

[0082] It should be noted that for non-optimized parameters (e.g., parameters that are not optimized to reduce the range of e), Equation 5 can be modified as:

[0083]

[0084] For the range -δ≤e≤2 bitdepth -1+δ

[0085] Modular subtractor 280 may perform an operation opposite to that of modular adder 230 (which adds the bias value to residual R) by removing the bias value added by modular adder 230 to residual R. According to some embodiments, for a given input x, the output y of modular subtractor 280 is expressed as:

[0086]

[0087] The output e of the modular adder 230 is expressed as:

[0088]

[0089] As described above, in some examples, a clipping function may be applied to the reconstructed samples to ensure that they are within a desired dynamic range. For a given bit depth, the desired dynamic range may be expressed as [0, 2 bitDepth -1]. In the 8 bpc example, the clipping function may limit the output of the modulo subtractor 280 to [0, 7].

[0090] Accordingly, as described above, by using appropriate quantization band sizes and added bias values, the image compression and storage system 110 ensures (eg, guarantees) a non-negative and bounded error with respect to the reconstructed image, which may improve the performance of the overdrive system 120 .

[0091] It should be noted that for unoptimized parameters, Equation 7 can be modified as:

[0092]

[0093] Figure 6 is a method of using an image compression and storage system 110 with an additional color conversion scheme according to an embodiment of the present disclosure Figure 2 Schematic diagram of the color conversion positive error reconstruction encoder 300 of the encoder 200 shown in FIG. The encoder 200 of the color conversion positive error reconstruction encoder 300 is similar to the encoder 200 of the above reference Figure 2 The encoder 200 described is identical and will not be repeated here.

[0094] According to an embodiment, each pixel of a frame may be represented by a plurality of color values ​​(e.g., red, green, and blue). Accordingly, each color value corresponds to a color channel of the pixel. For example, each color channel of a pixel may be represented by a color sub-pixel. In some embodiments, the color channels may be arranged in an RGB format including a red channel, a green channel, and a blue channel. In other embodiments, the color channels may be arranged in an RGBG format (e.g., In other embodiments, however, other color channel arrangements may be envisioned by those skilled in the art. Accordingly, in addition to spatial correlation, the image frame also includes correlations between color channels. Accordingly, a color transform may be performed to exploit such inter-channel correlations.

[0095] Reference Figure 6 , the color converter 310 and the inverse color converter 320 are provided at the original pixel input and the reconstructed pixel output of the encoder 200, respectively. More specifically, the color converter 310 obtains the original pixel X (e.g., the original red channel, the original green channel, and the original blue channel) and converts it into a color-converted pixel Z of the original image, and then provides it to the subtractor 220. For example, the color converter 310 can convert the original pixel X in RGB format into the color-converted pixel Z in YCoCg format, and the inverse color converter 320 can perform an inverse color conversion by converting the pixel from the YCoCg format to the RGB format. The subtractor 220 is as previously described with reference to Figure 2 The residual R is calculated based on the original sample and the predicted sample after the color transformation, and the modular subtractor 280 generates the reconstructed sample Z'. In some embodiments, the reconstructed sample Z' is provided to the inverse color transformer 320 to generate the reconstructed pixel X'. Accordingly, the color transformation can be used together with the positive reconstruction error to exploit the inter-channel correlation.

[0096] In some embodiments, as described above, because the error in the residual space (ie, e'-e) always maintains a non-negative value (eg, a positive value), the error in the signal space (ie, Z'-Z) also always maintains a non-negative value. Figure 6 In the color transformation scheme, positive errors are maintained only in the color transformation space (i.e., Z'-Z), and positive errors are not maintained in the image space (i.e., X'-X). In other words, even if positive errors are maintained in the color transformation space, positive image errors are not guaranteed in the image space. For example, assume that the color converter 310 is an RGB-YCoCg transform (where the forward color transform is represented by a 3×3 matrix T=[0.25 0.50 0.25; 0.50 0 -0.50; -0.25 0.50- 0.25]), and the original input X=[6 54] (where R=6, G=5, B=4). Based on these assumptions, Z=T×X=[5 1 0]. Next, assume that according to Figure 2 The quantizer design described in is Z' = [6 2 1], then the error Z of the color transformed image i '-Z i ≥0, is a positive value. By applying the inverse color transform by means of the inverse color transformer 320, X'=[7, 7, 3], this means that the error in the image space X'-X=[1, 2, -1], which includes negative values, i.e., -1. Therefore, although the positive error is maintained in the residual space and the color transform space, the positive error is not transferred to the image space. Accordingly, the embodiments of the present disclosure will describe a technique for ensuring non-negative errors in the image space while also utilizing color channel correlation. However, the described technique should not be interpreted as being limited to systems that generate non-negative errors. On the contrary, the described technique may also be applicable to systems that generate negative errors.

[0097] Figure 7 is a schematic diagram of a joint color channel entropy coding positive reconstruction error encoder 400 of the image compression and storage system 110 according to an embodiment of the present disclosure.

[0098] Reference Figure 7 According to various embodiments, the encoder 400 utilizes a joint color channel entropy encoding scheme in which quantized values ​​from each color channel of a pixel are provided to an entropy encoder 450. In other words, according to an embodiment, a separate quantizer 440a, 440b is provided for each color channel, wherein the quantized output from each quantizer 440a, 440b is jointly provided as a color channel input to the same entropy encoder 450. Accordingly, Figure 7 The embodiment shown in shows a red channel quantization branch 410a for the red channel and a green channel quantization branch 410b for the green channel, each fed into the same entropy encoder 450. In some embodiments, there may be n color channel quantization branches (where n is an integer greater than 1) based on the pixel format, for example, the RGB format includes three color channels, in which case there will be three quantizers each fed into the same entropy encoder 450; and another format including, for example, four color channels will have four quantizers each fed into the same entropy encoder 450.

[0099] In more detail, according to some embodiments, the sample predictor 480a for the red channel generates a prediction of the red pixel based on the previous image, and the subtractor 420a calculates the residual R based on the original pixel and the predicted pixel. The green channel quantization branch 410b can be configured in a similar manner to the red channel quantization branch 410a to generate the residual R of the green channel. The residual R of other channels (e.g., green channel, blue channel, etc.) is generated based on a similar process. In some embodiments, the original pixel X includes inter-channel correlation (e.g., there may be a relationship between samples among the color channels. For example, a combination of channel values ​​may occur at a specific frequency), and because differential pulse code modulation (DPCM) only uses spatial correlation, the inter-channel correlation is transferred to the residual space. The modulo adder 430a adds a bias value (which is a constant value). The residual R of other channels can be modified in a similar manner by the modulo adder of the corresponding branch (e.g., green channel quantization branch 410b). Because the modulo adder 430a (and the modulo adders of other channels) adds a constant value to the residual R, the inter-channel correlation remains unaffected by the modulo adder 430a again. Because the quantizer 440a removes some lower bit plane values ​​while retaining the most significant bit (MSB), the quantizer 440a (and similar quantizers in other channels) does not affect the inter-channel correlation of the residual R either. Therefore, the inter-channel correlation between the quantized indices of each channel can be maintained. These quantized indices can be provided to a lossless encoder such as an entropy encoder 450. Because the lossless encoder is lossless, a non-negative reconstruction error can be maintained.

[0100] According to some embodiments of the present disclosure, the green channel quantization branch 410b can be configured in a similar manner to the red channel quantization branch 410a to generate an inter-channel correlation quantization index, which can then be provided to the same entropy encoder 450 used by the red channel quantization branch 410a. In addition, according to an embodiment, in the case where the pixel is in RGB format, the blue channel quantization branch (not shown) can also be configured to provide the inter-channel correlation quantization index to the same entropy encoder 450. Accordingly, the entropy encoder 450 can obtain the inter-channel correlation quantization index from the red channel, the green channel, and the blue channel, and jointly encode the quantization index into one shot to utilize the inter-channel correlation included in each of the quantization indexes. Accordingly, by encoding all color channels together, the total number of bits used to encode the image frame can be reduced, and its details will be explained next.

[0101] In some embodiments, variable length binary codewords can be used to save the amount of storage space required to store the encoded data, and the general format of the entropy encoder 450 is such that the most frequent mode is assigned the shortest binary codeword and the least frequent mode is assigned the longest binary codeword. In the present disclosure herein, a mode refers to an n-tuple of values ​​of quantized indexes from different color channels, for example, an RGB triplet format, an RGBG quadruple format, an RB / GG pair or an RG / BG pair format, or other formats of pixels. Thus, for example, if the mode corresponding to the quantized index (0, 0, 0) is the most frequent mode, and (4, 4, 4) is the second most frequent mode, then (0, 0, 0) can be assigned the corresponding binary codeword "0", and (4, 4, 4) can be assigned the corresponding binary codeword "10". On the other hand, if the mode (3, 3, 2) is the least frequent mode, a longer binary codeword (e.g., "111111110") can be assigned to (3, 3, 2). Although the binary codeword for the less frequent mode can be longer than the actual mode itself, the bit saving achieved by the more frequent mode causes overall bit saving. Accordingly, according to the joint coding technique, the binary codeword "0" uses only 1 bit, while (0, 0, 0) uses 3 bits when each color is encoded independently, thereby saving bits in the case of the most frequent mode. Because this is the most frequent mode, the saved bits can add up to a considerable number. Here, the pattern (0, 0, 0) represents a quantized index of 0 for the red channel, a quantized index of 0 for the blue channel, and a quantized index of 0 for the green channel. Similarly, the pattern (4, 4, 4) represents a quantized index of 4 for the red channel, a quantized index of 4 for the blue channel, and a quantized index of 4 for the green channel.

[0102] According to some embodiments, the joint color channel entropy coding technique includes first training the image compression and storage system 110 to generate a lookup table (or other data structure) including information pointing to the frequency of occurrence of certain patterns in the quantized index. By collecting such information, a codeword (e.g., a binary codeword) can be generated and stored in the lookup table so that the codeword can be used to reduce the number of bits used while encoding. In some embodiments, training is performed offline, for example when the image compression and storage system 110 is manufactured or programmed. Therefore, once the training process is performed and the lookup table is generated, it is not necessary to perform the training process again. Accordingly, the training process can be performed offline because it is not performed during the actual encoding process.

[0103] Figure 8It is a histogram showing common patterns of quantized indexes that can be learned from training images, and emphasizes the advantages of pattern learning and how the learned pattern database can be applied to real-world scenarios. In some embodiments, the training process can be performed by identifying how many times each pattern (of the quantized index) appears in the training images. Figure 8 The histogram (H i ) shows the count of the occurrence of the pattern in the training image. In the histogram, each point on the x-axis represents the mode value of the quantized index (this will be explained later), and the y-axis represents the corresponding number of occurrences or probability of the pattern. The mode value of the quantized index can be calculated as val = M 2 ×Q[R]+M×Q[G]+Q[B], where M is the number of representative quantized indices for each color channel, Q[R] is the quantized index of the red channel, Q[G] is the quantized index of the green channel, and Q[B] is the quantized index of the blue channel. Accordingly, for example, if M=5, then val=25×Q[R]+5×Q[G]+Q[B]. Fig. 9 is a table showing post-quantization index patterns and associated values. For example, the post-quantization index pattern (4, 4, 4) is shown as corresponding to the value 124 (eg, (25×4)+(5×4)+4=124).

[0104] A sorted array S of pattern values ​​may be generated, where the values ​​in array S are sorted based on the number of times they appear in the training images. Thus, the first pattern value in array S (e.g., S[0]) may correspond to the most frequently occurring pattern of the quantized indices.

[0105] In some embodiments, the number of occurrences of each mode value may correspond to the average number of occurrences of the mode value across multiple training images. In some embodiments, the counts of occurrences according to each training image may be weighted equally (e.g., each training image is equally important), but in other embodiments, the counts of occurrences according to each of the training images may be weighted differently (e.g., because some training images may be considered more important or valuable than other training images). Thus, the weights of the images may vary depending on the design parameters of the training process. Accordingly, the weighted average histogram may be calculated as:

[0106]

[0107] Among them, w i is the weight assigned to the i-th training image, H i corresponds to the histogram of the i-th training image, and N is the sum of the weights. If the weights of the images are the same, then w i= 1. On the other hand, if the weights are different, weights can be assigned to the training images based on the importance of each image.

[0108] Return to reference Fig. 9 , the lookup table can be sorted based on the frequency of occurrence of each pattern. In the above example, the pattern (0, 0, 0) occurs most frequently and is therefore sorted and placed in the first position (e.g., entropy coding index 0). The pattern (4, 4, 4) occurs second most frequently and is therefore sorted and placed in the second position (e.g., entropy coding index 1). This process can be repeated until all or most of the entropy coding indexes are sorted. Accordingly, a table such as Fig. 9 , where S = {0, 124, 31, 120, 1, 5, 20, 4, 123, 25, 93, 100, 24, 90, 6, 30, 104, 115, 33, 81, 121, 91, 80, 26, 95, 34, 40, 15, 75, 21, 43, 32, 94, 9, 118, 45, 3, 119, 105, 101, 29, 23, 122, 16, 19, 8, 116, 41, 99, 35, 28, 76, 103, 46, 79, 92, 49, 106, 109, 65, 44, 18, 36 , 78, 84, 56, 96, 108, 48, 117, 10, 50, 2, 42, 110, 98, 83, 85, 7, 38, 86, 37, 11, 22, 82, 55, 14, 27, 102, 51, 70, 54, 17, 68, 88, 13, 77, 114, 74, 66, 59, 87, 58, 71, 47, 53, 107, 69, 39, 62, 61, 111, 113, 89, 73, 57, 64, 97, 63, 60, 12, 67, 52, 72, 112} and store it as an array in a lookup table. In some embodiments, other training methods can be used to generate similar lookup tables or data structures.

[0109] Figure 8 The lower half of shows a sorted histogram based on the sorted array S. As shown, the pattern values ​​are sorted from the most frequently occurring pattern to the least frequently occurring pattern. For example, entropy coding index 0 in the histogram occurs most frequently and is already in the first position. In this example, the second most frequently occurring pattern is entropy coding index 124, so it is moved to the second position, as shown in FIG. Figure 8 The next most frequently occurring pattern is entropy coding index 31, so it is moved to the third position. This process can be repeated for all or most of the entropy coding indices to generate a sorted histogram.

[0110] Once the training process is completed and the lookup table is generated, the entropy encoder 450 can then be used to perform joint color channel entropy coding. In some embodiments, the mode can be based on an n-tuple format, such as, for example, an RGB triplet format, an RGBG quadruple format, an RB / GG pair or an RG / BG pair format, or any other format of a pixel. The examples provided in this disclosure will be described by using the RGB triplet format as an example, but the same method is also applicable to other formats. The encoding process using the trained information stored in the lookup table (or other data structure) will now be described. In subsequent examples, a sorted array S={0,124,31,120,1,5,20,...,52,72,112} will be used as an example to help explain the embodiments. In some embodiments, the quantized indexes for the red, green, and blue channels of a pixel are provided as inputs to the entropy encoder 450.

[0111] According to one encoding method, the lookup table includes a sorted array S={0, 124, 31, 120, 1, 5, 20, ..., 52, 72, 112}. Here, the mode values ​​are sorted from the most frequently occurring mode to the least frequently occurring mode, wherein the most frequently occurring mode is entropy coding index 0, the second most frequently occurring mode is entropy coding index 1, and so on. Based on this sorting, according to this example, val 0 corresponds to entropy coding index 0, val 124 corresponds to entropy coding index 1, val 31 corresponds to entropy coding index 2, and so on. Fig.10 . Accordingly, for example, during encoding, if the quantized index is 124, the encoder searches for the value 124 in the lookup table, and then based on the position of 124 (which is the entropy encoding index corresponding to the second most frequently occurring mode), the encoder knows that val 124 corresponds to entropy encoding index 1. This process can be repeated for each mode value.

[0112] According to another encoding method, the sorted array S can be used to generate Fig.11The lookup table (idxArray) of the lookup table shown in . The idxArray can be an entropy coding index array arranged based on the corresponding mode value, where the mode value val is arranged in numerical order. Therefore, the encoder does not have to search the mode value in the idxArray. Instead, the encoder only searches for the mode value in the idxArray and extracts the corresponding entropy coding index (index), where index=idxArray[val]. For example, if the mode value is 124, the encoder goes to position 124 without searching and obtains the entropy coding index 1 from the idxArray. Accordingly, pre-calculation is performed to generate the idxArray without having to perform a search during the encoding process, thereby reducing the amount of calculation performed when obtaining information from the idxArray, thereby saving time and saving resources. On the other hand, the method requires storing a lookup table for the encoder and another lookup table for the decoder. That is, the entropy coding index array idxArray can be stored at the encoder and the array S can be stored at the decoder. On the other hand, the first encoding method does not require an idxArray, but calculations must be performed to search the lookup table. Therefore, a suitable encoding method can be selected based on design requirements.

[0113] Accordingly, entropy coding indexes can be obtained from these lookup tables, and entropy coding can be performed on the indexes by the entropy encoder using one or more known entropy coding techniques (such as, exponential-Golomb, Golomb-Rice or Huffman coding techniques). Accordingly, the quantized indexes from the three color channels can be jointly encoded by the entropy encoder 450.

[0114] In some embodiments, the decoder may be configured to decode the bits encoded by the encoder. More specifically, the decoder decodes the jointly encoded bitstream into color-separated quantized indices Q[R], Q[G], and Q[B], which are then quantized by inverse quantizers 460a, 460b (see Figure 7 ) is inverse quantized. Accordingly, the entropy coding index is matched with its corresponding mode value val in the lookup table to obtain the corresponding value, i.e., val=S[index]. Therefore, referring back to the above example, the entropy coding index 1 in the array S determines that the mode value is 124, i.e., S[1]=124, and the entropy coding index 2 determines that the mode value is 31, and so on. These values ​​can then be substituted into the following equations:

[0115] Q[R]=val / M 2

[0116] Q[G]=(val-Q[R]×M 2 ) / M

[0117] Q[B]=val-Q[R]×M2 -Q[G]×M

[0118] Where M is the number of representative quantized indices for each color channel to obtain quantized indices Q[R], Q[G], and Q[B] for color separation of RGB. If M is designed to be a power of 2, division can be avoided. That is, for example, if M=2, then Q[R]=val / 4=val>>2, where>> is a right shift operator.

[0119] Fig.12 is a table showing numerical examples of codebooks stored in the lookup table. Here, the table includes examples of ranges of entropy coding indexes shown as entropy coding index ranges and codewords for respective corresponding entropy coding indexes provided as bit streams including prefixes and suffixes. Note that Fig.12 The codewords shown in are only example groups of codewords, and in practice, other codewords may be used. In this example, there are 8 representative levels. For example, entropy coding indexes 0-3 are in the first level, 4-11 are in the second level, 12-27 are in the third level, 28-59 are in the fourth level, 60-123 are in the fifth level, 124-251 are in the sixth level, 252-507 are in the seventh level, and 508-511 are in the eighth level, where the length of the binary codeword increases as the sorted entropy coding index increases. Each prefix in the table specifies a level, and each suffix indicates a position within the corresponding level. As the prefix length increases (e.g., the level increases), the suffix length increases. In the example shown, the suffix corresponds to position -1. Entropy coding index 3 is in the first level including entropy coding indexes 0-3. The prefix for this level is 0, and entropy coding index 3 is in the 4th position of this level. Accordingly, the suffix is ​​decimal 3. The suffix length for this level is 2, so the suffix is ​​11. Accordingly, according to Fig.12 According to the table in , the binary codeword for entropy coding index 3 is 011.

[0120] Here, a higher entropy coding index corresponds to a lower pattern occurrence frequency, and a lower entropy coding index corresponds to a higher pattern occurrence frequency. As described above, the entropy coding index can be stored in an array idxArray that is arranged (e.g., indexed) based on the pattern value (patternValue). Therefore, idxArray[patternValue] is equal to the entropy coding index of patternValue. In the example where M=8, idxArray

[512] ={0,448,64,8,56,1,7,384,504,72,128,120,456,63,9,320,65,192,455,73,511,256,449,57,15,71,2,440,79,136,16,127,48,463,505,6,457,121,376,447,137,20 0,369,10,365,219,312,319,383,129,184,58,385,391,62,392,3,74,496,264,80,23,135,55,112,87,49,374,66,201,464,17,450,433,24,441,503,143,454,40,510,13,14,193,81,248,265,25 0,375,255,327,458,205,5,185,278,54,399,439,393,32,202,4,70,328,191,257,321,151,277,138,145,249,186,311,78,497,11,22,144,488,126,119,214,502,113,438,150,465,377,310,86 ,432,471,146,199,130,335,329,122,386,176,400,194,269,209,263,266,203,207,462,82,141,25,88,271,267,472,104,424,213,330,342,208,506,61,12,292,210,394,142,215,41,47,446,77，18，67，96，480，31，183，272，390，134，406，305，501，470，75，337，190，451，33，322，131，279，401，39，258，111，50，453，408，336，59，247，273，407，313，368，69，460，461，509，198，240，195，495，416，60，68，452，114，177，26，479，21，89，398，105，466，152，489，76，500，304，168，95，508，118，382，487，473，189，343，139，442，498，387,507，53，160，218，334，103，97，459，481，314，323，326，125，246，318，83，206，431，270，46，254，123，30，341，232，153，352，217，34，42，367，149，378，344，268，133，19，262，360，241，94，437，415，388，51，90，175，124，445，169，402，159，216，85，309，425，389，409，224，20，423，182，52，181，428，140，494，38，434，154，161，345，178，303，296，204，499，35，187，147，474，366，338，467，167，417，373，491，117，132，84，245，478，395，429，280，27，492，410，430,355,490,443,281,239,333,397,102,110,486,106,331,274,469,115,288,98,324,302,444,332,482,43,238,346,223,44,287,359,361,259,45,282,29,225,396,158,116,295,231,197, 148,174,188,242,289,351,468,196,28,426,325,353,233,436,261,493,170,381,427,91,435,297,306,253,414,155,370,222,37,36,422,294,485,379,419,477,211,418,354,317,166 ,179,162,347,107,405,230,483,251,476,315,380,362,92,252,290,156,157,411,403,108,93,260,180,226,301,235,234,475,364,298,484,99,243,316,358,109,237,404,221,165,2 , 29, 275, 286, 307, 350, 308, 101, 163, 227, 421, 363, 171, 172, 283, 372, 299, 244, 100, 357, 173, 164, 420, 371, 339, 356, 293, 228, 412, 276, 220, 291, 340, 212, 413, 236, 348, 284, 300, 285, 349}. Based on the idxArray, the mode value 0 corresponds to the entropy coding index 0 (e.g., idxArray[0]=0), which corresponds to the entropy coding index range [0, 3], prefix 0 and suffix 00. Therefore, the codeword for the mode value 0 is 000. Similarly, mode value 2 corresponds to entropy coding index 64 (e.g., idxArray[2]=64), which corresponds to the entropy coding index range [60, 123], prefix 11110, and suffix 000100. Therefore, the codeword for mode value 2 can be 11110000100. Accordingly, by mapping the entropy coding index to, Fig.12 The entropy encoder can encode the quantized triplet color value into a binary codeword using a lookup table.

[0121] 76,447,137,200,369,10,365,219,312,319,383,129,184,58,385,391,62,392,3,74,496,264,80,23,135,55,112,87, 49,374,66,201,464,17,450,433,24,441,503,143,454,40,510,13,14,193,81,248,265,250,375,255,327,458,205,5,185,278,54,399,439,393, 32,202,4,70,328,191,257,321,151,277,138,145,249,186,311,78,497,11,22,144,488,126,119,214,502,113,438,150,465,377,310,86,432,4 71,146,199,130,335,329,122,386,176,400,194,269,209,263,266,203,207,462,82,141,25,88,271,267,472,104,424,213,330,342,208,506,6 1,12,292,210,394,142,215,41,47,446,77,18,67,96,480,31,183,272,390,134,406,305,501,470,75,337,190,451,33,322,131,279,401,39,25 8,111,50,453,408,336,59,247,273,407,313,368,69,460,461,509,198,240,195,495,416,60,68,452,114,177,26,479,21,89,398,105,466,152 ,489,76,500,304,168,95,508,118,382,487,473,189,343,139,442,498,387,507,53,160,218,334,103,97,459,481,314,323,326,125,246,318,83,206,431,270,46,254,123,30,341,232,153,352,217,34,42,367,149,378,344,268,133,19,262,360,241,94,437,415,388,51,90,175,124,445,169,402,159,216,85,309,425,389,409,224,20,423,182,52,181,428,140,494,38,434,154,161,345,178,303,296,204,499,35,187,147,474,366,338,467,167,417,373,491,117,132,84,245,478,395,429,280,27,492,410,430,355,490,443,281,239,333,397,102,110,486,106,331,274,469,115,288,98,324,302,444,332,482,43,238,346,223,44,287,359,361,259,45,282,29,225,396,158,116,295,231,197,148,174,188,242,289,351,468,196,28,426,325,353,233,436,261,493,170,381,427,91,435,297,306,253,414,155,370,222,37,36,422,294,485,379,419,477,211,418,354,317,166,179,162,347,107,405,230,483,251,476,315,380,362,92,252,290,156,157,411,403,108,93,260,180,226,301,235,234,475,364,298,484,99,243,316,358,109,237,404,221,165,229,275,286,307,350,308,101,163,227,421,363,171,172,283,372,299,244,100,357,173,164,420,371,339,356,293,228,412,276,220,291,340,212,413,236,348,284,300,285,349}。,

[0122] Based on the array S, the mode value 0 corresponds to the entropy coding index 0 (e.g., S[0]=0), which corresponds to the entropy coding index range [0,3], prefix 0, and suffix 00. Therefore, the codeword for the mode value 0 is 000. Similarly, the mode value 2 corresponds to the entropy coding index 64 (e.g., S

[64] =2), which corresponds to the entropy coding index range [60,123], prefix 11110, and suffix 000100. Therefore, the codeword for the mode value 2 can be 11110000100. Accordingly, by mapping the entropy coding index to Fig.12 The entropy encoder can encode the quantized triplet color value into a binary codeword using a lookup table.

[0123] In some embodiments, during decoding, the decoder looks for the first zero in the prefix of the codeword. Based on the number of 1s that appear before the zero, the decoder can determine the range of the entropy coding index. For example, if the prefix is ​​110, the decoder can determine that the entropy coding index is in the range of 12-27. Similarly, if the prefix is ​​1110, the decoder can determine that the entropy coding index is in the range of 28-59. Accordingly, the length of the suffix can be determined based on the length of the prefix. In other words, if the prefix is ​​longer, the suffix can be longer. Next, the decoder looks at the value of the suffix after the zero. If the suffix value is all zero, the entropy coding index is the first entropy coding index in the entropy coding index range. Therefore, for example, if the decoder has determined that the entropy coding index range is 12-27, because the prefix is ​​110 and the suffix is ​​all zero (e.g., 0000), the decoder can determine that the entropy coding index is 12, which is the first entropy coding index in the range 12-27. If the suffix is ​​0001, the entropy coding index is 13, and if the suffix is ​​0010, the entropy coding index is 14, etc. Therefore, if the suffix is ​​all 1s (e.g., 1111), the entropy coding index is the last entropy coding index in the range 12-27, which is entropy coding index 27.

[0124] According to some embodiments, the entropy coding index range 508-511 is a special case where the prefix does not include any zeros. Therefore, the decoder can be set to look for a threshold number of ones, such as seven ones in this example. Therefore, if the decoder determines that there are seven ones in the prefix, the decoder can determine that the entropy coding index range is 508-511. Here, the suffix is ​​determined in a similar manner, where all zeros indicate the first entropy coding index of the entropy coding index range 508-511. Accordingly, Fig.12The example table of utilizes a mixture of Golomb-Rice code and exponential-Golomb code. Here, Golomb-Rice code is used when the prefix length is 1, and exponential-Golomb code is used when the prefix length is greater than 1. However, the use of Golomb-Rice code or exponential-Golomb code is only an example, and in practice, other entropy encoding methods may be used instead and / or may be used in combination with various entropy encoding methods.

[0125] Fig.13 is a diagram showing the mapping of entropy coding indices to a matrix for a RGBG format (e.g., When the pixel is in RGBG format, the same method as discussed with reference to the pixel in RGB format can be used. However, when each quantized color index is represented by 8 levels (e.g., M=8) (i.e., red, green, blue, and green), 4096 entropy coding indexes are possible. Therefore, creating 4096 entropy coding indexes in the lookup table may be expensive and resource-consuming. In order to save storage space in the lookup table, the entropy coding indexes may be divided into two groups of 64 entropy coding index arrays. That is, by dividing RGBG into pairs (e.g., RB and GG), each pair of RB and GG includes only 64 entropy coding indexes, which is much less than 4096 entropy coding indexes. Accordingly, the following equation may be used:

[0126] index rb =idxArray RB [8×Q[B]+Q[R]]

[0127] index g =idxArray G [8×Q[G 1 ]+Q[G 0 ]]

[0128] This results in the following array of entropy encoded indices:

[0129] idxArray RB ={0,4,15,27,62,33,16,3,2,6,13,23,48,53,36,8,11,9,17,31,52,59,37,29,22,18,25,35,51,56,49,44,46,38,43,60,58,45,40,41,24,42,54,61,55,34,21,20,12,28,39,57,50,30,19,10,1,7,32,47,63,26,14,5}, and

[0130] idxArray G={0,4,13,9,30,10,16,1,2,8,21,28,44,35,18,6,15,24,38,52,57,42,23,17,12,47,58,63,49,25,36,33,32,48,61,62,46,50,53,41,11,37,55,31,60,56,43,29,14,20,27,51,59,54,39,22,3,5,19,34,40,45,26,7}.

[0131] It can be seen that the entropy encoding indexes of the two sets of arrays are different. Therefore, when encoding red and blue, the RB array is used, and when encoding green, the G array is used.

[0132] The corresponding (sorted) array S of mode values ​​for this example could be:

[0133] S RB ={0,56,8,7,1,63,9,57,15,17,55,16,48,10,62,2,6,18,25,54,47,46,24,11,40,26,61,3,49,23,53,19,58,5,45,27,14,22,33,50,38,39,41,34,31,37,32,59,12,30,52,28,20,13,42,44,29,51,36,21,35,43,4,60}, and

[0134] S G ={0, 7, 8, 56, 1, 57, 15, 63, 9, 3, 5, 40, 24, 2, 48, 16, 6, 23, 14, 58, 49, 10, 55, 22, 17, 29, 62, 50, 11, 47, 4, 43, 32, 31, 59, 13, 30, 41, 18, 54, 60, 39, 21, 46, 12, 61, 36, 25, 33, 28, 37, 51, 19, 38, 53, 42, 45, 20, 26, 52, 44, 34, 35, 27}.

[0135] Fig.14900 is a flowchart illustrating an example method for performing joint color channel entropy coding according to an embodiment of the present disclosure. Assuming that the encoder has been trained and an array S has been stored in a lookup table thereon, the encoder receives an image frame. The image frame is composed of pixels and each pixel includes n color channels such as, for example, a red channel, a green channel, and a blue channel (902). Next, the image frame is encoded by a separate color channel encoding process via a separate color channel quantization branch for each of the n color channels that guarantees a non-negative reconstruction error. The encoder further generates a first sample group of quantized color indexes based on the color value of the residual of a first specific pixel in the image frame. Here, the first sample group corresponds to a first n-tuple of color channels (904). Next, the entropy encoder generates entropy coding for the first specific pixel based on data that maps the n-tuple of quantized indexes to entropy coding indexes. Here, the first n-tuple is mapped to a first entropy coding index (906).

[0136] In some embodiments, each of the separate color channel quantization branches generates a second sample group of quantized color indexes based on the color value of the residual of a second specific pixel in the image frame. Here, the second sample group corresponds to the second n-tuple (908). Next, the entropy encoder generates entropy coding for the second specific pixel based on data mapping the n-tuple of the quantized index to the entropy coding index, wherein the second n-tuple is mapped to the second entropy coding index (910). Accordingly, the received image frame can be encoded by utilizing the inter-channel correlation of the color of the image frame in the joint color channel entropy coding technique while maintaining a non-negative reconstruction error. Note that in some embodiments, the joint color channel entropy coding technique utilizing the inter-channel correlation of the color image can also be applied to encoders and / or decoders that do not necessarily guarantee a non-negative reconstruction error. In other words, the reconstruction error can be a negative value, and in this case, the joint color channel entropy coding technique can be utilized without departing from the scope of the embodiments described herein.

[0137] Fig.15 1000 is a flowchart illustrating an example method for performing joint color channel entropy encoding according to an embodiment of the present disclosure. According to the method, a first mode value is calculated based on a first n-tuple (1002). Once the first mode value is calculated, an entropy encoder references a lookup table to determine a corresponding first entropy encoding index (1004), the lookup table including an array of data for the calculated first mode value. Once the corresponding first entropy encoding index is determined, the first entropy encoding index is encoded (1006).

[0138] Fig.161100 is a flowchart illustrating another example method for performing joint color channel entropy coding according to an embodiment of the present disclosure. According to the method, a first mode value is calculated based on a first n-tuple (1102). Once the first mode value is calculated, a first entropy coding index is obtained from a lookup table of an array including data for the calculated first mode value according to the position of the array indicated by the first mode value. Here, the first entropy coding index of the mode value is pre-calculated when creating the lookup table (1104). Therefore, no additional calculations are performed while determining the first entropy coding index of the mode value. Once the corresponding first entropy coding index is determined, the first entropy coding index is encoded (1106).

[0139] Fig.17 1200 is a flowchart illustrating an example method for training an encoder according to an embodiment of the present disclosure. During training, for example, in an offline process, a first training image frame is received by a training encoder (1202). Next, a training reconstruction error value corresponding to the first training image frame is calculated. Here, the training reconstruction error value includes an n-tuple of quantized color values ​​(1204). In some embodiments, n corresponds to the number of color channels, and therefore, in the RGB format example, n=3. Next, a quantized index is generated based on the calculated training reconstruction error value (1205). Next, the frequency of occurrence of each n-tuple of the quantized index is identified (1206). Once the frequency of occurrence of each n-tuple of the quantized index is identified, an entropy coding index is assigned to each n-tuple of the quantized index according to the frequency of occurrence of the n-tuple (1208). Next, the n-tuples of the quantized index are sorted according to the frequency of occurrence (1210).

[0140] Fig.18 1300 is a flowchart illustrating an example method for decoding an entropy coded index according to an embodiment of the present disclosure. After the bits are encoded, the encoder may provide the encoded bits as a bitstream to a decoder for decoding. Accordingly, the decoder may first receive the encoded bitstream and then parse the bitstream to identify the entropy coded index (1302). Next, a color-separated quantized index corresponding to the entropy coded index is generated based on data that maps the entropy coded index to the color-separated quantized index. In some embodiments, such mapping data may be obtained by reference to a lookup table. Here, the color-separated quantized index is a set of n-tuples (1304). Next, each color of the color-separated quantized index may be inversely quantized by a corresponding color channel quantization branch of each color channel in the n color channels to generate a dequantized biased residual (1306).

[0141] Accordingly, the encoder may be trained offline, for example, during manufacture or assembly of the encoder, and thus, the lookup table may be hard-coded into a storage medium (e.g., a memory). Additionally, one or more methods described throughout various embodiments of the present disclosure may be implemented and performed by an image compression and storage system 110, which may be part of the frame compensation system 100 or other related systems shown in the figures by the present disclosure. For example, Figure 7 The entropy encoder 450 shown in FIG. 4 may be implemented in an encoder such as encoder 400 or other types of encoders known to those skilled in the art.

[0142] It will be understood that although the terms "first", "second", "third", etc. may be used herein to describe various elements, components, regions, layers and / or portions, these elements, components, regions, layers and / or portions should not be limited by these terms. These terms are used to distinguish one element, component, region, layer or portion from another element, component, region, layer or portion. Therefore, the first element, first component, first region, first layer or first portion described above may be referred to as a second element, second component, second region, second layer or second portion without departing from the spirit and scope of the present invention.

[0143] Spatially relative terms such as "below", "below", "down", "beneath", "above", and "on" may be used herein to describe the relationship between an element or feature and another (some) element or feature as shown in the drawings for the convenience of explanation. It will be understood that, in addition to the orientations depicted in the drawings, spatially relative terms are also intended to cover different orientations of the device in use or operation. For example, if the device in the drawings is turned over, the elements described as being "below", "below", or "below" other elements or features will then be oriented "above" the other elements or features. Therefore, the example terms "below" and "below" can cover both the above and below orientations. The device can be oriented in other ways (e.g., rotated 90 degrees or in other orientations), and the spatially relative descriptors used in this article should be interpreted accordingly.

[0144] It will be understood that when an element or layer is referred to as being "on," "connected to," or "coupled to" another element or layer, the element or layer can be directly on, directly connected to, or coupled to the other element or layer, or one or more intervening elements or layers may be present. Additionally, it will be understood that when an element or layer is referred to as being "between" two elements or layers, the element or layer can be the only element or layer between the two elements or layers, or one or more intervening elements or layers may also be present.

[0145] The terms used in this article are for the purpose of describing specific embodiments, and are not intended to limit the present invention. As used in this article, the singular forms "one" and "the" are also intended to include plural forms, unless the context clearly indicates otherwise. It will be further understood that when used in this specification, the terms "include", "comprise", "contain", and "have" indicate the existence of stated features, integral bodies, steps, operations, elements and / or parts, but do not exclude the existence or increase of one or more other features, integral bodies, steps, operations, elements, parts and / or their groups. As used in this article, the term "and / or" includes any and all combinations of one or more of the related listed items. When after the element list, expressions such as "at least one of" modify the entire element list without modifying the individual elements in the list.

[0146] As used herein, the terms "substantially," "about," and similar terms are used as terms of approximation and not as terms of degree, and are intended to account for the inherent deviations in measured or calculated values ​​that one of ordinary skill in the art would recognize. Further, the use of "may" when describing embodiments of the present invention refers to "one or more embodiments of the present invention." As used herein, the terms "use" and "used" may be considered synonymous with the terms "utilizing" and "utilized," respectively.

[0147] The electronic or electrical devices and / or any other related devices or components according to the embodiments of the present invention described herein can be implemented using any suitable hardware, firmware (e.g., application specific integrated circuits), software, or a combination of software, firmware, and / or hardware. For example, the various components of these devices can be formed on an integrated circuit (IC) chip or on separate IC chips. Further, the various components of these devices can be implemented on a flexible printed circuit film, a tape carrier package (TCP), a printed circuit board (PCB), or can be formed on a substrate. Further, the various components of these devices can be processes or threads in one or more computing devices, running on one or more processors, executing computer program instructions and interacting with other system components to perform the various functions described herein. Computer program instructions are stored in a memory, which can be implemented in a computing device using a standard memory device (such as, for example, a random access memory (RAM)). Computer program instructions can also be stored in other non-temporary computer-readable media such as, for example, a CD-ROM or a flash drive. In addition, those skilled in the art will recognize that the functions of various computing devices can be combined or integrated into a single computing device, or the functions of a particular computing device can be distributed across one or more other computing devices without departing from the spirit and scope of the exemplary embodiments of the present invention.

[0148] The operations performed by the components of the encoder and decoder can be performed by a "processing circuit" or "processor" for processing data or digital signals, which can include any combination of hardware, firmware and software. The processing circuit hardware can include, for example, an application-specific integrated circuit (ASIC), a general or dedicated central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), and a programmable logic device such as a field programmable gate array (FPGA). In the processing circuit, as used herein, each function is performed by hardware configured to perform the function (i.e., hard-wired hardware) or by more general hardware (such as a CPU) configured to execute instructions stored in a non-temporary storage medium. The processing circuit can be assembled on a single printed wiring board (PWB) or distributed throughout multiple interconnected PWBs. The processing circuit can contain other processing circuits; for example, the processing circuit can include two processing circuits (FPGA and CPU) interconnected on the PWB.

[0149] Unless otherwise defined, all terms (including technical and scientific terms) used in this article have the same meaning as those commonly understood by ordinary technicians in the field to which the present invention belongs. It will be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant field and / or the present invention, and should not be interpreted in an idealized or overly formal sense, unless explicitly defined in this article.

[0150] The embodiments described herein are merely examples. Those skilled in the art may recognize various alternative embodiments based on those embodiments specifically disclosed. Those alternative embodiments are also intended to be within the scope of this disclosure. Therefore, the embodiments are limited only by the appended claims and their equivalents.

Claims

1. A method for encoding, comprising: Receiving an image frame including pixels, wherein each of the pixels includes n color channels, and each color channel pixel of the n color channels includes inter-channel correlations between each other; generating, by a separate color channel quantization branch for each of the n color channels configured to ensure a reconstruction error having a non-negative value, a first sample group of quantized indexes based on a color value of a residual of a first specific pixel in the image frame, wherein each quantized index of the separate color channel quantization branch for the first sample group of quantized indexes includes the inter-channel correlation between each other, and wherein the first sample group corresponds to a first n-tuple; and An entropy code is generated for the first specific pixel by an entropy encoder based on the inter-channel correlation of the pixel transferred to the post-quantization index and data from each of the individual color channel quantization branches mapping an n-tuple of the first sample group of the post-quantization index to an entropy coding index, wherein the first n-tuple is mapped to a first entropy coding index.

2. The method according to claim 1, wherein: The generating of the first sample set of the quantized indices comprises computing, by each of the individual color channel quantization branches, the residual by subtracting a prediction based on previously reconstructed pixels from the received image frame.

3. The method according to claim 2, wherein: The generating of the first sample group of the quantized index further includes: quantizing, by a quantizer corresponding to each of the individual color channel quantization branches, the calculated residual corresponding to the same color channel quantization branch to generate the first sample group of the quantized index.

4. The method according to claim 1, in, Each of the n-tuples of the first sample group of the quantized index is associated with a frequency of occurrence of the corresponding n-tuple in a set of quantized training images, and Each of the n-tuples includes quantized color indexes of the n color channels, where n is greater than 1.

5. The method according to claim 4, further comprising: generating, by the separate color channel quantization branch, a second sample group of quantized indexes based on a color value of a residual of a second specific pixel in the image frame, wherein the second sample group corresponds to a second n-tuple; and The entropy encoder generates an entropy code for the second specific pixel based on the data, wherein the second n-tuple is mapped to a second entropy coding index.

6. The method according to claim 5, wherein: The second specific pixel is a neighboring pixel relative to the first specific pixel in the image frame.

7. The method according to claim 5, wherein: The entropy coding includes Exponential-Golomb coding, Golomb-Rice coding, Huffman coding, or a combination of Exponential-Golomb coding and Golomb-Rice coding.

8. The method according to claim 1, wherein: The data is stored as an array.

9. The method according to claim 8, wherein: The generating of the entropy coding further comprises: calculating a first mode value based on the first n-tuple; searching the calculated first mode value in the array including the data to determine the corresponding first entropy encoding index; and The first entropy coding index is encoded.

10. The method according to claim 8, wherein: The generating of the entropy coding further comprises: calculating a first mode value based on the first n-tuple; Obtaining the first entropy coding index from the array according to the position of the array indicated by the first mode value; and The first entropy coding index is encoded.

11. The method according to claim 1, wherein: The data mapping the n-tuple of the first sample group of the quantized index to the entropy coded index is generated by the following steps: receiving a first training image frame; Calculating a training reconstruction error value corresponding to the first training image frame; generating the first sample group of the quantized index based on the calculated training reconstruction error value, wherein the reconstruction error value includes the n-tuple of the first sample group of the quantized index; identifying a frequency of occurrence of the n-tuple of the first sample group of the quantized index; and Based on the occurrence frequency of the n-tuple of the quantized index, an entropy coding index is assigned to each of the n-tuples of the first sample group of the quantized index.

12. The method according to claim 11, further comprising: The n-tuples of the first sample group of the quantized indexes are sorted according to the occurrence frequencies.

13. The method according to claim 11, wherein: Mapping the n-tuple of the first sample group of the quantized indexes to the data of the entropy coded indexes is performed during an offline process.

14. An encoder comprising: n separate color channel quantization branches configured to generate n-tuples of quantized indices, each of the branches corresponding to one of the n color channels of pixels of the image frame, wherein each color channel pixel of the n color channels comprises an inter-channel correlation between each other, and each of the quantized indices for the separate color channel quantization branches comprises the inter-channel correlation between each other; an entropy encoder configured to receive the n-tuple of the quantized indexes from each of the separate color channel quantization branches, and configured to entropy encode the n-tuple of the quantized indexes based on the inter-channel correlation of the pixels transferred to the inter-channel correlation of the quantized indexes and data mapping the n-tuple of the quantized indexes to entropy encoding indexes from each of the separate color channel quantization branches, and Each of the n color channel quantization branches is configured to ensure a reconstruction error with a non-negative value, wherein n is greater than 1.

15. The encoder according to claim 14, wherein: Each of the n-tuples of the quantized index is associated with a frequency of occurrence of the corresponding n-tuple in a set of quantized training images.

16. The encoder according to claim 14, wherein: Each of the n-tuples includes the quantized indices of the n color channels.

17. The encoder according to claim 16, wherein: The data is stored as an array.

18. The encoder according to claim 16, wherein: The entropy coding includes Exponential-Golomb coding, Golomb-Rice coding, Huffman coding, or a combination of Exponential-Golomb coding and Golomb-Rice coding.

19. An image compression and storage system, comprising: a memory storing computer executable instructions; as well as A processor configured to execute the instructions and cause the system to perform operations including the following steps: Receiving an image frame including pixels, wherein each of the pixels includes n color channels, and each color channel pixel of the n color channels includes inter-channel correlations between each other; generating, by a separate color channel quantization branch for each of the n color channels configured to ensure a reconstruction error having a non-negative value, a first sample group of quantized indexes based on a color value of a residual of a first specific pixel in the image frame, wherein each quantized index of the separate color channel quantization branch for the first sample group of quantized indexes includes the inter-channel correlation between each other, and wherein the first sample group corresponds to a first n-tuple; generating, by an entropy encoder, an entropy code for the first specific pixel based on the inter-channel correlation of the pixel transferred to the post-quantization index and data from each of the individual color channel quantization branches mapping an n-tuple of the first sample group of the post-quantization index to an entropy coding index, wherein the first n-tuple is mapped to a first entropy coding index, wherein each of the n-tuples of the first sample group of the quantized index is associated with an occurrence frequency of the corresponding n-tuple in a set of quantized training images; and Each of the n-tuples includes the first sample group of the quantized indexes of the n color channels, wherein n is greater than 1.

20. The system of claim 19, wherein: The instructions further cause the system to perform operations comprising: generating, by the separate color channel quantization branch, a second sample group of quantized indexes based on a color value of a residual of a second specific pixel in the image frame, wherein the second sample group corresponds to a second n-tuple; and The entropy encoder generates an entropy code for the second specific pixel based on the data, wherein the second n-tuple is mapped to a second entropy coding index.

Images