Method of compression and decompression and image compression and storage system

Abstract

The present invention relates to methods of compression and decompression, and image compression and storage systems. A method of compressing a frame in an image compression and storage system includes mapping original samples to mapped samples based on a bit depth of the original samples and a maximum allowed error, determining residuals of the mapped samples based on previously reconstructed samples of the mapped samples, applying a modulo addition to the residuals to generate biased residuals, quantizing the biased residuals based on the maximum allowed error to generate quantized biased residuals, and encoding values corresponding to the quantized biased residuals to generate encoded values.

Description

[0001] Cross-references to related applications

[0002] This application claims priority and benefit to U.S. Provisional Application No. 63 / 063,004 (“Compression with Positive Reconstruction Error”), filed August 7, 2020, and U.S. Provisional Application No. 63 / 062,963 (“DPCM Codec with Higher Reconstruction Quality at Significant Gray Levels”), filed August 7, 2020, the entire contents of which are incorporated herein by reference.

[0003] This application also relates to U.S. Patent Application No. 17 / 070,893 (“Compression with Positive Reconstruction Error”), filed October 14, 2020, which claims priority and benefit to U.S. Provisional Patent Applications Nos. 63 / 063,004 and 63 / 062,963. Technical Field

[0004] The embodiments of this disclosure generally relate to image processing systems and methods. Background Technology

[0005] Flat panel displays, such as liquid crystal displays (LCDs) and light-emitting diode (LED) displays, are becoming increasingly common due to their small size, excellent image quality, and low power consumption.

[0006] Pixels in a display panel have specific response times. The data provided to a pixel for display may change faster than the pixel's response capability. This can lead to undesirable effects such as motion blur or ghosting. To make the displayed image more pleasing, image compensation techniques are necessary. Overdrive algorithms have been developed to compensate for image data in order to reduce motion blur exhibited by display pixels.

[0007] The information disclosed in this background section is only intended to enhance the understanding of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0008] Embodiments of this disclosure relate to systems and methods for image compression, storage, and decompression capable of ensuring bounded and non-negative reconstruction errors. In some embodiments, the system achieves this by partially encoding an image or frame using an appropriate quantization band size and an added bias. These systems and methods can improve the image compensation performance of overdriven systems.

[0009] Aspects of embodiments of the present disclosure relate to systems and methods for image compression, storage, decompression that can ensure zero reconstruction error for certain high priority gray scale values or at least lower reconstruction error for certain high priority gray scale values relative to other gray scale values.

[0010] According to some embodiments of the present disclosure, a method of compressing a frame in an image compression and storage system is provided, the method comprising: mapping an original sample to a mapped sample based on a bit depth of the original sample and a maximum allowed error to ensure a minimum reconstruction error for a highest priority sample value; determining a residual for the mapped sample based on a previously reconstructed sample of the mapped sample; applying a modulo addition to the residual to generate a biased residual; quantizing the biased residual based on the maximum allowed error to generate a quantized biased residual; and encoding a value corresponding to the quantized biased residual to generate an encoded value.

[0011] In some embodiments, the highest priority sample value is 0 and mapping the original sample comprises calculating the mapped sample as:

[0012]

[0013] where f(x) represents the mapped sample, x represents the original sample, δ represents the maximum allowed error, c is a constant non-negative integer less than δ, and maxVal represents a maximum value of the original sample.

[0014] In some embodiments, the maximum value of the original sample is represented as:

[0015] maxVal = 2 bitdepth -1

[0016] where bitdepth represents a bit depth of the original sample, where the original sample corresponds to a color value of a pixel of the frame, and where the frame has a red, green, and blue (RGB) format or a red, green, blue, and green (RGBG) format.

[0017] In some embodiments, the bit depth of the original sample is 3, c is equal to 0, and the maximum allowed error is 1.

[0018] In some embodiments, the highest priority sample value is greater than 0 and mapping the original sample comprises calculating the mapped sample as:

[0019]

[0020] where f(x) represents the mapped sample, i is the highest priority sample value, x represents the original sample, δ represents the maximum allowed error, c is a constant non-negative integer less than δ, and maxVal represents a maximum value of the original sample.

[0021] In some embodiments, determining the residual of the mapped sample comprises subtracting the mapped previous reconstructed sample from the mapped sample to generate the residual of the mapped sample.

[0022] In some embodiments, the value corresponding to the quantized bias residual is a quantization index corresponding to the quantized bias residual.

[0023] In some embodiments, the method further comprises performing inverse quantization on the quantized bias residual to generate a reconstructed bias residual; adding the mapped previous reconstructed sample to the reconstructed bias residual to generate a reconstructed bias sample; applying a modulo subtraction to the reconstructed bias sample to generate a mapped reconstructed sample; and inverse mapping the mapped reconstructed sample to an unmapped reconstructed sample based on the highest priority sample value, a bit depth of the original sample, and a maximum allowed error.

[0024] In some embodiments, the highest priority sample value is 0, and inverse mapping the mapped reconstructed sample comprises calculating the unmapped reconstructed sample as:

[0025]

[0026] where g(x') represents the unmapped reconstructed sample, x' represents the mapped reconstructed sample, δ represents the maximum allowed error, c is a constant non-negative integer less than δ, maxVal represents a maximum value of the original sample, and Clip(0, maxVal, x'-δ+c) represents a clipping function that limits a value of x'-δ+c to a minimum of 0 and a maximum of maxVal.

[0027] In some embodiments, the highest priority sample value is greater than 0, and inverse mapping the mapped reconstructed sample comprises calculating the unmapped reconstructed sample as:

[0028]

[0029] where g(x') represents the unmapped reconstructed sample, i is the highest priority sample value, x' represents the mapped reconstructed sample, δ represents the maximum allowed error, c is a constant non-negative integer less than δ, maxVal represents a maximum value of the original sample, Clip(0, maxVal, x'-δ+c) represents a clipping function that limits a value of x'-δ+c to a minimum of 0 and a maximum of maxVal, and Clip(0, maxVal, x'-δ+c-1) represents a clipping function that limits a value of x'-δ+c-1 to a minimum of 0 and a maximum of maxVal.

[0030] In some embodiments, the method further comprises generating a prediction of the unmapped previous reconstructed sample based on the unmapped reconstructed sample; and mapping the prediction of the unmapped previous reconstructed sample to the mapped previous reconstructed sample.

[0031] In some embodiments, the method further includes generating a prediction of the mapped previously reconstructed sample based on the mapped reconstructed sample.

[0032] In some embodiments, the reconstruction error of the highest priority sample value is less than or equal to the reconstruction error of other values of the original sample.

[0033] In some embodiments, applying the modulo addition to the residual to generate the biased residual is based on a bit depth of the original sample and a maximum allowed error.

[0034] According to some embodiments of the present invention, there is provided a method of decompressing stored image data corresponding to a frame in an image compression and storage system, the method comprising: decoding an encoded value corresponding to an original sample of the frame to generate a decoded value corresponding to a quantized biased residual; performing inverse quantization on the decoded value to generate a reconstructed biased residual; adding a prediction of a mapped previously reconstructed sample to the reconstructed biased residual to generate a reconstructed biased sample; applying a modulo subtraction to the reconstructed biased sample to generate a mapped reconstructed sample; and inverse mapping the mapped reconstructed sample to an unmapped reconstructed sample based on a highest priority sample value, a bit depth of the original sample, and a maximum allowed error.

[0035] In some embodiments, the highest priority sample value is 0, and inverse mapping the mapped reconstructed sample includes calculating the unmapped reconstructed sample as:

[0036]

[0037] where g(x') represents the unmapped reconstructed sample, x' represents the mapped reconstructed sample, δ represents the maximum allowed error, c is a constant non-negative integer less than δ, maxVal represents a maximum value of the original sample, and Clip(0, maxVal, x' - δ + c) represents a clipping function that limits the value of x' - δ + c to a minimum of 0 and a maximum of maxVal.

[0038] In some embodiments, the highest priority sample value is greater than 0, and inverse mapping the mapped reconstructed sample includes calculating the unmapped reconstructed sample as:

[0039]

[0040] where g(x') represents the unmapped reconstructed sample, i is the highest priority sample value, x' represents the mapped reconstructed sample, δ represents the maximum allowed error, c is a constant non-negative integer less than δ, maxVal represents a maximum value of the original sample, Clip(0, maxVal, x'-δ+c) represents a clipping function that limits a value of x'-δ+c to a minimum of 0 and a maximum of maxVal, and Clip(0, maxVal, x'-δ+c-1) represents a clipping function that limits a value of x'-δ+c-1 to a minimum of 0 and a maximum of maxVal.

[0041] In some embodiments, the method further includes generating a prediction of the mapped previous reconstructed sample based on the unmapped reconstructed sample.

[0042] According to some embodiments of the present disclosure, there is provided an image compression and storage system, comprising: a processor; and a processor memory local to the processor, wherein the processor memory has instructions stored on the processor memory that, when executed by the processor, cause the processor to perform: mapping original samples to mapped samples based on a bit depth of the original samples and a maximum allowed error to ensure a minimum reconstruction error of a highest priority sample value; determining residuals of the mapped samples based on mapped previous reconstructed samples; applying modulo addition to the residuals to generate biased residuals; quantizing the biased residuals based on the maximum allowed error to generate quantized biased residuals; encoding values corresponding to the quantized biased residuals to generate encoded values for storage in a storage medium; decoding the encoded values to generate decoded values corresponding to the quantized biased residuals; performing inverse quantization on the decoded values to generate reconstructed biased residuals; adding a prediction of the mapped previous reconstructed samples to the reconstructed biased residuals to generate reconstructed biased samples; applying modulo subtraction to the reconstructed biased samples to generate mapped reconstructed samples; and inverse mapping the mapped reconstructed samples to unmapped reconstructed samples based on the highest priority sample value, the bit depth of the original samples, and the maximum allowed error.

[0043] In some embodiments, a difference between the original samples and the unmapped reconstructed samples for the highest priority sample value is less than or equal to a difference between the original samples and the unmapped reconstructed samples for other values of the original samples. BRIEF DESCRIPTION OF DRAWINGS

[0044] The accompanying drawings are included to provide a further understanding of examples of the present disclosure and are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and serve to explain the principles of the present disclosure.

[0045] Figure 1 is a schematic diagram of a frame compensation system utilizing an image compression and storage system according to some embodiments of the present disclosure.

[0046] Figure 2Ais a schematic diagram of an encoder of an image compression and storage system according to some embodiments of the present disclosure.

[0047] Figure 2B is a schematic diagram of a decoder of an image compression and storage system according to some embodiments of the present disclosure.

[0048] Figure 3A is a schematic diagram illustrating a part of an encoder and a part of a decoder according to some embodiments of the present disclosure.

[0049] Figure 3B is Figure 3A is a simplified diagram, Figure 3B illustrates identity operation of an entropy encoder and an entropy decoder according to some embodiments of the present disclosure.

[0050] Figure 4A illustrates quantization tables utilized by a quantizer and an inverse quantizer according to some embodiments of the present disclosure.

[0051] Figure 4B illustrates operation of a quantizer and an inverse quantizer according to some examples of the present disclosure.

[0052] Figure 4C and Figure 4D illustrate uniform quantization tables and non-uniform quantization tables utilized by a quantizer and an inverse quantizer, respectively, according to some embodiments of the present disclosure.

[0053] Figure 5A is a schematic diagram of an encoder with in-loop mapping according to some embodiments of the present disclosure.

[0054] Figure 5B is a schematic diagram of an encoder with off-line / out-of-loop mapping according to some embodiments of the present disclosure.

[0055] Figure 6A and Figure 6B are schematic diagrams of a decoder with in-loop mapping and a decoder with off-line / out-of-loop mapping, respectively, according to some embodiments of the present disclosure.

[0056] Figure 7A and Figure 7B are tables illustrating numerical examples of potential reconstruction errors of different sample values given different values of a maximum allowed error δ and a constant c according to some embodiments of the present disclosure.

[0057] Figure 8 is a table illustrating numerical examples of potential reconstruction errors of different sample values given different significance of a non-zero sample value according to some embodiments of the present disclosure.

[0058] Figure 9A and Figure 9BThis is a table illustrating numerical examples of potential reconstruction errors for different sample values ​​of the mapping and inverse mapping functions of an image compression and storage system that does not guarantee non-negative reconstruction errors, according to some embodiments of the present disclosure. Detailed Implementation

[0059] The detailed description set forth below is intended as a description of exemplary embodiments of systems and methods for image compression, storage, and decompression provided in this disclosure, and is not intended to represent the only form in which this disclosure can be constructed or utilized. The description, in conjunction with illustrated embodiments, illustrates the features of this disclosure. However, it should be understood that the same or equivalent functions and structures can be implemented by different embodiments that are also intended to be included within the scope of this disclosure. As referred to elsewhere herein, the same reference numerals are intended to indicate the same elements or features.

[0060] Figure 1 This is a schematic diagram of a frame compensation system 1 utilizing an image compression and storage system 10 according to some embodiments of the present disclosure.

[0061] In some embodiments, the image compression and storage system 10 includes an encoder 100 for encoding (e.g., compressing) data to reduce its size, a storage medium 200 for storing the encoded data, and a decoder 300 for decoding (e.g., decompressing) the stored data to reconstruct it. Storing certain types of data, such as 4K image frames, consumes significant amounts of memory. Here, the encoder 100 and decoder 300 allow for the use of a smaller storage medium 200, which can result in space and cost savings. Figure 1As illustrated in the middle, in some examples, the image compression and storage system 10 can form part of a frame compensation system 1 in which the overdrive system 20 uses a previous frame to compensate a current frame (e.g., to reduce ghosting effects or blurring). In some examples, the truncation and delay circuit 30 receives a current frame (e.g., an 8-bit per color (bpc) frame), truncates the current frame (i.e., reduces the number of bpc of the pixel values of the frame, e.g., to 3 bpc) and delays the current frame to produce a truncated previous frame. The image compression and storage system 10 then compresses and stores the truncated previous frame for later consumption by the overdrive system 20. Here, the truncation and compression can make the stored frame significantly smaller in size than the original (previous) frame. For example, in the case of an RGB input frame, where each color value is represented with 8 bits (or 24 bits per pixel (bpp)), the truncation can result in a 3 bpc / 9 bpp frame representation, and a 1.5: 1 compression performed by the encoder 100 can further reduce this size to 2 bpc / 6 bpp. However, embodiments of the present disclosure are not limited to RGB input frames, and any suitable input frame format can be used, such as RGBG (for a five-cell type sub-pixel arrangement). For example, in the case of an RGBG input frame, the truncation can reduce the frame size from 16 bpp to 6 bpp, and a 1.5: 1 compression performed by the encoder 100 can further reduce the size to 4 bpp. Moreover, embodiments of the present disclosure are not limited to using the above truncation operation, and in some examples, the truncation and delay circuit 30 can be replaced with a delay block, and the truncation operation can be omitted from the process described herein.

[0062] The performance of the overdrive system 20 can be improved if the reconstruction error (i.e., the error introduced by the image compression and storage system 10, which is all quantization error) is only non-negative. State-of-the-art encoding schemes typically optimize the mean square error (MSE) or the peak signal-to-noise ratio (PSNR), which means that the polarity of the error in the reconstructed image can be both positive and negative. According to some embodiments, the image compression and storage system 10 improves the performance of the overdrive system 20 relative to the prior art by ensuring that the reconstruction error (i.e., the quantization error), defined as the difference between the reconstructed frame and the original frame, is always non-negative (i.e., greater than or equal to zero) and is also limited to a set maximum value.

[0063] Throughout this disclosure, a frame refers to data of an image frame comprising a plurality of pixels. Each pixel of a frame can be represented with a plurality of color values (e.g., red, green, and blue).

[0064] Figure 2A is a schematic diagram of an encoder 100 of the image compression and storage system 10 according to some embodiments of the present disclosure. Figure 2Bis a schematic diagram of a decoder 300 of an image compression and storage system 10 according to some embodiments of the present disclosure.

[0065] Referring to Figure 2A According to some embodiments, the encoder 100 utilizes a predictive coding scheme in which residuals of original samples (e.g., truncated frames) are coded instead of the original samples themselves. The residual (also referred to as error e) is defined as the difference between the original sample and a prediction (e.g., a prediction of a previous sample) based on one or more neighboring samples. In some embodiments, the sample predictor 110 generates a prediction of a previous reconstructed sample (or predicted sample), and the subtracter 120 computes a residual based on the original sample and the predicted sample. The computed residual can be a positive or negative value. The modulo adder 130 adds a bias value to the residual in order to reduce the signaling cost of 1-bit sign information. The quantizer 140 quantizes the biased residual to generate a quantized biased value (or quantized biased residual), and passes a corresponding quantization index to the entropy encoder 150 for further compression (e.g., lossless compression such as entropy encoding). This compressed data (or encoded value) is stored in the storage medium 200. As will be explained further below, in some embodiments, the bias value is a function of the quantization step size.

[0066] According to some embodiments, to generate the predicted sample, the inverse quantizer 160 performs an inverse operation of the operation performed by the quantizer 140 to generate a reconstructed biased residual, the adder 170 adds a prediction of a previous reconstructed sample to the reconstructed biased residual to generate a reconstructed biased sample, and the modulo subtracter 180 subtracts the value added by the modulo adder 130 to generate the reconstructed sample. In some examples, a clipping function can be applied to the reconstructed sample to ensure that the reconstructed sample is within a desired dynamic range, which can be represented as [0, 2 bitdepth -1] for a given bit depth bitdepth. This reconstructed sample (e.g., clipped reconstructed sample) is used by the sample predictor 110 to predict the next sample.

[0067] Referring to Figure 2B In some embodiments, the decoder 300 includes an entropy decoder 310 that decompresses the stored samples (or stored encoded values) received from the storage medium 200, and the same inverse quantizer 160, adder 170, modulo subtracter 180, and sample predictor 110 as described above. Except for the entropy decoder 310, the rest of the components of the decoder 300 are the same as and configured in the same manner as in the encoder 100. As such, for the sake of brevity, their detailed descriptions will not be repeated here.

[0068] Since the decoder 300 has no access to the original samples and can only access the reconstructed samples, the sample predictor 110 of the decoder 300 uses the reconstructed samples to generate a prediction of a previous reconstructed sample. Moreover, since the operations of the encoder 100 and the decoder 300 mirror each other, the sample predictor 110 of the encoder 100 also uses the same reconstructed samples to generate a prediction of a previous reconstructed sample, so that the encoder 100 can be synchronized with the decoder 300, even though the encoder 100 does have access to the original samples.

[0069] In some embodiments, the encoder 100 and the decoder 300 operate on a single color value of a pixel. Thus, multiple encoders 100 and decoders 300 can operate in parallel (e.g., concurrently / simultaneously) to process individual colors of a pixel.

[0070] Figure 3A is a schematic diagram illustrating a portion of an encoder and a portion of a decoder according to some embodiments of the present disclosure. Figure 3B is a simplified diagram of Figure 3A Figure 3B illustrates the identity operation of an entropy encoder and an entropy decoder according to some embodiments of the present disclosure.

[0071] According to some embodiments, the operations performed by the components of the encoder 100 and the decoder 300, other than the quantizer 140, are lossless operations. For example, Figure 3A The sequential operation of the entropy encoder 150 and the entropy decoder 310 shown in Figure 3B is an identity operation and can thus be practically omitted, resulting in

[0072] Error (residual) = R - e' = Q[e'] - e' Equation (1)

[0073] where e' is the biased residual input to the quantizer 140, R is the output of the inverse quantizer 160, and Q[.] is the quantization and inverse quantization function of the combination of the quantizer 140 and the inverse quantizer 160.

[0074] ​Accordingly, if the error in the residual space is kept non-negative, the error in the signal space will also be non-negative. Furthermore, if the error in the residual space is bounded (i.e., R - e' < δ, where δ is a positive integer representing the maximum allowed error), the error in the signal space will also be limited to δ. This relationship holds even in the presence of the modulo adder 130 and the modulo subtractor 180. In contrast, transform coding used in the prior art (e.g., High Efficiency Video Coding (HEVC), JPEG-2000, or VESA Display Compression-M (VDC-M)) can not be suitable to ensure non-negative reconstruction error because positive error in the transform domain does not ensure non-negative reconstruction error in the signal space. This can be due to the fact that a transform such as a discrete cosine transform (DCT) preserves the L2 norm (or energy) between the original coefficients and the transform coefficients, not the L1 norm.

[0075] Accordingly, in some embodiments, the modulo adder 130 and the quantizer 140 are configured in a manner that ensures the error in the residual space is kept non-negative and bounded (i.e., limited to δ) in a manner that ensures the error in the signal space is kept non-negative and bounded.

[0076] The quantizer 140 and the inverse quantizer 160 ensure non-negative, bounded error by adjusting the quantization band size and the reconstruction points in each band appropriately. In some embodiments, given the maximum allowed error δ, the quantizer 140 utilizes uniform or non-uniform quantization bands with a maximum quantization step size of δ + 1 and uses the reconstruction point at the right edge (i.e., the maximum value) of the band. Furthermore, in some embodiments, the value zero is at the right edge of the band (e.g., the first quantization band with index 1) so that zero appears in the reconstructed values. This can be due to the fact that the residual peaks at zero.

[0077] Figure 4A FIGURE 1 illustrates a quantization table utilized by the quantizer 140 and the inverse quantizer 160 according to some embodiments of the present disclosure. Figure 4B FIGURE 2 illustrates the operation of the quantizer 140 and the inverse quantizer 160 with input e' and output R according to some examples of the present disclosure. Figure 4C and Figure 4D FIGURES 3 and 4 respectively illustrate a uniform quantization table and a non-uniform quantization table utilized by the quantizer 140 and the inverse quantizer 160 according to some embodiments of the present disclosure.

[0078] Referring to Figure 4A , the quantizer 140 uses uniform bands with a quantization step size of δ + 1 with a reconstruction point at the right edge of each band. In some embodiments, the first quantization band has a right edge of zero (i.e., a reconstruction point of zero). Here, the biased residual e' is an integer. In some embodiments, when using the modulo add parameters and the modulo subtract parameters defined in equation 6 and equation 8 (also referred to as the optimized modulo add parameters and the optimized modulo subtract parameters), the number of bands NB can be represented as

[0079]

[0080] where δ represents the maximum allowed error and Ceiling(.) represents the ceiling function. Here, δ + 1 can be the maximum allowed step size, i.e., a smaller step size can be chosen. In some other embodiments, when using non-optimized modulus addition and modulus subtraction parameters (i.e., parameters not defined by Equation 6 and Equation 8), the number of bands NB can be represented as

[0081]

[0082] In some embodiments, when quantizing the bias residual, the quantizer 140 computes the quantized index of the bias residual as:

[0083]

[0084] where -δ≤e'≤2 bitdepth - 1 otherwise, when using non-optimized modulus addition and modulus subtraction parameters. bitdepth - 1 + δ.

[0085] I[e'] represents the quantized index, Ceiling(.) represents the ceiling function, e' represents the bias residual, and δ represents the maximum allowed error.

[0086] Similarly, performing inverse quantization on the quantized bias residual can include computing the reconstructed bias residual as

[0087] Q[e'] = (δ + 1) I[e'] Equation (5)

[0088] where Q(e') represents the reconstructed bias residual, I[e'] represents the quantized index, and e' represents the bias residual. Equation 2 and Equation 3 apply to embodiments that utilize uniform quantization. When non-uniform quantization is used, the maximum step size can be δ + 1.

[0089] Figure 4B Equation 2 and Equation 3 are illustrated in an example where δ = 2 and the bit depth (i.e., the number of bits per color) is 3. Here, the bit depth refers to the color information stored in an image and affects the dynamic range of the image. The higher the bit depth of an image, the more colors the image can store. The simplest image (1-bit image) can only display two colors, black and white. This is because 1 bit can only store one of two values (0 (white) and 1 (black)). As the bit depth increases, the dynamic range also increases, which increases the number of quantization bands.

[0090] As Figure 4BAs illustrated in the example, a bias residual e' of 1 results in a reconstructed value of 3.

[0091] According to some embodiments, the modulus adder 130 calculates the bias residual e' as:

[0092]

[0093] This means that the bias residual e' is within the range that can be expressed as the following formula:

[0094] -δ≤e′≤2 bitdepth -1 Formula (7)

[0095] The upper limit of the bias residual e' allows for the reduction of errors in certain bias values ​​and thus improves the performance of non-uniform quantization bands.

[0096] For example, when δ = 2 and bit depth = 3, -2 ≤ e′ ≤ 7, and Figure 4C The uniform quantization table shown and Figure 4D The non-uniform quantization table shown can be utilized by quantizer 140. Figure 4C In this context, a uniform band size of 2 is used, and the quantization error of all four bands is defined by δ (i.e., 2). However, embodiments of this disclosure also allow... Figure 4D The non-uniform bands of the quantization table shown have smaller quantization errors in the third and fourth bands, defined by 1 (instead of 2). In some examples, quantizer 140 reserves narrower bands for those that are statistically more important (e.g., the bands with more bias residuals, or the bands whose values ​​have a greater impact on the performance of overdrive system 20). Thus, the smaller quantization error achieved by the operation of modulo adder 130 can improve the performance of overdrive system 20.

[0097] Modular subtractor 180 can perform the opposite operation of modular adder 130 by removing the bias value added to the residual by modular adder 130. According to some embodiments, for a given input x, the output y of modular subtractor 180 is represented as...

[0098]

[0099] As noted above, in some examples, the shearing function can be applied to the reconstructed sample to ensure that the reconstructed sample is within the desired dynamic range, and for a given bit depth, the reconstructed sample can be represented as [0, 2]. bitdepth -1]. In the 8bpc example, the shearing function can restrict the output of the modulo subtractor 180 to [0, 7].

[0100] Thus, as described above, by using appropriate quantization band sizes and increased bias, the image compression and storage system 10 ensures (e.g., guarantees) a non-negative and bounded error on the reconstructed image, which can improve the performance of the overdrive system 20.

[0101] In addition to ensuring a bounded, non-negative reconstruction error, according to some embodiments, it is desirable for the reconstruction error to be zero or at least less than the error for other gray values for certain gray values (e.g., the gray value of zero). For example, for a color depth of three, the range of gray values can be from 0 to 7. In some examples, as the gray value drops from 7 to 0, the overdrive compensation can assign greater importance (higher priority) to the gray values, with 0 having the highest importance (highest priority). In such examples, a high reconstruction error can be acceptable for low priority gray values such as 5, 6, and 7. This is in contrast to prior art codecs in which all gray values are treated equally and the reconstruction error is independent of the magnitude (or gray value) of the pixel sample.

[0102] According to some embodiments, the image compression and storage system 10 is configured to ensure a smaller reconstruction error for important gray values, as well as a non-negative reconstruction error on other gray values. In some embodiments, the encoder 100 modifies the original gray values in order to achieve a lower error for certain gray values. This can come at the expense of higher error on other unimportant gray values.

[0103] Figure 5A is a schematic diagram of an encoder 400 with in-loop mapping according to some embodiments of the present disclosure. Figure 5B is a schematic diagram of an encoder 400-1 with off-line / out-of-loop mapping according to some embodiments of the present disclosure. Figure 6A and Figure 6B are schematic diagrams of a decoder 500 with in-loop mapping and a decoder 500-1 with off-line / out-of-loop mapping, respectively, according to some embodiments of the present disclosure.

[0104] According to some embodiments, the encoder 400 / 400-1 includes a mapper (e.g., a gray value mapping circuit) 202 that maps the gray value x of an original sample to a mapped value (also referred to as a mapped sample) f(.) prior to encoding, and further includes an inverse mapper (e.g., an inverse gray value mapping circuit) 204 that performs the inverse operation g(.) of the mapper 202. Thus, the encoder 400 / 400-1 encodes the original sample in the mapped space (rather than the signal space). Here, the dynamic range of the mapped space can be the same as that of the original space. In other words, the bit depth of the mapped space and the original space can be the same.

[0105] According to some embodiments, in Figure 5AIn the in-loop implementation illustrated in the diagram, encoder 400 includes a mapper 202 that operates on the original samples before adder 170, a second mapper 203 that operates on the output of sample predictor 110, and further includes an inverse mapper 204 that operates on the output of modulo subtractor 180. In these embodiments, inverse mapper 204 performs an inverse mapping operation on the reconstructed samples of the mapping generated by modulo subtractor 180 and generates unmapped reconstructed samples, which are used by sample predictor 110 to produce predictions of the unmapped previously reconstructed samples. Second mapper 203 maps the prediction of the unmapped previously reconstructed samples to the mapped previously reconstructed samples for subtraction from the mapped samples at subtractor 120.

[0106] In some embodiments, Figure 5B In the offline / out-of-loop implementation illustrated in the diagram, the inverse mapper 204 does not operate on the input of the sample predictor 110, and thus, the second mapper 203 does not need to operate on the output of the sample predictor 110. In such an embodiment, the mapping operation is performed entirely outside the encoder loop. Since the operations of mapper 202 and inverse mapper 204 are performed outside the codec, this out-of-loop implementation may be easier to leverage with existing codecs. Furthermore, due to the fewer operations / computations performed, the out-of-loop implementation has lower complexity compared to the in-loop implementation. However, the in-loop implementation may result in higher performance.

[0107] like Figure 6A and Figure 6B As illustrated in the diagram, decoder 500 for in-ring mapping implementation utilizes both a second mapper 203 and an inverse mapper 204, while decoder 500-1 for offline mapping implementation utilizes a single inverse mapper 204 (without requiring a mapper).

[0108] In either implementation, the encoder 400 / 400-1 includes the same components having the same configuration as the encoder 100, except for the mappers 202 and 203 and the inverse mapper 204. Similarly, in both the online and offline implementations, the decoder 500 / 500-1 includes the same components having the same configuration as the decoder 300, except for the second mapper 203 and the inverse mapper 204. As such, for the sake of brevity, the description of the elements in the encoder 400 / 400-1 and the decoder 500 / 500-1 that are the same as the encoder 100 and the decoder 300, respectively, will not be repeated here. Although the modulo adder 130, the quantizer 140, the inverse quantizer 160, and the modulo subtractor 180 ensure bounded non-negative reconstruction errors, the encoders 400 and 400-1 and the decoders 500 and 500-1 are not limited thereto. For example, when negative reconstruction errors can be tolerated (e.g., by the overdriven system 20), any suitable modulo adder, quantizer, inverse quantizer, and modulo subtractor can be used.

[0109] According to some embodiments, when the gray scale value of zero is important (e.g., when a reconstruction error of zero is desired), the mapping function f(.) of the mapper 202 is represented as:

[0110]

[0111] where δ represents the maximum allowed error between a reconstructed sample and an original sample, c is a constant non-negative integer that is less than δ (i.e., c < δ), and maxVal is the maximum value of the mapper input sample (e.g., the original sample) x for a given bit depth (or bits per color), maxVal can be represented as:

[0112] maxVal = 2 bitdepth -1 Equation (10)

[0113] Further, the inverse mapping function g(.) of the inverse mapper 204 can be represented as:

[0114]

[0115] where x' represents the input to the inverse mapper 204 (e.g., the mapped reconstructed sample output by the modulo subtractor 180), and Clip(0, maxVal, x' - δ + c) is a clipping function that limits the output value x' - δ + c to the minimum value of zero and the maximum value of maxVal. In other words, for all values of x' - δ + c that are less than zero, the clipping function produces zero, for all values of x' - δ + c that are greater than maxVal, the clipping function produces maxVal, and otherwise produces x' - δ + c.

[0116] As will be explained below, when the constant c = 0, there is no reconstruction error at the original sample value of 0, although this can come at the cost of increasing the error for other original sample values beyond the maximum allowed error δ.

[0117] Figure 7A and Figure 7B is a table illustrating numerical examples of potential reconstruction errors for different sample values given a maximum allowed error δ and different values of the constant c according to some embodiments of the present disclosure. In Figure 7A and Figure 7B In the table illustrated in Figure 5A , the value x' represents a possible output of the modulo subtractor 180 in an online implementation of the encoder 400 (of

[0118] In Figure 7A , it is assumed that the bit depth is 3 (i.e., the original sample value range from 0 to 7), the maximum allowed error δ = 1 and the constant c = 0. In this table, the value x' for each value of x can have one or more possibilities depending on the predicted reconstructed sample, x. For example, in the case of x = 3 and f(x) = 4, there can be multiple cases: 1) when the sample predictor 110 predicts a value of 3, the residual e' is 1 (= 4 - 3), the quantization index = 1, the output R of the inverse quantizer 160 = 2, the output x' of the modulo subtractor 180 = 5 (= 3 + 2), resulting in a reconstructed value y = 4; 2) when the sample predictor 110 predicts a value of 2, the residual e' = 2 (= 4 - 2), the quantization index = 1, R = 2, x' = 4 (= 2 + 2), resulting in a reconstructed value y = 3; and 3) when the sample predictor 110 predicts a value of 5, the residual e' = -1 (= 4 - 5), the quantization index = 0, R = 0, x' = 5 (= 0 + 5), again resulting in a reconstructed value y = 4.

[0119] As is apparent from Figure 7A , in the case of the constant c = 0, the reconstruction error for the original sample of 0 (e.g., the statistically most important sample value) is zero. However, the error for the original sample 5 (e.g., a low-priority sample) can reach 2, which is larger than the maximum allowed error δ of 1.

[0120] In Figure 7B , it is assumed that the bit depth is 3, the maximum allowed error δ = 2, and the constant c = 1. Here, in the case of the constant c being larger than zero, it is no longer guaranteed that the possible reconstruction error for the original sample of 0 is zero; however, the maximum error at the original sample 0 is still lower than or equal to the maximum error for other original sample values.

[0121] According to some embodiments, when the gray scale values for i > 0 are important, the mapping function f(.) of the mapper 202 is represented as:

[0122]

[0123] And the inverse mapping function g(.) of the inverse mapper 204 can be expressed as:

[0124]

[0125] Figure 8 Table 1 is a numerical example illustrating potential reconstruction errors for different sample values given the importance of a non-zero sample value according to some embodiments of the present disclosure.

[0126] In the example of Figure 8 , it is assumed that the bit depth is 3, the maximum allowed error δ = 1, and the constant c = 0. In this example, the original sample 4 can be statistically more important to the overdrive system 20, and the reconstruction error for this sample is zero. However, this comes at the cost of the maximum error for the original samples 3 and 5 being greater than desired.

[0127] In the example of Figure 7A and Figure 7B and Figure 8 , the encoder 400 / 400-1 and the decoder 500 / 500-1 ensure non-negative reconstruction errors. However, in examples where negative reconstruction errors can be tolerated (e.g., embodiments utilizing any suitable modulo adder / subtractor and quantizer / inverse quantizer, not necessarily embodiments of the encoder 400 / 400-1 and the decoder 500 / 500-1), and the most important original sample value is zero, the mapping function F(.) and the inverse mapping function G(.) of the functions f(.) and g(.) can be expressed as:

[0128] F(x) = maxVal - x Equation (14)

[0129]

[0130] In such examples, the mapping function F(.) and the inverse mapping function G(.) can be alternatively expressed as:

[0131]

[0132]

[0133] Figure 9A Table 1 is a numerical example illustrating potential reconstruction errors for different sample values given the importance of a non-zero sample value according to some embodiments of the present disclosure. Figure 9B Table 2 is a numerical example illustrating potential reconstruction errors for different sample values given the importance of a non-zero sample value according to some embodiments of the present disclosure. In Figure 9A and Figure 9BIn the table, it is assumed that the bit depth = 3 and the maximum allowed error δ = 1. Here, from the perspective of the overdrive system 20, the original sample 0 is the most important, and the reconstruction error of this sample is zero in both tables.

[0134] Generally, in the related art image / video encoders, such as DSC and DPCM, there is no particular emphasis on certain gray values. Therefore, the size of the reconstruction error is generally independent of the gray value. Moreover, when quantization is applied to the transform coefficients in the related art, the transform coding used in codecs such as HEVC, JPEG-2000, and VDC-M is not suitable to satisfy the non-negative error, and the positive error in the transform domain does not imply the positive error in the signal space.

[0135] In contrast, according to some embodiments of the present disclosure, the image compression and storage system provides higher priority to certain gray values and exhibits lower reconstruction error for these gray values. This can be achieved by performing pixel-wise mapping of the original image to another space and compressing the image in the mapped space. In addition to keeping lower error for important gray values, in some embodiments, the encoder and decoder of the image compression and storage system also ensure the non-negative error of the reconstructed image, which can improve the performance of the corresponding overdrive system.

[0136] The operations performed by the constituent components of the encoder 100 / 400 / 400-1 and the decoder 300 / 500 / 500-1 can be performed by “processing circuitry” or “processors” that can include any combination of hardware, firmware, and software for processing data or digital signals. Processing circuitry hardware can include, for example, application specific integrated circuits (ASICs), general or special purpose central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), and programmable logic devices such as field programmable gate arrays (FPGAs). In processing circuitry, as used herein, each function is performed by hardware configured (i.e., hardwired) to perform that function, or by more general hardware (such as a CPU) configured to execute instructions stored in non-transitory storage media. Processing circuitry can be fabricated on a single printed wiring board (PWB) or distributed among several interconnected PWBs. Processing circuitry can contain other processing circuitry; for example, processing circuitry can include two processing circuitries, an FPGA and a CPU, interconnected on a PWB.

[0137] It will be understood that, although the terms“first,”“second,”“third,” etc. can be used herein to describe various elements, components, regions, layers and / or sections, these elements, components, regions, layers and / or sections should not be limited by these terms. These terms are used to distinguish one element, component, region, layer or section from another element, component, region, layer or section. Thus, a first element, component, region, layer or section discussed below could be termed a second element, component, region, layer or section without departing from the scope of the inventive concept.

[0138] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the inventive concept. As used herein, the singular forms“a,”“an” and“the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms“comprises” and / or“comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein, the term“and / or” includes any and all combinations of one or more of the associated listed items. In addition, use of“can,”“may,”“might,”“could,”“would,”“should,”“might,”“will,” and the like, that which describes existent possibilities or one or more implementations of the inventive concept, is not intended to limit or restrict the scope of the inventive concept to the described possibilities or implementations. Moreover, the terms“exemplary,”“by way of example,” and“for example” are used herein to mean“an example of” among other things. Any implementation of the inventive concept in the form of examples described herein is not meant to limit or restrict the scope of the inventive concept to the described examples.

[0139] As used herein, the terms“use,”“in use,” and“used” can be considered synonymous with the terms“utilize,”“in utilization,” and“utilized,” respectively.

[0140] For the purposes of this disclosure, “at least one of X, Y, and Z” and“at least one selected from the group consisting of X, Y, and Z” can be interpreted to include only X, only Y, only Z, or any combination of two or more of X, Y, and Z, such as, for example, XYZ, XYY, YZ, and ZZ.

[0141] While the disclosure has been described in detail and with reference to illustrative embodiments thereof, it will be understood that the embodiments described are not limited to the precise embodiments and that various changes and modifications can be practiced. It is intended that the scope of the disclosure be limited only by the appended claims and their equivalents.

Claims

1. A method of compressing a frame in an image compression and storage system, the method comprising: mapping an original sample to a mapped sample based on a bit depth of the original sample and a maximum allowed error to ensure a minimum reconstruction error of a highest priority sample value; determining a residual of the mapped sample based on a previously reconstructed sample of the mapping; applying a modulo addition to the residual to generate a biased residual; quantizing the biased residual based on the maximum allowed error to generate a quantized biased residual; and encoding a value corresponding to the quantized biased residual to generate an encoded value, wherein applying the modulo addition to the residual comprises calculating the biased residual as: where e' represents the biased residual, e represents the residual, bitdepth represents the bit depth, and δ represents the maximum allowed error.

2. The method of claim 1, wherein, the highest priority sample value is 0, and wherein mapping the original sample comprises calculating the mapped sample as: where f(x) represents the mapped sample, x represents the original sample, c is a constant non-negative integer less than δ, and maxVal represents a maximum value of the original sample.

3. The method of claim 2, wherein the maximum value of the original sample is represented as: maxVal = 2 bitdepth -1 where the original sample corresponds to a color value of a pixel of the frame, and where the frame has a red, green, and blue (RGB) format or a red, green, blue, and green (RGBG) format.

4. The method of claim 2, wherein, the bit depth of the original sample is 3, c is equal to 0, and the maximum allowed error is 1.

5. The method of claim 1, wherein, the highest priority sample value is greater than 0, and wherein mapping the original sample comprises calculating the mapped sample as: where f(x) represents the mapped sample, i is the highest priority sample value, x represents the original sample, c is a constant non-negative integer less than δ, and maxVal represents a maximum value of the original sample.

6. The method of claim 1, wherein determining the residual of the mapped sample comprises: subtracting the previously reconstructed sample of the mapping from the mapped sample to generate the residual of the mapped sample.

7. The method of claim 1, wherein, the value corresponding to the quantized biased residual is a quantization index corresponding to the quantized biased residual.

8. The method of claim 1, further comprising: performing inverse quantization on the quantized biased residual to generate a reconstructed biased residual; adding the previously reconstructed sample of the mapping to the reconstructed biased residual to generate a reconstructed biased sample; applying a modulo subtraction to the reconstructed biased sample to generate a reconstructed mapped sample; and inverse mapping the reconstructed mapped sample to an unmapped reconstructed sample based on the highest priority sample value, the bit depth of the original sample, and the maximum allowed error.

9. The method of claim 8, wherein, the highest priority sample value is 0, and wherein inverse mapping the reconstructed mapped sample comprises calculating the unmapped reconstructed sample as: where f(x) represents the mapped sample, x represents the original sample, c is a constant non-negative integer less than δ, and maxVal represents a maximum value of the original sample. where g(x') represents the unmapped reconstructed sample, x' represents the mapped reconstructed sample, c is a constant non-negative integer less than δ, maxVal represents a maximum value of the original sample, and Clip(0, maxVal, x'-δ+c) represents a clipping function that limits a value of x'-δ+c to a minimum of 0 and a maximum of maxVal.

10. The method of claim 8, wherein, the highest priority sample value is greater than 0, and where inversely mapping the mapped reconstructed sample includes calculating the unmapped reconstructed sample as: where g(x') represents the unmapped reconstructed sample, i is the highest priority sample value, x' represents the mapped reconstructed sample, c is a constant non-negative integer less than δ, maxVal represents a maximum value of the original sample, Clip(0, maxVal, x'-δ+c) represents a clipping function that limits a value of x'-δ+c to a minimum of 0 and a maximum of maxVal, and Clip(0, maxVal, x'-δ+c-1) represents a clipping function that limits a value of x'-δ+c-1 to a minimum of 0 and a maximum of maxVal.

11. The method of claim 8, further comprising: generating a prediction of an unmapped previously reconstructed sample based on the unmapped reconstructed sample; and mapping the prediction of the unmapped previously reconstructed sample to a mapped previously reconstructed sample.

12. The method of claim 8, further comprising: generating a prediction of the mapped previously reconstructed sample based on the mapped reconstructed sample.

13. The method of claim 1, wherein, a reconstruction error of the highest priority sample value is less than or equal to reconstruction errors of other values of the original sample.

14. A method of decompressing stored image data corresponding to a frame in an image compression and storage system, the method comprising: decoding an encoded value corresponding to an original sample of the frame to generate a decoded value corresponding to a quantized bias residual; performing inverse quantization on the decoded value to generate a reconstructed bias residual; adding a prediction of a mapped previously reconstructed sample to the reconstructed bias residual to generate a reconstructed bias sample; applying a modulo subtraction to the reconstructed bias sample to generate a mapped reconstructed sample; and inversely mapping the mapped reconstructed sample to an unmapped reconstructed sample based on a highest priority sample value, a bit depth of the original sample, and a maximum allowed error to ensure a minimum reconstruction error of the highest priority sample value, where the mapped reconstructed sample is calculated as: where y represents the mapped reconstructed sample, x represents the reconstructed bias sample, bitdepth represents the bit depth, and δ represents the maximum allowed error.

15. The method of claim 14, wherein, the highest priority sample value is 0, and where inversely mapping the mapped reconstructed sample includes calculating the unmapped reconstructed sample as: where g(x') represents the unmapped reconstructed sample, x' represents the mapped reconstructed sample, c is a constant non-negative integer less than δ, maxVal represents a maximum value of the original sample, and Clip(0, maxVal, x'-δ+c) represents a clipping function that limits a value of x'-δ+c to a minimum of 0 and a maximum of maxVal. the highest priority sample value is 0, and where inversely mapping the mapped reconstructed sample includes calculating the unmapped reconstructed sample as: where g(x') represents the unmapped reconstructed sample, x' represents the mapped reconstructed sample, c is a constant non-negative integer less than δ, maxVal represents a maximum value of the original sample, and Clip(0, maxVal, x'-δ+c) represents a clipping function that limits a value of x'-δ+c to a minimum of 0 and a maximum of maxVal.

16. The method of claim 14, wherein, the highest priority sample value is greater than 0, and where inversely mapping the mapped reconstructed sample comprises calculating the unmapped reconstructed sample as: where g(x') represents the unmapped reconstructed sample, i is the highest priority sample value, x' represents the mapped reconstructed sample, c is a constant non-negative integer less than δ, maxVal represents a maximum value of the original sample, Clip(0, maxVal, x'-δ+c) represents a clipping function that limits a value of x'-δ+c to a minimum of 0 and a maximum of maxVal, and Clip(0, maxVal, x'-δ+c-1) represents a clipping function that limits a value of x'-δ+c-1 to a minimum of 0 and a maximum of maxVal.

17. The method of claim 14, further comprising: generating the prediction of the mapped previous reconstructed sample based on the unmapped reconstructed sample.

18. An image compression and storage system, comprising: a processor; and a processor memory local to the processor, wherein the processor memory has instructions stored thereon that, when executed by the processor, cause the processor to perform: mapping original samples to mapped samples based on a bit depth of the original samples and a maximum allowed error to ensure a minimum reconstruction error of a highest priority sample value; determining a residual of the mapped samples based on a mapped previous reconstructed sample; applying modulo addition to the residual to generate a biased residual; quantizing the biased residual based on the maximum allowed error to generate a quantized biased residual; encoding values corresponding to the quantized biased residual to generate encoded values for storage in a storage medium; decoding the encoded values to generate decoded values corresponding to the quantized biased residual; performing inverse quantization on the decoded values to generate a reconstructed biased residual; adding a prediction of the mapped previous reconstructed sample to the reconstructed biased residual to generate a reconstructed biased sample; applying modulo subtraction to the reconstructed biased sample to generate a mapped reconstructed sample; and inversely mapping the mapped reconstructed sample to an unmapped reconstructed sample based on the highest priority sample value, the bit depth of the original samples, and the maximum allowed error to ensure the minimum reconstruction error of the highest priority sample value, where applying the modulo addition to the residual comprises calculating the biased residual as: where e' represents the biased residual, e represents the residual, bitdepth represents the bit depth, and δ represents the maximum allowed error, wherein applying the modulo subtraction to the reconstructed offset samples comprises calculating the mapped reconstructed samples as: wherein y represents the mapped reconstructed samples and x represents the reconstructed offset samples.

19. The image compression and storage system of claim 18, wherein, the difference between the original sample and the unmapped reconstructed sample is less than or equal to the difference between the original sample and the unmapped reconstructed sample for other values of the original sample.

Images