Entropy coding of transform coefficient ordering

Abstract

The present disclosure provides a method for encoding input data in an encoder to generate encoded output data, wherein the method comprises instructing to convert the input data to base layer reconstruction data using a down-sampled version of the input data; and generating enhancement layer data providing instructions for enhancing the base layer reconstruction data, wherein the enhancement layer data comprises residual data representing a difference between a reconstruction generated from the base layer reconstruction data and data derived from the input data, wherein the residual data is subjected to at least one transform and at least one data compression method to generate the encoded output data, characterized in that the method comprises arranging the at least one transform to comprise a Hadamard transform to convert the residual data to corresponding transform coefficients, wherein different output elements of the Hadamard transform are arranged as different surfaces to be encoded, and selecting an output order for the transform coefficients in the surfaces to provide a sequence of transform coefficients for the at least one data compression method to generate the encoded output data, wherein the output order is selected to be at least one of a tile raster order and an n x n block "Z" order.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to an apparatus for encoding data, for example an apparatus for encoding image or video data. Furthermore, the present disclosure relates to a method for encoding data using the above-mentioned apparatus, for example a method for encoding image or video data (i.e. a method of encoding data using the above-mentioned apparatus). Furthermore, the present disclosure relates to an apparatus for decoding data, for example an apparatus for decoding image or video data. Furthermore, the present disclosure relates to a method for decoding data using the above-mentioned apparatus, for example a method for decoding encoded image or video data (i.e. a method of decoding data using the above-mentioned apparatus). Furthermore, the present disclosure relates to a computer program product for performing the above-mentioned methods. BACKGROUND

[0002] MPEG-2, MPEG-4 and MPEG-5 video encoding standards are well known. In order to achieve efficient compression during encoding according to these standards, a discrete cosine transform (DCT) is used to process a macroblock of pixels in a spatially represented array of pixels of an image to provide DCT coefficients in a transformed data macroblock.

[0003] The above-mentioned MPEG encoders were developed at a time when computing resources were relatively limited and images and video data were largely processed in a serial manner. Similar considerations apply when necessary modifications are made to decode the encoded data. However, modern computing devices are capable of processing data in a more parallel manner by employing multiple processing cores. These modern computing devices encourage the development of other types of encoding that are better suited to execution on parallel computing devices.

[0004] These other types of encoding include, among others, the VC-6 and LCEVC standards described in various published patent applications incorporated herein by reference. For example, the LCEVC standard is described in the ISO / IEC DIS 23094-2 Low Complexity Enhancement Video Coding draft text published at the MPEG 129 meeting in Brussels in January 2020. The VC-6 standard is described in publication SMPTE ST 2117-1 (21 July 2020). Both of these standard specifications are incorporated herein by reference.

[0005] In particular, the VC-6 standard is described in the following patent applications:

[0006] EP12756254.4, EP12756257.7, EP12756258.5,

[0007] EP12759220.2, EP12818920.6, EP12759221,

[0008] EP12799276.6, EP13723460.5, EP13722424.2,

[0009] EP13724216.0, PCT / EP2018 / 0756033, EP17386046.1,

[0010] EP1R3860022, PCT / GB2018 / 053546, PCT / GB2018 / 053555, PCT / GB2018 / 053547, PCT / GB2018 / 053554, PCT / GB2018 / 053548.

[0011] In particular, LCEVC type is described in the following patent applications:

[0012] EP12756254.4, EP12756257.7, EP 12756258.5,

[0013] EP12759220.2, EP12818920.6, EP12799276.6,

[0014] EP13708525.4, EP13723460.5, EP13722424.2,

[0015] EP13724216.0, EP14722769.8, EP16804893.2,

[0016] EP16718438.1, EP17783955.2, EP17783956.0,

[0017] EP17783957.8, PCT / EP2018 / 075603, PCT / GB2018 / 053548,

[0018] EP 18199014.4, PCT / GB2019 / 053068, PCT / GB2019 / 053066,

[0019] PCT / GB2019 / 053067, PCT / GB2019 / 053069, PCT / GB2020 / 050695,

[0020] PCT / GB2019 / 051974, PCT / GB2019 / 052152, PCT / GB2019 / 052151,

[0021] PCT / GB2019 / 052154, PCT / GB2019 / 052166, PCT / GB2019 / 052867, PCT / GB2019 / 053070, PCT / GB2019 / 053551, PCT / GB2019 / 053547,

[0022] PCT / GB2020 / 050492, PCT / GB2020 / 050574, PCT / GB2020 / 050693,

[0023] PCT / GB2020 / 050692, PCT / GB2020 / 050725 and PCT / GB2020 / 050695.

[0024] LCEVC and VC-6 do not employ the prediction and DCT implemented in various known MPEG, H.264 and H.265 standards. Beneficially, VC-6 and / or LCEVC can employ such known standards to implement a base layer of encoding to ensure backward compatibility with legacy systems, and then add at least one enhancement layer, implemented for example in software, that provides an enhancement to the base layer to enable scalable quality of decoded data, providing a balance between bit rate and video reproduction quality in the decoder.

[0025] In particular, in LCEVC and VC-6, a base layer of encoding is performed on a down-sampled original input image or video data. The difference between a decoded version of the base encoded data and the original input is computed on a frame-by-frame basis, and this difference forms the basis of residual data that can be added to the decoded version of the base encoded data. LCEVC and VC-6 both provide multiple levels of spatial scalability, where the residual data for each level can be computed in order to efficiently encode different levels of features at different spatial resolutions. This is unlike the comparison MPEG, H.264 and H.265 standards, which use neighbouring blocks within a frame or other frames in a video sequence to predict macroblocks. It allows for a high degree of parallel processing that is not achievable with H.264 and H.265, for example, due to inherent spatial and temporal correlation between blocks introduced by prediction.

[0026] One of the persistent technical problems in video encoding addressed by the present disclosure is how to encode data in a more efficient manner. For example, it is desirable to develop methods that can encode at least the residual data generated by the VC-6 and LCEVC standards in a more efficient manner. SUMMARY

[0027] The present disclosure aims to provide an improved apparatus for encoding at least residual data in a more efficient and effective manner. Furthermore, the present disclosure aims to provide an improved method for encoding residual data in a more efficient and effective manner. Furthermore, the present disclosure aims to provide an improved apparatus for decoding at least residual data in a more efficient and effective manner. Furthermore, the present disclosure aims to provide an improved method for decoding at least residual data in a more efficient and effective manner.

[0028] In a first aspect, the present disclosure provides a method for encoding input data in an encoder to generate encoded output data, wherein the method comprises instructing to convert the input data to base layer reconstruction data using a down-sampled version of the input data; and generating enhancement layer data providing instructions for enhancing the base layer reconstruction data, wherein the enhancement layer data comprises residual data representing a difference between a reconstruction generated from the base layer reconstruction data and data derived from the input data, wherein the residual data is subjected to at least one transform and at least one data compression method to generate the encoded output data, characterized in that the method comprises arranging the at least one transform to comprise a Hadamard transform to convert the residual data to corresponding transform coefficients, wherein different output elements of the Hadamard transform are arranged to be different surfaces to be encoded, and selecting an output order for the transform coefficients in the surfaces to provide a sequence of transform coefficients for the at least one data compression method to generate the encoded output data, wherein the output order is selected to be at least one of a tile raster order and an n x n block "Z" order.

[0029] An advantage of the present invention is that the reading out of transform (Hadamard) coefficients can improve data compression performance when generating the encoded output data.

[0030] Optionally, in the method, the transform coefficients are generated based on a predefined coding unit size (e.g. 2x2 or 4x4 residual blocks), and the tiles for the tile raster order comprise a set of transform coefficients larger than the coding unit size. The tile raster order can comprise encoding the tiles of transform coefficients, wherein the tiles are read out in a raster fashion.

[0031] Optionally, in the method, the tiling manner comprises encoding tiles of size 8x8 elements, 16x16 elements, 32x32 elements.

[0032] Optionally, in the method, the at least one data compression method comprises run-length encoding (RLE) followed by Huffman encoding.

[0033] Optionally, the n x n block "Z" order comprises one or more n x n "Z" order readouts of coefficients in a given coefficient group from a top-left coefficient to a top-right coefficient to a bottom-left coefficient to a bottom-right coefficient, where the n x n "Z" order readouts are repeated in a recursive n x n "Z" order between different transform coefficient groups. For example, the "Z" order can comprise one or more 2 x 2 "Z" order readouts of coefficients in a given coefficient group from a top-left coefficient to a top-right coefficient to a bottom-left coefficient to a bottom-right coefficient, where the 2 x 2 "Z" order readouts are repeated in a "Z" order when a matrix or surface of Hadamard transform coefficients comprises a multiple of 2 x 2 coefficient groups.

[0034] Optionally, the method is implemented to be compatible with the VC-6 or LCEVC standard.

[0035] According to a second aspect, there is provided an encoder operative to encode input data to generate corresponding encoded output data, wherein the encoder is operative to indicate a conversion of the input data to base layer reconstruction data using a down-sampled version of the input data, wherein the encoder is operative to generate enhancement layer data providing instructions for enhancing the down-sampled version of the input data, wherein residual data represents a difference between a reconstruction generated from the base layer reconstruction data and data derived from the input data, wherein the residual data is subjected to at least one transform and at least one data compression transform in the encoder to generate the encoded output data, characterized in that the at least one transform comprises a Hadamard transform to convert the residual data to corresponding Hadamard transform coefficients, wherein different output elements of the Hadamard transform are arranged as different surfaces to be encoded, wherein the encoder is configured to select an output order for the transform coefficients in the surfaces to provide a sequence of transform coefficients for the at least one data compression method to generate the encoded output data, wherein the output order is selected to be one of at least a tile raster order and an n x n block "Z" order.

[0036] Optionally, in the encoder, the at least one data compression method comprises run-length encoding (RLE) followed by Huffman encoding.

[0037] Optionally, in the encoder, the tile raster order comprises encoding tiles of transform coefficients, where the tiles are read out in a raster fashion.

[0038] Optionally, in the encoder, the tile raster order comprises encoding tiles having a size in a range of 8 x 8 elements to 64 x 64 elements, where the transform coefficients are selected in a raster order within each tile. For example, the tiles can have a size of 8 x 8 elements, 16 x 16 elements, or 32 x 32 elements.

[0039] Optionally, in the encoder, the n x n block "Z" order includes one or more n x n "Z" order readouts of coefficients in a given coefficient array from a top-left coefficient to a top-right coefficient to a bottom-left coefficient to a bottom-right coefficient, where the n x n "Z" order readouts are repeated in a recursive n x n "Z" order between different transform coefficient arrays. For example, the "Z" order can include one or more 2 x 2 "Z" order readouts of coefficients in a given coefficient array from a top-left coefficient to a top-right coefficient to a bottom-left coefficient to a bottom-right coefficient, where the 2 x 2 "Z" order readouts are repeated in a "Z" order when a matrix or surface of Hadamard transform coefficients includes a multiple of 2 x 2 coefficient arrays.

[0040] Optionally, the encoder is implemented to be compatible with a VC-6 or LCEVC standard.

[0041] According to a third aspect, there is provided a computer program product comprising a non-transitory computer-readable storage medium having computer-readable instructions stored thereon, the computer-readable instructions executable by a computerized device comprising processing hardware to perform the method of the second aspect described above.

[0042] According to a fourth aspect, there is provided a method for decoding encoded data in a decoder to generate decoded output data, wherein the method comprises: (i) receiving encoded data at the decoder; (ii) applying at least one inverse transform to convert the received encoded data into enhancement layer data, the enhancement layer data providing instructions for enhancing a base layer reconstruction derived from a downsampled version of input data, wherein the enhancement layer data comprises residual data to provide an enhancement thereof, wherein the residual data is subjected to at least one transform and at least one data compression method to generate the received encoded data, characterized in that the method further comprises: (iii) decompressing the received encoded data according to the at least one data compression method to generate intermediate decoded data; and (iv) arranging the at least one inverse transform to include an inverse Hadamard transform to convert transform coefficients derived from the intermediate decoded data into the residual data, wherein the transform coefficients are decoded by reading in a tile grid fashion or an n x n block "Z" order.

[0043] According to a fifth aspect, there is provided a computer program product comprising a non-transitory computer-readable storage medium having computer-readable instructions stored thereon, the computer-readable instructions executable by a computerized device comprising processing hardware to perform the method of the fourth aspect.

[0044] According to a sixth aspect, there is provided a decoder for decoding encoded data to generate decoded output data, wherein the decoder comprises: (i) an input for receiving encoded data; (ii) a data processing arrangement for applying at least one inverse transform to convert the received encoded data into enhancement layer data, the enhancement layer data providing instructions for enhancing a base layer reconstruction derived from a down-sampled version of the input data, wherein the enhancement layer data comprises residual data to provide an enhancement thereof, wherein the residual data is subjected to at least one transform and at least one data compression method to generate the encoded data, characterised in that the decoder further comprises: (iii) a decompression arrangement for decompressing the received encoded data according to the at least one data compression method to generate intermediate decoded data; and (iv) an inverse transform arrangement for applying at least one inverse transform as an inverse Hadamard transform to convert transform coefficients derived from the intermediate decoded data into the residual data, wherein the transform coefficients are decoded by reading in a tile grid fashion or n x n block "Z" order.

[0045] Additional aspects, advantages, features and objects of the present disclosure will become apparent to those skilled in the art from a detailed description of illustrative embodiments thereof, which is to be taken in conjunction with the appended claims.

[0046] It should be understood that features of the present disclosure are susceptible to being combined in various combinations without departing from the scope of the present disclosure defined by the appended claims. BRIEF DESCRIPTION OF DRAWINGS

[0047] The above summary, as well as the following detailed description of illustrative embodiments, can be better understood when read in conjunction with the appended drawings. For the purpose of illustrating the present disclosure, exemplary constructions of the present disclosure are shown in the drawings. However, the present disclosure is not limited to the specific methods and devices disclosed herein. In addition, those skilled in the art will understand that the drawings are not drawn to scale. Where possible, like elements have been indicated by the same numbers.

[0048] Embodiments of the present disclosure will now be described, by way of example only, and with reference to the following drawings in which:

[0049] Figure 1A is a schematic illustration of different scales of the Z order way of reading out coefficients from a Hadamard transform;

[0050] Figure 1B is an illustration of the basis of a Walsh-Hadamard transform (DWHT);

[0051] Figure 1C is a schematic illustration of a portion of tile grid order read out;

[0052] Figure 1D is a schematic illustration of a comparison of grid read out; and

[0053] Figures 2 to 21 is a graphical illustration of the performance benefits obtained by employing a tile and Z-order approach to reading out coefficients from a Hadamard transform when the Hadamard transform is applied to various types of image or video content.

[0054] In the drawings, underlined numerals are used to designate items in the drawings by the numerals being underlined. Un-underlined numerals are associated with items identified by the line connecting the un-underlined numeral with the item. DETAILED DESCRIPTION

[0055] In the following detailed description, illustrative embodiments of the disclosure and ways in which they can be implemented are set forth. Although a number of modes for carrying out the disclosure are described, one skilled in the art will recognize that other embodiments for carrying out or practicing the disclosure are also possible.

[0056] It is known by, for example, the enhancement coding standards LCEVC and VC-6 that encoding input data in an encoder to generate encoded output data can include indicating to convert the input data using a down-sampled version of the input data to base layer reconstruction data, and to generate enhancement layer data providing instructions for enhancing the base layer reconstruction data. For example, the input data can be down-sampled and provided to a base codec for encoding and subsequent decoding. The encoded base layer forms the encoding of the base layer reconstruction data transmitted to the decoder. The enhancement layer data includes residual data representing the difference between the reconstruction generated from the base layer reconstruction data (i.e., the subsequently decoded, encoded base layer) and the data derived from the input data. There can be multiple layers of residual data representing different spatial resolutions (e.g., LCEVC has two layers). Each residual data stream is subjected to at least one transform and at least one method of data compression to generate the encoded output data, i.e., a set of encoded residual streams. Quantization can also be performed after the transform. The encoded output data can be included in the encoded base layer (e.g., as part of a multiplexed combination of the base stream and the enhancement stream) or can be transmitted separately. The encoder can be suitably configured to perform this method.

[0057] Similarly, there can also be a corresponding method for decoding the encoded data in a decoder to generate decoded output data. This method includes receiving the encoded data at the decoder, and applying at least one inverse transform to convert the received encoded data to enhancement layer data. The inverse transform can transform a set of transform coefficients back to residual data. Dequantization can also be performed after the inverse transform. The enhancement layer data includes the previously described residual data generated at the encoder. The decoder can receive an encoded base layer and instruct to decode this encoded base layer using a base layer decoder (e.g., corresponding to the base layer encoder used by the previously described encoder). This can result in base layer reconstructed data at the decoder, which can be enhanced with the residual data. In one case, the reconstruction generated with the base layer reconstructed data is upsampled before the at least one layer of residual data is applied. The decoder can be suitably configured to perform this method.

[0058] More details can be found in the published material related to LCEVC and the VC-6 standard.

[0059] Walsh-Hadamard transforms are simple but effective methods of compressing data. Their significant advantage is that the basic transform requires no multiplication, only sum and difference. The discrete Walsh-Hadamard transform (DWHT) is obtained by a simple rearrangement of the rows of the discrete Hadamard matrix. The Hadamard matrix is a matrix with this property. By iteratively applying the following operation, a higher order matrix can be found:

[0060]

[0061] For example:

[0062]

[0063] The DWHT is simply obtained from the corresponding Hadamard matrix by normalizing and rearranging the rows in order (i.e., according to the number of sign changes). Thus, the four-point DWHT is given by:

[0064]

[0065] Figure 1BThe basic functions of the 1-D DWHT 140, 142, 144, 146 are depicted. In some cases, the normalization factor (1 / 4 in equation 3) can be omitted, making the transform matrix similar to the Hadamard matrix of equation 2. This is possible, for example, if the normalization is incorporated into other functions such as quantization. Within LCEVC and VC-6, the DWHT or Hadamard transform based transform is referred to as the Directional Decomposition (DD) transform. It can be applied as a 4x4 transform matrix or as a larger 16x16 transform matrix, the latter case being referred to as the Directional Decomposition Square (DDS) transform, as it is equivalent to applying the DD transform twice. Directional decomposition has energy compression and decorrelation properties, as the rows of the Hadamard matrix are orthogonal, and the energy of the signal is represented by the “average” coefficient associated with the first row of the Hadamard matrix, the other coefficients representing differences in different directions. For many original image blocks, the signal energy tends to be fairly evenly distributed across the block. After the transform, the data is decorrelated in the horizontal and vertical directions, and for many blocks, one dominant coefficient, the average coefficient, contains most of the signal energy.

[0066] The Hadamard function is applied to a data array of residual elements generated during the VC-6 and LCEVC encoding operation to provide corresponding Hadamard transform elements, referred to herein as transform coefficients. By way of example, each row of a 4x4 Hadamard matrix (DD transform) can be labeled as one of A, H, V, and D, corresponding to the average, horizontal difference, vertical difference, and diagonal difference within the coding unit (typically a 2x2 or 4x4 residual block). As described, for example, in the LCEVC standard, a 2x2 coding unit containing a residual is flattened into a 4x1 vector, which is then transformed using the Hadamard matrix to generate a 4x1 vector with [A, H, V, D] coefficient terms. Each of these coefficients can be grouped together with other similar coefficients, such that the transform generates four “surfaces”, i.e. frames or matrices of transform coefficient values related to a particular one of A, H, V, and D. For example, there can be an A surface, an H surface, a V surface, and a D surface. For a 16x16 DDS transform, there can be 16 surfaces. The Hadamard transform can be performed in parallel on the coding units, followed by entropy encoding, including run length encoding followed by Huffman coding.

[0067] In the comparative LCEVC standard implementation, the transform coefficients of each surface are read out in a major row raster order on the surface for RLE. Figure 1D An example of this ordering is shown, in which the transform coefficient values 160 are read out sequentially along each row of the surface (e.g. from left to right along each row, as shown by lines 162).

[0068] In this example, the inventors have recognized that the order in which transform coefficients are selected before RLE can affect compression efficiency. They have also recognized that residual data generated as part of enhancement coding (e.g., performed in LCEVC or VC-6) typically exhibits spatial correlation based on the original spatial correlation within the original video frame. Since the enhancement coding described herein generates enhancement data based on a comparison of the original data with an upsampled reconstruction from a lower spatial resolution, the nature of this enhancement data differs from the residual data generated in comparative video codecs such as H.264 and H.265, where, in the latter case, the residual data is generated based on local spatial predictions (e.g., through neighboring blocks) or motion vector-mediated temporal predictions (e.g., through other frames). Therefore, residual data within standards such as LCEVC and VC-6 possesses different properties, such as spatial correlation, which is undesirable in comparative residual processing methods. The inventors have recognized that this spatial correlation can span multiple coding units, and therefore, transform coefficients within a surface can also exhibit spatial correlation. For example, a horizontal line can extend across multiple coding units. Therefore, this example allows for the selection of the readout order of the transform coefficients, which allows for beneficial compression during subsequent RLE and / or Huffman coding. Additionally, it has been found that the improved readout order may depend on the content being encoded. Therefore, a content-dependent readout order can be selected during encoding and passed to the decoder to allow for corresponding readouts after decompression. It has been found that these benefits are independent of any quantization performed (e.g., before or after readout).

[0069] In some of the instances described herein, two alternatives to the primary row raster readout order are presented. These alternatives include a tile raster order and an n×n block “Z” order. The benefits of each order may vary depending on one or more of the content type and the required bit rate, so one of the three orders can be selected during encoding to provide optimal compression and passed as configuration data to the decoder for decoding.

[0070] Figure 1A An example of the “Z” readout order performed on the surface of the transform coefficients is shown. Figure 1A The top left corner shows the “Z” order of the 2×2 blocks 110 with transformation coefficients 102, 104, 106, and 108 applied to the surface. Figure 1A The rest of the diagram illustrates how this “Z” order can be applied recursively. For example, the top right corner shows the “Z” order applied to the 16 transform coefficient blocks 120 formed by four 2×2 blocks 110. Then, the bottom left corner shows the “Z” order applied to the 2×2 blocks 120, and this pattern is repeated again for the coefficients 130 in the bottom right corner.

[0071] For more details, see the reference.Figure 1A The 2x2 transform coefficient block from the surface is read in "Z" order, from the top left coefficient to the top right coefficient to the bottom left coefficient to the bottom right coefficient, i.e. in "Z" order, as shown in the top left corner. Figure 1A The 4x4 transform coefficient group is read from the four 2x2 transform coefficient groups as shown in the top right corner, where the reading of the four groups is also performed in "Z" order as shown. Figure 1A This "Z" read order is also shown for 16x16 transform coefficient groups and 32x32 transform coefficient groups. When the transform coefficients have been read in the order indicated in the middle, Figure 1A When the transform coefficients have been read in the order indicated in the middle, a corresponding 1-dimensional (1-D) sequence of transform coefficients is generated, which is readily encoded using run length encoding, Huffman encoding or other entropy encoding methods to generate compressed data for transmission to a decoder. Quantization can also be performed before or after the read. Beneficially, a combination of run length encoding followed by Huffman encoding is employed when encoding data for VC-6 and LCEVC.

[0072] In certain cases, Figure 1A The "Z" order shown is shown to provide improved encoding compared to the main row raster order. For certain content types, it is also found that a tile raster order with tiles of configurable size provides compression improvements. In certain cases, it is found that reading the transform coefficients in a tiled manner provides better compression results than the "Z" read order described above. In the case of tile raster order, the coefficients are read in a raster order within a defined tile, and the read transform coefficients are then compressed using run length encoding, Huffman encoding or a combination of run length encoding and Huffman encoding (with optional quantization as described above). It has been found that tiles of size in the range of 8x8 element arrays to 64x64 element arrays, for example tiles of size 8x8 element arrays, 16x16 element arrays or 32x32 element arrays provide beneficial results. It will be appreciated that rectangular format element arrays can be used instead of the square element arrays described above.

[0073] Figure 1C An example of a tile raster order applied to surface 150 of transform coefficients 152 is shown. The size of the example tile 154 is shown as d x coefficients x d y coefficients. In certain cases, for square tiles, d x = d y= d, where d can be set to one of 8, 16, 32, or 64. As partially shown for the first three tiles, the readout order moves in a raster order within each tile, e.g., moving along each row of the tile before moving to the next tile, as shown. The tile raster order can continue across the entire surface 150 of transform coefficients 152 to generate the 1D sequence. It should be noted that the definition of these tiles is independent of the original coding unit size used as a basis for the transform (where the transform can be applied in parallel to each coding unit to generate the surface).

[0074] In summary, in a comparative standardization practice, data can be simply rastered from a Hadamard transform to generate a sequence of coefficients for compression using, for example, run-length encoding (RLE). However, it was unexpectedly discovered that reading out the Hadamard coefficients in one or more of the "Z" order or tiling approaches even more beneficially improves compression performance. This improvement in compression performance is believed to occur due to spatial correlation within a given image frame or video frame. This improvement in compression performance is shown, for example, in Figures 2 to 21

[0075] The "Z" ordering is essentially encoding the coefficients in a recursive zigzag pattern, first encoding four coefficients in a Z pattern, then encoding 4 Z patterns in a Z pattern; as shown, for example, in Figure 1A This process repeats with increasing scale until, for example, the entire image is encoded. However, it will be appreciated that when the surface length in a given image is not a power of two, boundary conditions slightly complicate the Z-order algorithm, but various effective implementations of Z-order readout are still feasible.

[0076] Experimentally, embodiments of the present disclosure are able to improve data compression performance when encoding different types of image and video content, for example, enhancing the achievable compression by about 40% compared to raster readout of the coefficients. However, for certain types of image and video content, the enhancement in achievable compression is significantly less than 40%, as shown, for example, in Figures 2 to 21

[0077] Figure 2 and 3 Results 200 and 300 are shown for respective encoding based on the DD transform (4x4 Hadamard transform) and the DDS transform (16x16 Hadamard transform). The relative change AR is shown in percentage on the y-axis compared to the original main row raster order, for example:

[0078]

[0079] where R new is the bit rate obtained from the newly proposed ordering method, and R orig ​​is the original primary row raster order. A value AR < 0% indicates an improvement over the original order, and a value AR > 0% indicates a decrease in compression efficiency. The test was performed using a set number of (e.g., 100) frames with a fixed quantization parameter. Figure 2 and 3 shows the improvement in the proposed order as the encoding bitrate is increased. Figure 2 and 3 shows the results for the first test video, “cactus”. Figure 4 and 5 shows the results 400 and 500 for the second test video, “BasketballDrive”, Figure 14 and 15 shows the results 1400 and 1500 for the third test video, “Fornite”, and Figure 16 and 17 shows the results 1600 and 1700 for the fourth test video, “ParkRunning3”, similar to the DD and DDS transforms.

[0080] Figure 6 shows the test results 600 for the DD transform for the four different content types described above, and Figure 7 shows the test results 700 for the DDS transform. It can be seen that the increase in bitrate can vary depending on the content type and the type of read order. Thus, the bitrate can be increased by selecting a particular read order for a particular content item, e.g., based on test encoding at encoding time and / or previous experimental results.

[0081] Figure 8 and 9 shows the results 800 and 900 similar to the results 200 and 300 but for a different target bitrate range (implemented by a different set of quantization parameters). Similarly, Figure 10 and 11 shows the results 1000 and 1100 similar to the results 400 and 500, Figure 18 and 19 shows the results 1800 and 1900 similar to the results 1400 and 1500, and Figure 20 and 21 shows the results 2000 and 2100 similar to the results 1600 and 1700 but for a different target bitrate range. Figure 12 and 13 shows the results 1200 and 1300 for the DD and DDS transforms, respectively, in a different second target bitrate range.

[0082] It will be understood that embodiments of this disclosure, namely advanced encoders, are readily implemented in custom digital circuits such as gate arrays, FPGAs, etc.; alternatively, embodiments of this disclosure are readily implemented using computing hardware such as multi-core processors that, in operation, execute one or more software products containing machine-executable instructions.

[0083] This aforementioned tiling or "Z"-order readout of the Hadamard transform coefficients can be used in the encoder and corresponding decoder to communicate over data communication networks, such as via... Enhanced data compression is achieved when transmitting image or video content.

[0084] It will be understood that when encoding the array of transformed residual data, each of the 4 DD components or 16 DDS components is typically encoded individually using run-length encoding (i.e., as a set of 4 or 16 surfaces) before compression via variable-length encoding (or other entropy encoding techniques, such as arithmetic or range encoding). While the compression introduced by this second step is little to no dependent on the order of the data in the DD or DDS components, the compression that might be introduced by the first step is expected to depend on the order.

[0085] For example, if the same given value appears twice in a DD component (hereinafter referred to as "the surface"), there is no benefit to stroke length compression if the occurrences are separated from each other. On the other hand, if the scanning mode of the data is accidentally selected so that repeated values ​​are immediately and continuously read into the stroke length encoder, the size of the second symbol is preserved, in exchange for the cost of specifying a stroke length of 2. Typically, 2D arrays such as surfaces are scanned in raster order. This allows long strokes of the same value to be adjacent to each other. However, real images have natural noise or measurement noise and short strokes, so long strokes only appear after a large amount of quantization (large stride). The quantization parameters are sometimes different between patches of the image, which is one reason why the different order used in the embodiments of this disclosure may be beneficial for assisting stroke length encoding.

[0086] Accordingly, embodiments of the present disclosure beneficially employ allowing a given encoder to switch between different ordering modes when encoding elements other than full grids, in order to reduce the bitrate or file size without changing the quality. The improved mode of element reading is optionally hardcoded (e.g., incorporated into an FPGA used to implement a given encoder) or determined through intelligent online evaluation of the data (e.g., using software executable by computing hardware, e.g., using machine learning / artificial intelligence (ML / AI) capable of iteratively learning an optimal strategy to select between modes in response to different value distributions appearing in the elements to be encoded). In the case of hardcoding, the decoder must follow the hardcoded order of the decoder per channel. Otherwise, the order must be signaled through metadata in the encoded bitstream provided by the given encoder to the corresponding decoder.

[0087] One example of a more local order is the Z-ordering described above, for example. Another example is to tile the array in tiles and first order the data in the tile order of the tiles and then in the grid order within the tiles. From practical experience with implementing embodiments of the present disclosure, it was observed that the tile size has an impact and in some cases, the tile grid order is preferred over the Z-order. Depending on the quantization step size, a trade-off can be selected whether to employ the Z-order reading or not the tile reading: in lossless mode (e.g., quantization step size of 1), the benefit of implementing the mode selection can be small, but can increase with the step size (e.g., as shown in the chart).

[0088] While ordering / traversing in real (image) space is well known, the idea of different ordering in the DD component space (as employed in embodiments of the present disclosure) is not obvious and the benefit of compression can be huge. Instead of deciding independently on a per-frame basis the order for a surface, the encoder can decide to use a predefined strategy that beneficially switches the ordering based on the entropy or statistical distribution of the surface data. Beneficially, the bitstream generated by the encoder signals to the decoder whether any ordering must be changed between frames.

[0089] Modifications of embodiments of the application can be made by persons of ordinary skill in the art without departing from the scope of the application as defined by the appended claims. Expressions such as "including", "including a", "including an", "including the", "including their", "including his", "including her", "including its", "including their", "including his", "including her", "including one or more of, "including at least one of, "comprising", "comprising a", "comprising an", "comprising the", "comprising their", "comprising his", "comprising her", "comprising its", "comprising their", "comprising his", "comprising her", "comprising one or more of, "comprising at least one of, "containing", "containing a", "containing an", "containing the", "containing their", "containing his", "containing her", "containing its", "containing their", "containing his", "containing her", "containing one or more of, "containing at least one of, "including the", "including their", "including his", "including her", "including its", "including their", "including his", "including her", "including one or more of, "including at least one of, "having", "having a", "having an", "having the", "having their", "having his", "having her", "having its", "having their", "having his", "having her", "having one or more of, "having at least one of, "characterized by", "characterized by a", "characterized by an", "characterized by the", "characterized by their", "characterized by his", "characterized by her", "characterized by its", "characterized by their", "characterized by his", "characterized by her", "characterized by one or more of, "characterized by at least one of, "comprised of", "comprised of a", "comprised of an", "comprised of the", "comprised of their", "comprised of his", "comprised of her", "comprised of its", "comprised of their", "comprised of his", "comprised of her", "comprised of one or more of, "comprised of at least one of, "consisting of", "consisting of a", "consisting of an", "consisting of the", "consisting of their", "consisting of his", "consisting of her", "consisting of its", "consisting of their", "consisting of his", "consisting of her", "consisting of one or more of, "consisting of at least one of, "defined by", "defined by a", "defined by an", "defined by the", "defined by their", "defined by his", "defined by her", "defined by its", "defined by their", "defined by his", "defined by her", "defined by one or more of, "defined by at least one of, "characterized by", "characterized by a", "characterized by an", "characterized by the", "characterized by their", "characterized by his", "characterized by her", "characterized by its", "characterized by their", "characterized by his", "characterized by her", "characterized by one or more of, "characterized by at least one of, "comprised of", "comprised of a", "comprised of an", "comprised of the", "comprised of their", "comprised of his", "comprised of her", "comprised of its", "comprised of their", "comprised of his", "comprised of her", "comprised of one or more of, "comprised of at least one of, "consisting of", "consisting of a", "consisting of an", "consisting of the", "consisting of their", "consisting of his", "consisting of her", "consisting of its", "consisting of their", "consisting of his", "consisting of her", "consisting of one or more of, "consisting of at least one of, "defined by", "defined by a", "defined by an", "defined by the", "defined by their", "defined by his", "defined by her", "defined by its", "defined by their", "defined by his", "defined by her", "defined by one or more of, "defined by at least one of, "including", "including a", "including an", "including the", "including their", "including his", "including her", "including its", "including their", "including his", "including her", "including one or more of, "including at least one of, "having", "having a", "having an", "having the", "having their", "having his", "having her", "having its", "having their", "having his", "having her", "having one or more of, "having at least one of, "including the", "including their", "including his", "including her", "including its", "including their", "including his", "including her", "including one or more of, "including at least one of, "having", "having a", "having an", "having the", "having their", "having his", "having her", "having its", "having their", "having his", "having her", "having one or more of, "having at least one of, "including the", "including their", "including his", "including her", "including its", "including their", "including his", "including her", "including one or more of, "including at least one of, "having", "having a", "having an", "having the", "having their", "having his", "having her", "having its", "having their", "having his", "having her", "having one or more of, "having at least one of, "including the", "including their", "including his", "including her", "including its", "including their", "including his", "including her", "including one or more of, "including at least one of, "having", "having a", "having an", "having the", "having their", "having his", "having her", "having its", "having their", "having his", "having her", "having one or more of, "having at least one of, "including the", "including their", "including his", "including her", "including its", "including their", "including his", "including her", "including one or more of, "including at least one of, "having", "having a", "having an", "having the", "having their", "having his", "having her", "having its", "having their", "having his", "having her", "having one or more of, "having at least one of, "including the", "including their", "including his", "including her", "including its", "including their", "including his", "including her", "including one or more of, "including at least one of, "having", "

Claims

1. A method for encoding input data in an encoder to generate encoded output data, wherein the method comprises: Instructing the conversion of the input data into base layer reconstruction data using a downsampled version of the input data; and An enhancement layer data is generated, providing instructions for enhancing the base layer reconstruction data. The enhancement layer data includes residual data representing the difference between the reconstruction generated from the base layer reconstruction data and the data derived from the input data. This residual data is then subjected to at least one transformation and at least one data compression method to generate the encoded output data. Its features The method includes: The at least one transform is arranged to include a Hadamard transform to convert the residual data into corresponding transform coefficients, wherein the different output elements of the Hadamard transform are arranged as different surfaces to be encoded, wherein each surface is an arrangement of the transform coefficients of the corresponding component in the transform, and The output order of the transform coefficients in the corresponding surfaces is arranged to provide a sequence of transform coefficients for the at least one data compression method to generate the encoded output data. The output order is arranged as at least one of a tile grid order and an n×n block "Z" order, wherein the size of the tile is greater than 1×1.

2. The method according to claim 1, characterized in that... The at least one data compression method includes run-length encoding followed by Huffman encoding.

3. The method according to claim 1 or 2, characterized in that... The method includes arranging the tile grid order to encode tiles ranging in size from 8×8 elements to 64×64 elements, wherein transformation coefficients are selected based on the grid order within each tile.

4. The method according to claim 3, characterized in that... The tile grid order includes encoding tiles of sizes of 8×8 elements, 16×16 elements, or 32×32 elements.

5. The method according to claim 1 or 2, characterized in that... The n×n block "Z" sequence comprises one or more n×n "Z" sequence readouts from the top left coefficient to the top right coefficient to the bottom left coefficient and then to the bottom right coefficient in a given coefficient group, wherein the n×n "Z" sequence readouts are repeated recursively in n×n "Z" sequence between different transformation coefficient groups.

6. The method according to claim 1 or 2, characterized in that... The method is implemented to be compatible with the LCEVC standard.

7. An encoder that, during operation, encodes input data to generate corresponding encoded output data. The encoder therein instructs the conversion of the input data into base layer reconstruction data using a downsampled version of the input data. The encoder generates enhancement layer data that provides instructions for enhancing the downsampled version of the input data. The residual data represents the difference between the reconstruction generated from the base layer reconstruction data and the data derived from the input data. The residual data is subjected to at least one transformation and at least one data compression transformation in the encoder to generate the encoded output data. Its features The at least one transformation includes a Hadamard transform to convert the residual data into corresponding Hadamard transform coefficients. The different output elements of the Hadamard transform are arranged as different surfaces to be encoded, wherein each surface is an arrangement of the transform coefficients of the corresponding component in the transform. The encoder is configured to arrange the transform coefficients in the corresponding surface in an output order to provide a sequence of transform coefficients for the at least one data compression method to generate the encoded output data. The output order is arranged as at least one of a tile grid order and an n×n block "Z" order, wherein the size of the tile is greater than 1×1.

8. The encoder according to claim 7, characterized in that... The at least one data compression transformation includes run-length encoding followed by Huffman encoding.

9. The encoder according to claim 7 or 8, characterized in that... The tile grid order includes encoding tiles ranging in size from 8×8 elements to 64×64 elements, wherein the transformation coefficients are selected according to the grid order within each tile.

10. The encoder according to claim 9, characterized in that... Tiling methods include encoding tiles of sizes of 8×8, 16×16, or 32×32 elements.

11. The encoder according to claim 7 or 8, characterized in that... The n×n block "Z" sequence comprises one or more n×n "Z" sequence readouts from the top left coefficient to the top right coefficient to the bottom left coefficient and then to the bottom right coefficient in a given coefficient group, wherein the n×n "Z" sequence readouts are repeated recursively in n×n "Z" sequence between different transformation coefficient groups.

12. The encoder according to claim 7 or 8, characterized in that... The encoder is implemented to be compatible with the LCEVC standard.

13. A method for decoding encoded data in a decoder to generate decoded output data, wherein the method comprises: (i) receiving encoded data output from the method as described in any one of claims 1 to 6 at the decoder; (ii) Applying at least one inverse transform to convert the received coded data into enhancement layer data, the enhancement layer data providing instructions for enhancing the base layer reconstruction derived from a downsampled version of the input data, wherein the enhancement layer data includes residual data to provide its enhancement, wherein the residual data is subjected to at least one transform and at least one data compression method to generate the received coded data. Its features The method further includes: (iii) Decompressing the received encoded data according to the at least one data compression method to generate intermediate decoded data; and (iv) Arrange the at least one inverse transform to include an inverse Hadamard transform to convert the transform coefficients derived from the intermediate decoded data into the residual data. The transformation coefficients are decoded by reading in a tile grid or in an n×n block "Z" order, wherein the size of the tile is greater than 1×1.

14. A computer program product comprising a non-transitory computer-readable storage medium having computer-readable instructions stored thereon, the computer-readable instructions being executable by a computerized means including processing hardware to perform the method according to any one of claims 1-6 and 13.

15. A decoder for decoding encoded data to generate decoded output data, wherein the decoder comprises: (i) an input for receiving encoded data output from the encoder as described in any one of claims 7 to 12; (ii) A data processing arrangement for applying at least one inverse transform to convert received coded data into enhancement layer data, the enhancement layer data providing instructions for enhancing a base layer reconstruction derived from a downsampled version of the input data, wherein the enhancement layer data includes residual data to provide its enhancement, wherein the residual data is subjected to at least one transform and at least one data compression method to generate the coded data. Its features The decoder further includes: (iii) A decompression arrangement for decompressing the received encoded data according to the at least one data compression method to generate intermediate decoded data; and (iv) An inverse transform arrangement for applying the at least one inverse transform as an inverse Hadamard transform to convert the transform coefficients derived from the intermediate decoded data into the residual data. The transformation coefficients are decoded by reading in a tile grid or in an n×n block "Z" order, wherein the size of the tile is greater than 1×1.

Images