Entropy coding for signal enhancement coding

Abstract

There is provided a method of encoding a video signal, the method comprising: receiving an input frame; processing the input frame to generate at least one set of residual data, the residual data enabling a decoder to reconstruct the input frame from a reconstructed reference frame; and applying a run length encoding operation to the residual data set, where the run length encoding operation includes generating a run length encoded byte stream, the run length encoded byte stream including a set of symbols representing non-zero data values of the residual data set and counts of consecutive zero values of the residual data set. In some embodiments, the method includes applying a Huffman coding operation to the set of symbols. A decoding method and apparatus, and a computer readable medium are also provided.

Description

BACKGROUND

[0001] Hybrid backward compatible encoding techniques have previously been proposed in, for example, WO 2014 / 170819 and WO 2018 / 046940, the contents of which are incorporated herein by reference.

[0002] A method is proposed therein which parses a data stream into a first portion of encoded data and a second portion of encoded data; implements a first decoder to decode the first portion of encoded data into a first rendition of a signal; implements a second decoder to decode the second portion of encoded data into reconstruction data specifying how to modify the first rendition of the signal; and applies the reconstruction data to the first rendition of the signal to produce a second rendition of the signal.

[0003] Another method is also proposed therein in which a set of residual elements are available to reconstruct a rendition of a first temporal sample of a signal. A set of spatio-temporal correlation elements associated with the first temporal sample is generated. The set of spatio-temporal correlation elements indicates a degree of spatial correlation between a plurality of residual elements and a degree of temporal correlation between first reference data based on the rendition and second reference data based on a rendition of a second temporal sample of the signal. The set of spatio-temporal correlation elements is used to generate output data.

[0004] Typical video encoding techniques involve applying entropy encoding operations to output data. There is a need for a low complexity, simple and fast entropy encoding scheme to be applied to the reconstruction data or residual elements or other similar residual data of the techniques proposed above. SUMMARY

[0005] According to an aspect of the invention, there is provided a method of entropy encoding or decoding residual data, wherein the residual data is available to correct or enhance a base stream, such as data of a video frame encoded using legacy video encoding techniques.

[0006] According to an aspect of the invention, there is provided a method of encoding a video signal. The method comprises receiving an input frame; processing the input frame to generate residual data, the residual data enabling a decoder to reconstruct the input frame from a reconstructed reference frame; and applying a run length encoding operation to the residual data, wherein the run length encoding operation comprises generating a run length encoded byte stream comprising a set of symbols representing non-zero data values of the residual data and counts of consecutive zero values of the residual data.

[0007] In this way, the encoding method provides a low complexity, simple and fast solution for data compression of the residual data. The solution exploits unique characteristics of the residual data, such as the relative occurrence rate of zero values, the possible grouping of zero values based on the scanning order of the transform process generating the residual data (if a transform is used) and the relative diversity of data values and their possible relative frequency / rarity in the residual data.

[0008] The set of symbols can be sequential in the encoded byte stream. The count of consecutive zero values can also be referred to as a run of zeros. The residual data for which the run length encoding operation is performed can represent quantized residual data, preferably a quantized set of transform coefficients. The quantized set of transform coefficients can be ordered by layer, i.e. by the same type of coefficient set, by plane, by quality level, or by surface. These terms are further described and defined herein. The symbols can be sequential in the corresponding scanning order of the transform operation, so that the residual data can be easily matched with the reconstructed reference frame.

[0009] Preferably, the run length encoding operation comprises encoding non-zero data values of the residual data into symbols of at least a first type and encoding the count of consecutive zero values into symbols of a second type, so that the residual data is encoded as a sequence of different types of symbols. Thus, the residual data can be encoded as a sequence of symbols comprising data value types and runs of zero types. The types facilitate speed and ease of decoding at the decoder and also facilitate another entropy encoding operation, as detailed below. Structuring the data into a minimized set of fixed length codes of different types facilitates the subsequent steps.

[0010] In certain embodiments, the run length encoding operation comprises encoding data values of the residual data into symbols of a first type and symbols of a third type, each comprising a portion of the data value, so that the portions can be combined at the decoder to reconstruct the data value. Each type of symbol can have a fixed size and can be a byte. Data values larger than a threshold or larger than the size available in a type of symbol can be easily transmitted or stored using a byte stream. Structuring the data into bytes or symbols facilitates not only decoding but also entropy encoding of the fixed length symbols, as elucidated below. Introducing a third type of symbol allows for three types of symbols that are easily distinguishable from each other at the decoder side. In case of types of symbols, the terms block, byte (where the symbol is a byte), context or type can be used.

[0011] The run length encoding operation can comprise comparing the size of each data value of the residual data to be encoded to a threshold and encoding each data value into a symbol of a first type when the size is below the threshold and encoding a portion of each data value into a symbol of the first type and encoding a portion of each data value into a symbol of a third type when the size is above the threshold.

[0012] If the size is above a threshold, the method may include setting a flag in a symbol of the first type of symbol indicating that a portion of the represented data value is encoded into another symbol of a third type of symbol. The flag may be an overflow flag or overflow bit and, in some instances, may be the least significant bit of a symbol or byte. In the case where the least significant bit of a symbol is a flag, the data value or portion of the data value may be included in the remaining bits of the byte. Setting the overflow bit as the least significant bit facilitates ease of combination with subsequent symbols.

[0013] The method may further include inserting a flag in each symbol that indicates the type of symbol to be encoded next in the run-length encoded byte stream. The flag may be an overflow bit as described above or may further be a run flag or run bit that indicates whether the next symbol includes a run of zeros or a data value or a portion of a data value. In the case where the flag indicates whether the next symbol includes a run (or a count of consecutive zeros), the flag may be a sign bit, preferably the most significant bit of the symbol. The count or data value of consecutive zeros may be included in the remaining bits of the symbol. Thus, a 'run of zeros' or second type symbol includes 7 available bits of a byte for the count and a 'data' or first type symbol includes 6 available bits for data values ​​where the overflow bit is not set and 7 available bits for data values ​​where the overflow bit is set.

[0014] In general, the flag may be different for each type of symbol and may indicate the type of symbol that follows in the stream.

[0015] In a preferred embodiment of the above aspect, the method further includes applying another entropy encoding operation to the set of symbols generated by the run-length encoding operation. Thus, the fixed-length symbols intentionally generated by the structure of the run-length encoding operation can be converted into variable-length codes to reduce the overall size of the data. The run-length encoding structure is designed to form high-frequency fixed-length symbols to facilitate improvement of the other entropy encoding operation and overall reduction of data size. The other entropy encoding operation utilizes the probability or frequency of occurrence of symbols in the byte stream formed by the run-length encoding operation.

[0016] In preferred examples, the other entropy encoding operation is a Huffman encoding operation or an arithmetic encoding operation. Preferably, the method further comprises applying a Huffman encoding operation to the set of symbols to generate Huffman encoded data comprising a set of codes representing the run length encoded byte stream. The Huffman encoding operation receives the symbols of the run length encoded byte stream as input and outputs a plurality of variable length codes in a bit stream. Huffman encoding is an encoding technique that can efficiently reduce fixed length codes. The structure of the run length encoded bytes means that the two types of symbols are highly likely to be copied frequently, meaning that each cycle symbol will only require a few bits. Across a plane, the same data values are likely to be copied (especially in the case of quantized residuals) and so the variable length codes can be small (but repeated).

[0017] Huffman encoding is also well optimized for software implementation (as intended here) and is computationally efficient. It uses minimal memory and has faster decoding when compared to other entropy encoding techniques, as the processing steps only involve stepping through a tree. The computational benefits arise from the shape and depth of the tree formed by the particular design of the run length encoding operation.

[0018] The encoding operation can be performed on a set of coefficients, i.e. a layer, a plane of a frame, a quality level, or a complete surface or frame. That is, the statistics or parameters of each operation can be determined based on the set of data values to be encoded and the zero data values or symbols representing those values.

[0019] The Huffman encoding operation can be a canonical Huffman encoding operation, such that the Huffman encoded data comprises a code length for each unique symbol in the set of symbols, the code length representing the length of the code used to encode the corresponding symbol. Canonical Huffman encoding facilitates a reduction in the encoding parameters that need to be signalled between the encoder and decoder in the metadata by ensuring that the same code length is attached to a symbol in a sequential manner. The codebook of the decoder can be inferred and only the code lengths need to be sent. Thus, canonical Huffman encoders are particularly efficient for shallow trees where there are a large number of different data values.

[0020] In certain examples, the method further comprises comparing a data size of at least a portion of the run length encoded byte stream with a data size of at least a portion of the Huffman encoded data; and outputting the run length encoded byte stream or the Huffman encoded data in the output bit stream based on the comparison. Each block or segment of input data is thus selectively sent to reduce the overall data size, ensuring that the decoder can easily distinguish between the encoding types. The difference can be within a threshold or tolerance, such that although one scheme can produce smaller data, the computational efficiency can mean that one scheme is preferred.

[0021] To distinguish the schemes, the method can further comprise adding a flag to the configuration metadata accompanying the output bitstream indicating whether the bitstream represents a run-length encoded byte stream or a Huffman encoded data. This is a fast and efficient way of signaling. Alternatively, the decoder can recognize the scheme used by the structure of the data, e.g. Huffman encoded data can have a header part and a data part, while run-length encoded data can only comprise a data part and thus the decoder can be able to distinguish the two schemes.

[0022] The Huffman encoding operation can comprise generating separate frequency tables for the symbols of the first type and the symbols of the second type. Furthermore, separate Huffman encoding operations can be applied for each type of symbol. These concepts contribute to the particular efficiency of the variable length coding scheme. For example, in case the same data values are copied across a plane, as is likely in video coding where a scene can have similar colors or errors / enhancements, these values can only need a few codes (where the codes do not morph due to 'zero's run' type of symbols). Thus, this embodiment of the invention is particularly advantageous for use in combination with the run-length encoding operation described above. As shown above, in case the values are quantized, the symbols here can be copied and there can not be too many different values within the same frame or coefficient group.

[0023] Using different frequency tables for each type of symbol provides particular benefits of the Huffman encoding operation after a run-length encoding operation of a particular design.

[0024] The method can thus comprise identifying the type of the symbol to be encoded next, selecting a frequency table based on the type of the next symbol, decoding the next symbol using the selected frequency table and outputting the encoded symbol in sequence. The frequency table can correspond to a unique codebook.

[0025] The method can comprise generating a stream header that can comprise an indication of the plurality of code lengths, such that a decoder can derive the code lengths and the corresponding symbols for the canonical Huffman decoding operation. For example, the code lengths can be the length of each code used to encode a particular symbol. The code lengths thus have a corresponding code and a corresponding symbol. The stream header can be a first type of stream header and further comprise an indication of the symbols associated with respective ones of the plurality of code lengths. Alternatively, the stream header can be a second type of stream header and the method can further comprise ordering the plurality of code lengths in the stream header based on a predetermined order of the symbols corresponding to each of the code lengths, such that the code lengths can be associated with the corresponding symbols at the decoder. Each type will depend on the symbols to be decoded and provide an efficient signaling of the encoding parameters. The code lengths can be signaled as a difference between a length and another length, preferably the smallest code length signaled.

[0026] The second type of stream header can further include a flag indicating that a symbol in a predetermined order of a set of possible symbols is not present in the set of symbols in the run-length encoded byte stream. In this way, only the required length is included in the header.

[0027] The method can further include comparing the number of unique codes in the Huffman encoded data to a threshold and generating the first type of stream header or the second type of stream header based on the comparison.

[0028] The method can further include comparing the number of non-zero symbols or data symbols to a threshold and generating the first type of stream header or the second type of stream header based on the comparison.

[0029] According to another aspect of the present disclosure, a method of decoding a video signal can be provided, the method including retrieving an encoded bitstream; decoding the encoded bitstream to generate residual data, and reconstructing an original frame of the video signal from the residual data and a reconstructed reference frame, wherein the step of decoding the encoded bitstream includes applying a run-length encoding operation to generate the residual data, wherein the run-length encoding operation includes identifying a set of symbols representing non-zero data values of the residual data and a set of symbols representing counts of consecutive zero values of the residual data; parsing the sets of symbols to derive the non-zero data values and the counts of consecutive zero values of the residual data; and generating the residual data from the non-zero data values and the counts of consecutive zero values.

[0030] The run-length encoding operation can include identifying symbols of a first type of symbol representing non-zero data values of the residual data; identifying symbols of a second type of symbol representing counts of consecutive zero values, such that a sequence of different types of symbols is decoded to generate the residual data; and parsing the set of symbols according to the respective type of each symbol.

[0031] The run-length encoding operation can include identifying symbols of a first type of symbol and symbols of a third type of symbol, the first and third types of symbol each representing a portion of a data value; parsing the symbols of the first type of symbol and the symbols of the third type of symbol to derive the portions of the data value; and combining the derived portions of the data value into the data value.

[0032] The method can further include retrieving, from the symbols of the first type of symbol, an overflow flag indicating whether a portion of a data value of the symbol of the first type of symbol is included in a subsequent symbol of a third type of symbol.

[0033] The method can further include retrieving, from each symbol, a flag indicating a subsequent type of symbol expected in the set of symbols.

[0034] An initial symbol can be assumed to be of the first type of symbol, such that the initial symbol is parsed to derive at least a portion of a data value.

[0035] The step of decoding the encoded bitstream can comprise applying a Huffman encoding operation to the encoded bitstream to generate the set of symbols representing non-zero data values of the residual data and the set of symbols representing counts of consecutive zero values of the residual data, wherein the step of applying a run length encoding operation is performed on the set of symbols to generate the residual data.

[0036] The Huffman encoding operation can be a canonical Huffman encoding operation.

[0037] The method can further comprise retrieving a flag from configuration metadata accompanying the encoded bitstream indicating whether the encoded bitstream comprises a run length encoded byte stream or Huffman encoded data; and selectively applying the Huffman encoding operation based on the flag.

[0038] The method can further comprise identifying an initial type of symbol expected to be derived in the encoded bitstream; applying a Huffman encoding operation to the encoded bitstream based on a set of encoding parameters associated with the expected initial type of symbol to derive an initial symbol; retrieving a flag from the initial symbol indicating a subsequent type of symbol expected to be derived from the bitstream; and further applying the Huffman encoding operation to the bitstream based on a set of encoding parameters associated with the subsequent type of symbol expected to be derived from the bitstream to derive a subsequent symbol.

[0039] The method can further comprise iteratively retrieving a flag from a decoded symbol indicating a subsequent type of symbol expected to be derived from the bitstream and applying the Huffman encoding operation to the bitstream based on a set of encoding parameters associated with the subsequent type of symbol expected to be derived from the bitstream to derive a subsequent symbol.

[0040] The method can therefore comprise the steps of decoding a symbol using a Huffman encoding operation, decoding a symbol using a run length encoding operation, identifying a type of symbol expected next according to the run length encoding operation and decoding a next symbol using the Huffman encoding operation.

[0041] The method can further comprise retrieving a stream header comprising an indication of a plurality of code lengths to be used for a canonical Huffman encoding operation; associating each code length with a respective symbol according to the canonical Huffman encoding operation; and identifying a code associated with each symbol based on the code lengths according to the canonical Huffman encoding operation. The codes can therefore be associated with the same sequential symbols as the code lengths.

[0042] The stream header can comprise an indication of a symbol associated with a respective code length of the plurality of code lengths and the step of associating each code length with a symbol comprises associating each code length with a corresponding symbol in the stream header.

[0043] The step of associating each code length with a corresponding symbol includes associating each code length with a symbol in a predetermined set of symbols in an order determined by the order in which each code length is retrieved.

[0044] The method can further include not associating a symbol in the predetermined set of symbols with a corresponding code length, wherein the flag of the stream header indicates that no code length exists in the stream header for the symbol.

[0045] According to an aspect of the application, there is provided a method of encoding a video signal. The method comprises receiving an input frame; processing the input frame to generate residual data, the residual data enabling a decoder to reconstruct the input frame from a reconstructed reference frame; and applying a Huffman encoding operation to the residual data, wherein the Huffman operation comprises generating an encoded bitstream comprising a set of codes encoding a set of symbols representing the residual data.

[0046] According to another aspect of the application, there can be provided a method of decoding a video signal, the method comprising retrieving an encoded bitstream; decoding the encoded bitstream to generate residual data and reconstructing an original frame of the video signal from the residual data and a reconstructed reference frame, wherein the step of decoding the encoded bitstream comprises applying a Huffman encoding operation to generate the residual data, wherein the Huffman encoding operation comprises generating a set of symbols representing the residual data by comparing the encoded bitstream to reference codes mapped to corresponding symbols and generating the residual data from non-zero data values and counts of consecutive zero values.

[0047] According to another aspect, there can be provided an apparatus for encoding a data set into an encoded data set. The apparatus is configured to encode an input video according to the steps described above. The apparatus can comprise a processor configured to perform the method of any of the aspects described above.

[0048] According to another aspect, there can be provided an apparatus for decoding a data set into a video reconstructed from the data set. The apparatus is configured to decode an output video according to the steps described above. The apparatus can comprise a processor configured to perform the method of any of the aspects described above.

[0049] An encoder and a decoder can also be provided.

[0050] According to further aspects of the application, there can be provided computer readable media which, when executed by a processor, cause the processor to perform any of the methods of the aspects described above. BRIEF DESCRIPTION OF DRAWINGS

[0051] Examples of systems and methods according to the application will now be described with reference to the accompanying drawings, in which:

[0052] Figure 1 A high level schematic of the encoding process is shown;

[0053] Figure 2 A high-level diagram showing the decoding process;

[0054] Figure 3 A high-level diagram showing the process of forming residual data;

[0055] Figure 4 A high-level diagram showing the process of forming additional residual data at different quality levels;

[0056] Figure 5 A high-level diagram showing the process of reconstructing a frame from residual data;

[0057] Figure 6 Display hierarchical data structures;

[0058] Figure 7 Shows another hierarchical data structure;

[0059] Figure 8 Demonstrate the example encoding process;

[0060] Figure 9 Demonstrate the example decoding process;

[0061] Figure 10 showing the structure of a first data symbol;

[0062] Figure 11 showing the structure of a second data symbol;

[0063] Figure 12 Show the structure of the run symbol;

[0064] Figure 13 Show the structure of the first stream head;

[0065] Figure 14 Show the structure of the second stream head;

[0066] Figure 15 Show the structure of the third stream head;

[0067] Figure 16 Show the structure of the fourth stream head;

[0068] 17A to 17E show a Huffman tree; and

[0069] Figure 18 Shows the run-length encoding state machine. DETAILED DESCRIPTION

[0070] The present invention relates to methods. In particular, the present invention relates to methods for encoding and decoding signals. Processing data may include, but is not limited to, obtaining, deriving, outputting, receiving, and reconstructing data.

[0071] The encoding techniques discussed herein are flexible, adaptable, efficient and computationally inexpensive encoding formats that combine a video encoding format, a base codec (e.g., AVC, HEVC or any other current or future codec) with enhancement levels of encoded data that are encoded using different techniques.

[0072] The techniques use a reduced sample source signal that is encoded using a base codec to form a base stream. An enhancement stream is formed using an encoded set of residuals that correct or enhance the base stream, for example, by increasing resolution or by increasing frame rate. There can be multiple levels of enhancement data in a hierarchical structure. Notably, it is generally expected that the base stream can be decoded by a hardware decoder, while the enhancement stream is expected to be suitable for software processing implementation with suitable power consumption.

[0073] There is a need for methods and systems for efficiently transmitting and storing enhancement encoding information.

[0074] Any entropy encoding operations used in the new encoding techniques are tailored to the specific requirements or constraints of the enhancement stream and have low complexity as essential. Such requirements or constraints include: the possible reduction in computational power resulting from the need for software decoding of the enhancement stream; the need for combination of the decoded set of residuals with the decoded frames; the possible structure of the residual data, i.e., a relatively high proportion of zero values and highly variable data values over a large range; the nuances of the input quantization block of coefficients; and the structure of the enhancement stream as a set of discrete residual frames separated into planes, layers, etc. The entropy encoding must also be suitable for multiple enhancement levels within the enhancement stream.

[0075] The inventors' research has established that modern entropy encoding schemes used in video, such as context-based adaptive binary arithmetic coding (CABAC) or context adaptive variable length coding (CAVLC) are unlikely to be suitable. For example, given the structure of the input data, a predictive mechanism can be unnecessary or can not produce enough benefit to compensate for its computational burden. Moreover, in general, arithmetic coding is computationally more expensive and software implementation is often undesirable.

[0076] It should be noted that the constraints placed on the enhancement stream mean that simple and fast entropy encoding operations are essential to enable the enhancement stream to efficiently correct or enhance individual frames of the base decoded video. It should be noted that in some scenarios, the base stream is also substantially decoded at the same time before combination, which puts a strain on resources.

[0077] This current document preferably meets the requirements of the following ISO / IEC documents: "Call for Proposals for Low Complexity Video Coding Enhancements", ISO / IEC JTC1 / SC29 / WG11 N17944, Macau, China, October 2018 and "Requirements for Low Complexity Video Coding Enhancements", ISO / IEC JTC1 / SC29 / WG11 N18098, Macau, China, October 2018. Furthermore, the methods described herein can be incorporated into the product supplied by V-Nova International Limited.

[0078] The general structure of the proposed coding scheme to which the presently described techniques can be applied uses a reduced-sampled source signal encoded by a base codec, adds a level of correction data to the decoded output of the base codec to generate a corrected image, and then adds another level of enhancement data to an increased-sampled version of the corrected image.

[0079] Thus, the streams are considered to be a base stream and an enhancement stream. Notably, it is generally expected that the base stream can be decoded by a hardware decoder, while the enhancement stream is suitable for software processing implementation with suitable power consumption.

[0080] This structure forms multiple degrees of freedom, allowing great flexibility and adaptability to many scenarios, making the coding format suitable for many use cases, including Over-the-Top (OTT) transmission, live streaming, live Ultra-HD (UHD) broadcast, etc.

[0081] Although the decoded output of the base codec is not intended for viewing, it is a fully decoded video at a lower resolution, making the output compatible with existing decoders, and can also be used as a lower resolution output if deemed suitable.

[0082] In general, a residual refers to the difference between the values of a reference array or frame and the actual array or frame of data. It should be noted that this generalized example is agnostic to the nature of the encoding operation performed and the nature of the input signal. A reference to "residual data" as used herein refers to data derived from a set of residuals, such as the set of residuals themselves or the output of a set of data processed to perform operations on the set of residuals.

[0083] In certain examples described herein, multiple encoded streams can be generated and sent independently to a decoder. That is, in an example method of encoding a signal, the signal can be encoded using at least two encoding levels. A first encoding level can be performed using a first encoding algorithm and a second level can be encoded using a second encoding algorithm. The method can include obtaining a first portion of a bitstream by encoding the first level of the signal, obtaining a second portion of the bitstream by encoding the second level of the signal, and sending the first portion of the bitstream and the second portion of the bitstream as two independent bitstreams.

[0084] It should be noted that the entropy encoding techniques presented herein are not limited to the multiple LoQs presented in the figures, but provide utility in any residual data used to provide enhancement or correction to video frames reconstructed from encoded streams encoded using legacy video encoding techniques such as HEVC. For example, the utility of the encoding techniques to encode residual data for different LoQs is particularly beneficial.

[0085] It should be noted that examples of encoding and decoding schemes are provided Figures 1 to 5 wherein the entropy encoding techniques presented herein can provide utility, but it should be understood that the encoding techniques can be used in general to encode residual data.

[0086] Certain examples described herein relate to generalized encoding and decoding processes that provide a hierarchical, scalable, flexible encoding technique. A first portion of a bitstream or a first independent bitstream can be decoded using a first decoding algorithm, and a second portion of a bitstream or a second or independent bitstream can be decoded using a second decoding algorithm. The first decoding algorithm is capable of being decoded by a legacy decoder using legacy hardware.

[0087] Returning to the initial process described above that provides two levels of enhancement within a base stream and an enhancement stream, Figure 1 An example of a generalized encoding process is depicted in the block diagram of FIG. 1. An input full resolution video 100 is processed to generate various encoded streams 101, 102, 103. A first encoded stream (encoded base stream) is produced by feeding a reduced-sampled version of the input video to a base codec (e.g., AVC, HEVC, or any other codec). The encoded base stream can be referred to as a base layer or base level. A second encoded stream (encoded level 1 stream) is produced by processing residuals obtained by taking the difference between a reconstructed base codec video and a reduced-sampled version of the input video. A third encoded stream (encoded level 0 stream) is produced by processing residuals obtained by taking the difference between an up-sampled version of a corrected version of the reconstructed encoded base video and the input video.

[0088] The downsampling operation can be applied to the input video to produce a downsampled video to be encoded by the base codec. The downsampling can be done in both vertical and horizontal directions, or alternatively only in the horizontal direction.

[0089] Each enhancement stream encoding process can not necessarily include an upsampling step. For example, in Figure 1 the first enhancement stream is conceptually a corrected stream, while the second enhancement stream is upsampled to provide an enhancement level.

[0090] Referring in more detail to the process of generating the enhancement streams, to generate the encoded level 1 stream, the encoded base stream is decoded (114) (i.e. a decoding operation is applied to the encoded base stream to generate a decoded base stream). The difference between the decoded base stream and the downsampled input video is then formed (110) (i.e. a subtraction operation is applied to the downsampled input video and the decoded base stream to generate a first set of residuals).

[0091] Here, the term residuals is used in the same way as is known in the art, i.e. the error between a reference frame and a desired frame. Here, the reference frame is the decoded base stream and the desired frame is the downsampled input video. Thus, the residuals used in the first enhancement level can be considered as a corrected video, since it 'corrects' the decoded base stream to the downsampled input video used in the base encoding operation.

[0092] Likewise, the entropy encoding operation described later is suitable for any residual data, e.g. any data associated with the set of residuals.

[0093] The difference is then encoded (115) to generate the encoded level 1 stream (102) (i.e. an encoding operation is applied to the first set of residuals to generate a first enhancement stream).

[0094] As shown above, the enhancement streams can comprise a first level of enhancement 102 and a second level of enhancement 103. The first level of enhancement 102 can be considered as a corrected stream. The second level of enhancement 103 can be considered as another level of enhancement that converts the corrected stream to the original input video.

[0095] The second level of enhancement 103 is formed by encoding another set of residuals, which are the difference (119) between an upsampled (117) version of the decoded (118) level 1 stream and the input video 100.

[0096] As mentioned, the upsampled stream is compared with the input video, which forms another set of residuals (i.e. a differencing operation is applied to the reformed upsampled stream to generate another set of residuals). The other set of residuals is then encoded (121) as an encoded level 0 enhancement stream (i.e. an encoding operation is then applied to the other set of residuals to generate another encoded enhancement stream).

[0097] Thus, as Figure 1 explained in the introduction and described above, the output of the encoding process is a base stream 101 and one or more enhancement streams 102, 103, which preferably comprise a first level of enhancement and another level of enhancement.

[0098] Figure 2 A corresponding generalized decoding process is depicted in the block diagram of Fig. 2. The decoder receives the three streams 101, 102, 103 generated by the encoder as well as the header containing additional decoding information. The encoded base stream is decoded by a base decoder corresponding to the base codec used in the encoder and its output is combined with the decoded residuals obtained from the encoded level 1 stream. The combined video is upsampled and further combined with the decoded residuals obtained from the encoded level 0 stream.

[0099] In the decoding process, the decoder can parse the headers (global configuration, picture configuration, data block) and configure the decoder based on those headers. To reform the input video, the decoder can decode each of the base stream, the first enhancement stream and the further enhancement stream. The frames of the streams can be synchronized and then combined to derive the decoded video.

[0100] In each of Figure 1 and 2 , the level 0 and level 1 encoding operations can comprise steps of transformation, quantization and entropy encoding. Similarly, at the decoding stage, the residuals can pass through an entropy decoder, a dequantizer and an inverse transformation module. Any suitable encoding and corresponding decoding operations can be used. However, preferably, the level 0 and level 1 encoding steps can be performed in software.

[0101] In summary, the methods and devices herein are based on an overall algorithm, which is built via existing encoding and / or decoding algorithms (e.g. MPEG standards such as AVC / H.264, HEVC / H.265, etc. and non-standard algorithms such as VP9, AV1, etc.) to serve as a baseline for the enhancement layers for different encoding and / or decoding algorithms accordingly. The idea behind the overall algorithm is to encode / decode the video frames in a hierarchical manner, as opposed to using the block-based approach used in the MPEG series algorithms. Encoding the frames in a hierarchical manner comprises generating residuals for the full frame and then generating residuals for the extracted frames, etc.

[0102] Video compression residual data for a full-size video frame can be referred to as LoQ-0 (e.g., 1920x1080 for an HD video frame), while video compression residual data for a decimated frame can be referred to as LoQ-x, where x represents the number of times hierarchical decimation has occurred. In Figure 1 and 2 In the described example of FIG. 1, variable x has a maximum value of 1 and thus there are 2 hierarchical levels that will generate compression residuals.

[0103] Figure 3 An example of how LoQ-1 can be generated at an encoding device is illustrated. In the current figure, the AVC / H.264 encoding / decoding algorithm is used as the baseline algorithm to describe the overall algorithm and method, but it should be understood that other encoding / decoding algorithms can be used as the baseline algorithm without any impact on the way the overall algorithm works.

[0104] Of course, it will be understood that Figure 3 The blocks of FIG. 1 are merely examples of how the broad concepts can be implemented.

[0105] Figure 3 The figure of FIG. 2 illustrates the process of generating entropy encoded residual data for the LoQ-1 hierarchical level. In the example, the first step is to decimate the incoming uncompressed video by a factor of 2. This decimated frame is then passed through the base encoding algorithm (in this case, the AVC / H.264 encoding algorithm), where an entropy encoded reference frame is then generated and stored. A decoded version of the encoded reference frame is then generated, and the difference between the decoded reference frame and the decimated frame (LoQ-1 residual) will form the input to the transform block.

[0106] The transform (e.g., a Hadamard-based transform in the illustrated example) converts this difference into 4 components (or planes), namely A (average), H (horizontal), V (vertical), and D (diagonal). These components are then quantized via the use of a variable called'stepwidth' (e.g., residual values can be divided by the stepwidth and the nearest integer value selected). A suitable entropy encoding process is the subject of the present disclosure and described in detail below. These quantized residuals are then entropy encoded in order to remove any redundant information. The quantized, encoded coefficients or components (Ae, He, Ve, and De) are then placed in a serial stream, with a definition packet inserted at the beginning of the stream, using a file serialization routine to accomplish this final stage. The packet data can include, for example, the specifications of the encoder, the type of increased sampling to be employed, whether to discard the A and D planes, and other information to enable a decoder to decode the stream, among such information.

[0107] Both the reference data (entropy encoded half-size baseline frame) and the entropy encoded LoQ-1 residual data can be buffered, transmitted, or stored for use by a decoder during the reconstruction process.

[0108] To produce the LoQ-0 residual data, the quantized output is bifurcated and inverse quantization and transform procedures are performed on it in order to reconstruct the LoQ-1 residual, which is then added to the decoded reference data (encoded and decoded) in order to obtain a video frame closely resembling the originally extracted input frame.

[0109] This process ideally mimics the decoding process and therefore does not use the originally extracted frame.

[0110] Figure 4 An example of how LoQ-0 can be generated at the encoding device is illustrated. To derive the LoQ-0 residual, the reconstructed LoQ-1 size frame is derived as described in the previous section.

[0111] The next step is to perform an up-sampling of the reconstructed frame to full size (by a factor of 2). At this point, various algorithms can be used to enhance the up-sampling process, such as nearest, bilinear, sharp or cubic algorithms. This reconstructed full size frame, referred to as the 'predicted' frame, is then subtracted from the original uncompressed video input which forms some residual (the LoQ-1 residual).

[0112] Similar to the LoQ-1 process, the difference is then transformed, quantized, entropy encoded and file serialized, which then forms the third and final data. This can be buffered, transmitted or stored for later use by the decoder. As can be seen, a component referred to as the "predicted average" (described below) can be derived using the up-sampling process and used in place of the A (average) component to further improve the efficiency of the encoding algorithm.

[0113] Figure 5 A decoding process is schematically shown as to how it can be performed in a particular example. The entropy encoded data, LOQ-1 entropy encoded residual data and LOQ-0 entropy encoded residual data (e.g. as file serialized encoded data). The entropy encoded data includes a reduced size (e.g. half size, i.e. having dimensions W / 2 and H / 2 relative to a full frame having dimensions W and H) encoded base.

[0114] The entropy encoded data is then decoded using a decoding algorithm corresponding to the algorithm that has been used to encode those data (in the example, an AVC / H.264 decoding algorithm). At the end of this step, a decoded video frame having a reduced size (e.g. half size) is produced (indicated in the example as AVC / H.264 video).

[0115] In parallel, LoQ-1 entropy encoded residual data is decoded. As discussed above, LoQ-1 residual is encoded using four coefficients or components (A, V, H and D) which, as shown in the figure, have a size of one quarter of the full frame size, i.e. W / 4 and H / 4. This is because, as discussed in previous patent applications US 13 / 893,669 and PCT / EP2013 / 059847, the contents of which are incorporated herein by reference, the four components contain all the information about the residual and are generated by applying a 2x2 transform kernel to the residual (for LoQ-1, the size would be W / 2 and H / 2, i.e. the same size of the entropy encoded data at reduced size). The four components are entropy decoded, then dequantized and finally transformed back to the original LoQ-1 residual by using an inverse transform (in this case, a 2x2 Hadamard inverse transform).

[0116] The decoded LoQ-1 residual is then added to the decoded video frame to produce a reconstructed video frame at reduced size (in this case, half size), identified as half 2D size reconstruction.

[0117] This reconstructed video frame is then upsampled to bring it to full resolution (thus, in this example, from half width (W / 2) and half height (H / 2) to full width (W) and full height (H)) using, for example, a bilinear, bicubic, sharp, etc. upsampling filter. The upsampled reconstructed video frame will be the predicted frame (full size, WxH) to which the LoQ-0 decoded residual is then added.

[0118] In particular, LoQ-0 encoded residual data is decoded. As discussed above, LoQ-0 residual is encoded using four coefficients or components (A, V, H and D) which, as shown in the figure, have a size of half of the full frame size, i.e. W / 2 and H / 2. This is because, as discussed in previous patent applications US 13 / 893,669 and PCT / EP2013 / 059847, the contents of which are incorporated herein by reference, the four components contain all the information about the residual and are generated by applying a 2x2 transform kernel to the residual (for LoQ-0, the size would be W and H, i.e. the same size of the full frame). The four components are entropy decoded (see below procedure), then dequantized and finally transformed back to the original LoQ-0 residual by using an inverse transform (in this case, a 2x2 Hadamard inverse transform).

[0119] The decoded LoQ-0 residual is then added to the predicted frame to produce a reconstructed full video frame - the output frame.

[0120] In Figure 6The data structure is represented in an exemplary manner. As discussed, the above description has been made with reference to a particular size and baseline algorithm, but the above-described approach is applicable to other sizes and or baseline algorithms, and the above description is given by way of example only of the more general concepts described.

[0121] In the encoding / decoding algorithm described above, there are generally 3 planes (e.g., YUV or RGB) with two levels of quality, LoQ, in each plane, which are described as LoQ-0 (or top level, full resolution) and LoQ-1 (or lower level, reduced size resolution, e.g., half resolution). Each plane can represent a different color component of the video signal.

[0122] Each LoQ contains four components or layers, namely A, H, V, and D. This gives a total of 3 x 2 x 4 = 24 surfaces, of which 12 are full size (e.g., W x H) and 12 are reduced size (e.g., W / 2 x H / 2). Each layer can include a coefficient value for a particular one of the components, e.g., layer A can include a top-left A coefficient value for each 2 x 2 block of an input image or frame.

[0123] Figure 7 An alternative view of the proposed hierarchical data structure is illustrated. The encoded data can be separated into information blocks. Each payload can be ordered into the information blocks in a hierarchical manner. That is, each payload is grouped into a plane, then within each plane, each level is grouped into a layer and each layer includes a set of information blocks for that layer. A level represents an enhancement of each level (enhancement of a first or another level) and a layer represents a set of transform coefficients.

[0124] The method can include retrieving, for each plane, information blocks for two levels of enhancement. The method can include retrieving 16 layers for each level (e.g., if a 4 x 4 transform is used). Thus, each payload is ordered into a set of information blocks for all layers in each level and then for all layers in the next level of the plane. Then, the payload includes a set of information blocks for the layers of the first level of the next plane, etc.

[0125] Thus, the method can decode the header and output the entropy encoded coefficients grouped by plane, level, and layer belonging to the image enhancement being decoded. Thus, the output can be an (nPlanes) x (nLevel) x (nLayer) array surface with element surface [nPlanes] [nLevel] [nLayer].

[0126] It should be noted that the entropy coding techniques provided herein can be performed on each group, i.e., per surface, per plane, per level of quality (LoQ), or per layer. It should be noted that entropy coding can be performed on any residual data, not necessarily on quantized and / or transformed coefficients.

[0127] As Figure 4 and 5 illustrated, the proposed encoding and decoding operations include an entropy coding stage or operation. The entropy coding operation is proposed below to include one or more of a run length encoding component (RLE) and a Huffman encoding component. These two components can be interrelated to provide additional benefits.

[0128] As mentioned and as explained in Figure 8 , the input to the entropy encoder is a surface (e.g., residual data derived from a quantized set of residuals, as explained in this example) and the output of the process is an entropy coded version of the residuals (Ae, He, Ve, De). However, as mentioned above, it should be noted that entropy coding can be performed on any residual data, not necessarily on quantized and / or transformed coefficients. Figure 9 A corresponding high level decoder is illustrated with reversed input and output. That is, the entropy decoder takes as input the entropy coded residuals (Ae, He, Ve, De) and outputs the residual data (e.g., quantized residuals in the example illustrated here).

[0129] It should be noted that, in general, Figure 8 RLE decoding is introduced prior to Huffman decoding, but as will be apparent throughout this application, this order is not limiting and the stages can be interchangeable, interrelated, or in any order.

[0130] Similarly, it should be noted that run length encoding can be provided without Huffman encoding, and in comparative cases, run length encoding and Huffman encoding can not be performed (replaced by alternative lossless entropy encoding or no entropy encoding at all). For example, in production examples where emphasis on data compression is reduced, the encoding pipeline can not include entropy coding, but the benefits of residuals can be in hierarchical storage for distribution.

[0131] A comparative run length encoder (RLE) can compress data by encoding sequences of the same data value as a single data value and a count of the data value. For example, the sequence 555500022 can be encoded as (4,5)(3,0)(2,2). That is, there is a run of four 5s, a run of three 0s, followed by a run of two 2s.

[0132] To encode the residual data, a modified RLE encoding operation is presented. To exploit the structure of the residual data, it is proposed to encode only runs of zeros. That is, each value is transmitted as a value that transmits each zero as a run of zeros. For example, the modified RLE encoding operation described herein can be used to provide entropy encoding and / or decoding in one or more LoQ layers, such as shown in Figures 1 to 5

[0133] Thus, the entropy encoding operation includes parsing the residual data and encoding zero values as runs of consecutive zeros.

[0134] As shown above, to provide an additional level of data compression (which can be lossless), the entropy encoding operation further applies a Huffman encoding operation to the run-length encoded data.

[0135] Huffman encoding and run-length encoding have previously been paired together, for example in facsimile encoding (e.g., ITU Recommendations T.4 and T.45) and JPEG File Interchange Format, but have been superseded in the past 30 years. The current description presents techniques and methods to implement Huffman encoding as well as RLE encoding, techniques and methods to efficiently exchange data and metadata between an encoder and a decoder, and techniques and methods to reduce the overall data size of this combination when encoding residual data.

[0136] A Huffman code is an optimal prefix code for data compression (e.g., lossless compression). A prefix code is a code system in which no code works as a prefix of any other code. That is, a Huffman encoding operation takes a set of input symbols and converts each input symbol to a corresponding code. The codes are selected based on the frequency of each symbol occurring in the original data set. In this way, smaller codes can be associated with more frequently occurring symbols in order to reduce the overall size of the data.

[0137] To facilitate Huffman encoding, the input data is preferably structured as symbols. The output of the run-length encoding operation is preferably structured as a byte stream of encoded data.

[0138] In one example, the run-length encoding operation outputs two types of bytes or symbols. The first type of symbol is the value of a non-zero pixel and the second type of symbol is a run of consecutive zero values. That is, the amount of zero values that occur consecutively in the original data set.

[0139] In another example, to encode certain data values, two bytes or symbols can be combined. That is, for example, in the case of a pixel value greater than a threshold value, two bytes or symbols can be used to encode the data value in the symbol. The two symbols can be combined at the decoder to reform the original data value of the pixel.

[0140] ​Each symbol or byte may contain 6 or 7 data bits and one or more flags or bits indicating the symbol type.

[0141] In an example implementation, run-length encoded data may be efficiently encoded by inserting one or more flags in each symbol that indicate the next symbol to be expected in the byte stream.

[0142] To facilitate synchronization between the encoder and decoder, the first byte of a stream of run-length encoded data may be a symbol of the type of data value. This byte may then indicate the next type of byte that has been encoded.

[0143] In some embodiments, the flag indicating the next byte may be different depending on the type of symbol. For example, the flag or bit may be located at the beginning, end, or both of the byte. For example, at the beginning of a byte, the bit may indicate that the next symbol may be a run of zeros. At the end of a byte, the bit may be an overflow bit, indicating that the next symbol contains a portion of a data value to be combined with the data value in the current symbol.

[0144] The following describes a specific implementation example of the run length encoding operation. RLE has three contexts: RLC_RESIDUAL_LSB, RLC_RESIDUAL_MSB and RLC_ZERO_RUN. Figure 10 、 11 The structure of these bytes is described in 1 and 12.

[0145] In the first type of symbols, Figure 10 , the 6 least significant bits of non-zero pixels are encoded using the RLC_RESIDUAL_LSB described in [1]. A run bit can be provided to indicate that the next byte is encoding a count of runs of zeros. If the pixel value does not fit within the 6 data bits, the overflow bit can be set. When the overflow bit is set, the context of the next byte will be of type RLC_RESIDUAL_MSB. That is, the next symbol will include a data byte, which can be combined with the bits of the current symbol to encode the data value. When the overflow bit is set, the next context cannot be a run of zeros and therefore the symbol can be used to encode data.

[0146] Figure 10 Indicates an instance of this type of symbol. If the data value to be encoded is greater than a threshold of 64, or the pixel value does not fit within the 6 data bits, then the overflow bit may be set by the encoding process. If the overflow bit is set to the least significant bit of the byte, then the remaining bits of the byte may encode the data. If the pixel value fits within the 6 bits, then the overflow bit may not be set and the run bit may be included, which indicates whether the next symbol is a data value or a run of zeros.

[0147] Figure 11The second type of symbol RLC_RESIDUAL_MSB explained in the background encodes bits 7 to 13 of a pixel value that does not fit in 6 data bits. Bit 7 of this type of symbol encodes whether the next byte is a run of zeros.

[0148] Figure 12 The third type of symbol RLC_ZERO_RUN explained in the background encodes 7 bits of a zero run count. That is, the symbol includes the number of consecutive zeros in the residual data. If more bits are needed to encode the count, then the run bit of the symbol is higher. The run bit can be the most significant bit of the byte. That is, in the case that a number of consecutive zeros requires more than 7 available bits, such as 178 or 255 available bits, the run bit indicates that the next bit will indicate another run of zeros.

[0149] Optionally, an additional symbol can include a second run of zeros that can be combined with the first run, or can include a set of bits that can be combined at the decoder with bits of the first symbol to indicate a value of the count.

[0150] As indicated above, to enable the decoder to start based on a known context, the first symbol in the encoded stream can be a symbol of the residual data value type, that is, can be of the RLC_RESIDUAL_LSB type.

[0151] In a particular example, RLE data can be organized into blocks. Each block can have an output capacity of 4096 bytes. RLE can switch to a new block in the following cases:

[0152] - the current block is full;

[0153] - the current RLE data is a run and less than 5 bytes remain in the current block; and / or

[0154] - the current RLE data produces a LSB / MSB pair and less than 2 bytes remain in the current block.

[0155] In summary, the run length encoding operation can be performed on residual data of a new encoding technique that includes encoding a set of data values and a count of consecutive zero values into a stream. In a particular implementation, the output of the run length encoding operation can be a stream of bytes or symbols, where each byte is one of three types or contexts. The byte indicates the next expected type of byte in the byte stream.

[0156] It should be noted that these structures are provided as examples, and different bit encoding schemes can be applied while following the functional teachings described herein. For example, the encoding of the least and most significant bits can be swapped and / or different bit lengths can be made. Also, the flag bit can be located at a different predefined location within the byte.

[0157] As indicated above, a Huffman encoding operation can be applied to the byte stream to further reduce the size of the data. The process can form a frequency table of the relative occurrence of each symbol in the stream. From the frequency table, the process can generate a Huffman tree. A Huffman code for each symbol can be generated by assigning code bits to each branch taken from the root of the tree until the symbol is reached.

[0158] In a preferred implementation of the particularity of run length encoded symbols for quantized residuals (i.e., residual data), a canonical Huffman encoding operation can be applied. Canonical Huffman codes can reduce the storage requirements of the code set depending on the structure of the input data. The canonical Huffman procedure provides a way of generating codes that implicitly contain the information of which symbol the code word is applicable to. The Huffman codes can be set under a set of conditions, for example, all codes of a given length can have consecutive values in dictionary order in the same order as the symbols they represent and shorter codes precede longer codes in dictionary order. In a canonical Huffman encoding implementation, only the code word and the length of each code need to be transmitted for the decoder to be able to reproduce each symbol.

[0159] After applying the canonical Huffman encoding operation, the data size of the output data can be compared to the data size after the run length encoding operation. If the data size is smaller, the smaller data can be transmitted or stored. The comparison of data size can be performed based on the size of a block of data, the size of a layer, plane or surface, or the overall size of a frame or video. To signal to the decoder how the data was encoded, a flag can be transmitted in the header of the data that the data was encoded using Huffman encoding, RLE encoding, or both. In an alternative implementation, the decoder can be able to identify that the data has been encoded using a particular encoding operation based on the characteristics of the data. For example, as indicated below, the data can be split into a header and a data portion, where canonical Huffman encoding is used in order to signal the code length of the encoded symbols.

[0160] Canonical Huffman encoded data includes a portion that signals the code length for each symbol to the decoder and a portion that includes the bit stream representing the encoded data. In the particular implementation presented herein, the code length can be signaled in a header portion and the data is signaled in a data portion. Preferably, the header portion will be signaled for each surface but can be signaled for each block or another subdivision depending on the configuration.

[0161] In the proposed example, the header may differ depending on the number of non-zero codes to be encoded. The number of non-zero codes to be encoded (e.g., for each surface or block) may be compared to a threshold, and the header used based on this comparison. For example, when there are more than 31 non-zero codes, the header may indicate all symbols in sequence, starting with a predetermined signal, such as 0. The length of each symbol may be signaled in sequence. When there are fewer than 31 non-zero codes, each symbol value and its corresponding code length may be signaled in the header.

[0162] Figures 13 to 16 The description header format and code length may depend on the specific implementation in which the amount of non-zero codes is written to the stream header.

[0163] Figure 13 This indicates that there may be more than 31 non-zero values ​​in the data. The header contains the minimum and maximum code lengths. The code length of each symbol is sent sequentially. A flag indicates that the length of the symbol is non-zero. The code length bits are then sent as the difference between the code length and the minimum signaled length. This reduces the overall size of the header.

[0164] Figure 14 Description similar to Figure 13 But a header used when there are less than 31 non-zero codes. The header further contains the number of symbols in the data, followed by the symbol value and the length of the codeword for that symbol, also sent as a difference.

[0165] Figure 15 and 16 Indicates additional headers to be sent in peripheral situations. For example, when the frequency is all zeros, the stream header can be as follows Figure 14 The two values ​​31 in the minimum and maximum length fields indicate a special case. When there is only one code in the Huffman tree, the stream header can be as follows Figure 16 As indicated in , a value of 0 in the minimum and maximum length fields is used to indicate a special case and the symbol value is used next.

[0166] In this latter example, when only one symbol value is present, this may indicate that only one data value is present in the residual data.

[0167] The encoding process can thus be summarized as follows −

[0168] Parse the residual data to identify consecutive counts of data values ​​and zero values.

[0169] A set of symbols is generated, where each symbol is a byte and each byte includes a data value or a run of zeros and an indication of the next symbol in the set of symbols. The symbol may include an overflow bit indicating that the next symbol includes a portion of the data value included in the current symbol or a run of zeros.

[0170] Each fixed length symbol is converted to a variable code using canonical Huffman coding. Canonical Huffman coding parses the symbols to identify the frequency of occurrence of each symbol in the set of symbols and assigns codes to the symbols based on the frequency and symbol value (for the same code length).

[0171] A set of code lengths is generated and output, each code length associated with a symbol. The code lengths are the length of each code assigned to each symbol.

[0172] The variable length codes are combined into a bit stream.

[0173] The encoded bit stream is output.

[0174] In the particular implementation described, it is noted that the maximum code length depends on the number of encoded symbols and the number of samples used to derive the Huffman frequency table. For N symbols, the maximum code length is N-1. However, for a symbol to have a k-bit Huffman code, the number of samples used to derive the frequency table needs to be at least:

[0175]

[0176] where F i is the i-th Fibonacci number.

[0177] For example, for an 8-bit symbol, the theoretical maximum code length is 255. However, for an HD 1080p video, the maximum number of samples used to derive the frequency table for the RLE context for residuals is 1920*1080 / 4 = 518 400, which is between S26 and S27. Thus, for this example, the symbol cannot have a Huffman code larger than 26 bits. For 4K, this number increases to 29 bits.

[0178] For completeness, using Figure 17, an example of Huffman coding of a set of symbols is provided, in this example the symbols are alphanumeric. As shown above, Huffman codes are optimal prefix codes that can be used for lossless data compression. A prefix code is a code system in which no code word is a prefix of any other code word.

[0179] To find the Huffman code for a given set of symbols, a Huffman tree needs to be formed. First, the symbols are sorted by frequency, for example:

[0180] Symbol Frequency A 3 B 8 C 10 D 15 E 20 F 43

[0181] The two lowest elements are removed from the list and become leaf elements, with the frequency of the parent node being the sum of the frequencies of the two lower elements. In Figure 17a A partial tree is illustrated in Figure 17.

[0182] The sorted new frequency list is:

[0183] Symbol Frequency C 10 * 11 D 15 E 20 F 43

[0184] Next, the loop is repeated, combining the two lowest elements as described in Figure 17b

[0185] The new list is:

[0186] Symbol Frequency D 15 E 20 * 21 F 43

[0187] This process is repeated until only one element remains in the list and the following corresponding table as described in Figure 17c , 17d , 17e.

[0188] Symbol Frequency * 21 * 35 F 43

[0189] Symbol Frequency F 43 * 56

[0190] Symbol Frequency * 99

[0191] Once the tree is constructed, to generate a Huffman code for a symbol, one traverses from the root to this symbol, outputting a 0 each time a left branch is taken and a 1 each time a right branch is taken. In the above example, this gives the following code:

[0192] Symbol Code Code length A 1010 3 B 1011 3 C 100 2 D 110 2 E 111 2 F 0 0

[0193] The code length of a symbol is the length of its corresponding code.

[0194] To decode a Huffman code, one starts at the root and traverses the tree, taking the left path when a 0 is read and the right path when a 1 is read. When a leaf is hit, the symbol is found.

[0195] As described above, RLE symbols can be encoded using Huffman encoding. In preferred examples, canonical Huffman encoding is used. In implementation of canonical Huffman encoding, one can use a Huffman encoding program and transform the resulting codes into canonical Huffman codes. That is, rearrange the table so that the same length is in the same order as the symbols it represents and the value of the later code will always be higher than the previous code. When the symbols are letters, they can be rearranged in alphabetical order. When the symbols are values, they can be in order of value and change the corresponding codewords to comply with the above constraints. The canonical Huffman encoding and / or other Huffman encoding methods described herein can be different from the basic example of Figures 17a to 17e .

[0196] ​In the examples described above, the Huffman encoding operation can be applied to the data symbols encoded using the modified run length encoding operation. It is further proposed to form a frequency table for each RLE context or state. For each type of symbol encoded by the RLE encoding operation, there can be a different set of Huffman codes. In one implementation, there can be a different Huffman encoder for each RLE context or state.

[0197] It is contemplated that one stream header can be sent for each RLE context or symbol type, i.e. multiple stream headers can indicate the code length for each code set, i.e. each Huffman encoder or each frequency table.

[0198] Examples of the specific implementation steps of the encoder are described below. In one implementation, the encoding of the data output by the RLE is performed in three main steps:

[0199] 1. Initialize the encoder (e.g. by the InitialiseEncode function):

[0200] a. Initialize each encoder (one encoder is initialized for each RLE state), where the corresponding frequency table is generated by the RLE;

[0201] b. Form the Huffman tree;

[0202] c. Calculate the code length for each symbol; and

[0203] d. Determine the minimum and maximum code length.

[0204] 2. Write the code length and code table to the stream header for each encoder (e.g. by the WriteCodeLengths function):

[0205] a. Assign codes to the symbols (e.g. using the AssignCodes function). It should be noted that the function for assigning codes to the symbols is slightly different from the example described above Figures 17a to 17e , as it ensures that all codes of a given length are sequential, i.e. using canonical Huffman encoding;

[0206] b. Write the minimum and maximum code length to the stream header; and

[0207] c. Write the code length for each symbol to the stream.

[0208] 3. Encode the RLE data:

[0209] a. Set the RLE current context to RLC_RESIDUAL_LSB;

[0210] b. Encode the current symbol in the input stream by the encoder corresponding to the current context;

[0211] c. Use the RLE state machine to obtain the next context (see Figure 18 ). If this is not the last symbol of the stream, go to b; and

[0212] d. If the RLE data is only one block and the encoded stream is larger than the RLE, store the RLE encoded stream rather than the Huffman encoded stream.

[0213] For each RLE block, the Huffman encoder forms a corresponding Huffman encoded portion.

[0214] The output of the encoding process described herein is thus either a Huffman encoded bitstream or a run length encoded byte stream. The corresponding decoder thus employs a corresponding Huffman decoding operation in reverse to the encoding operation. Nonetheless, there are subtle differences to the decoding operation described below to improve efficiency and accommodate the specifics of the encoding operation described above.

[0215] When the encoder signals in a header or other configuration metadata that the RLE or Huffman encoding operation has been used, the decoder can first identify that a Huffman decoding operation should be applied to the stream. Similarly, in the manner described above, the decoder can identify whether the Huffman encoding has been used according to the data stream by identifying the number of portions of the stream. If there is a header portion and a data portion, the Huffman decoding operation should be applied. If there is only a data portion for the stream, only the RLE decoding portion can be applied.

[0216] The entropy decoder performs the inverse transform of the decoder. In this example, the Huffman decoding is followed by the run length decoding.

[0217] The following example describes the decoding process where in the example of an encoded stream, the bitstream has been encoded using the canonical Huffman encoding operation and run length encoding operation as described above. However, it will be appreciated that the Huffman encoding operation and run length encoding operation can be applied separately to decode streams that are encoded in different ways.

[0218] The input to the process is an encoded bitstream. The bitstream can be retrieved from a storage file or streamed locally or remotely. The encoded bitstream is a series of sequential bits that do not have a structure that can be immediately discerned. Only by applying the Huffman encoding operation to the bitstream can a series of symbols be derived.

[0219] The process therefore first applies the Huffman encoding operation to the bitstream.

[0220] The first encoded symbol is assumed to be a symbol of the data value type. In this way, the Huffman encoding operation can use the correct codebook for the symbol type when the Huffman encoding operation uses different frequency tables or parameters.

[0221] A canonical Huffman encoding operation must first retrieve the appropriate encoding parameters or metadata to facilitate decoding the bitstream. In the example here, the parameters are a set of code lengths retrieved from the stream header. For simplicity, one set of code lengths will be described as being exchanged between the encoder and decoder, but it should be noted that, as described above, multiple sets of code lengths can be signaled so that the encoding operation can use different parameters for each type of symbol it expects to decode.

[0222] The encoding operation assigns the code lengths received in the stream header to the corresponding symbols. As shown above, the stream header can be sent in different types. When the stream header contains a set of code lengths and corresponding symbols, each code length can be associated with a corresponding symbol value. When the code lengths are sent sequentially, the decoder can associate each code length received with a symbol in a predetermined order. For example, the code lengths can be retrieved in the order 3, 4, 6, 2, etc. The encoding process can then associate each length with a corresponding symbol. Here, the order is sequential - (symbol, length) - (0, 3) (1, 4) (3, 6) (4, 2).

[0223] As Figure 15 As explained in the Background, some symbols can not be sent but a flag can indicate that the symbol has a corresponding zero code length, i.e., the symbol does not exist in the bitstream to be decoded (or is unused).

[0224] The canonical Huffman encoding operation is then able to assign a code to each symbol based on the code length. For example, when the code length is 2, the code for the symbol will be 'lx'. Where x indicates that the value is not important. In a particular implementation, the code will be '10'.

[0225] When the lengths are the same, the canonical Huffman encoding operation assigns codes based on the sequential order of the symbols. Each code is assigned so that when the bitstream is parsed, the decoder can keep checking the next bit until there is only one possible code. For example, the first sequential symbol of length 4 can be assigned the code 1110 and the second sequential symbol of length 4 can be assigned the code 1111. Thus, if the first bit of the bitstream is 1110x, when the bitstream is parsed, there is only a possibility that the first sequential symbol of length 4 has been encoded.

[0226] Thus, once the encoding operation has established the set of codes associated with each symbol, the encoding operation can move to parsing the bitstream. In some embodiments, the encoding operation will build a tree in order to establish the symbol associated with a code of the bitstream. That is, the encoding operation will take each bit of the bitstream and traverse the tree according to the value of the bit in the bitstream until a leaf is found. Once a leaf is found, the symbol associated with the leaf is output. The process continues at the root of the tree with the next bit of the bitstream. In this way, the encoding operation can output a set of symbols derived from the set of codes stored in the encoded bitstream. In a preferred embodiment, the symbols are each bytes, as described elsewhere herein.

[0227] The set of symbols can then be passed to a run length encoding operation. The run length encoding operation identifies the type of symbol and parses the symbol to extract a run of data values or zeros. From the runs of data values and zeros, the operation can reform the encoded original residual data.

[0228] As shown above, the output of the Huffman encoding is potentially a set of symbols. The challenge of the next decoding step is to extract the relevant information from those symbols, noting that each symbol includes data in a different format and each symbol will represent different information, even though the information is not immediately discernible from the symbol or byte.

[0229] In a preferred example, the encoding operation consults a state machine as explained in Figure 18 In summary, the encoding operation first assumes that the first symbol is a data value type (the data value can of course be 0). From this, the encoding operation can identify whether the byte contains an overflow flag or a run flag. Overflow and run flags are described above and explained in Figure 10

[0230] The encoding operation can first check the least significant bit of the symbol (or byte). Here, this is the overflow flag (or bit). If the least significant bit is set, then the encoding operation identifies that the next symbol will also be a data value and belongs to the RLC_RESIDUAL_MSB type. The remaining bits of the symbol will be the least significant bits of the data value.

[0231] The encoding process moves to parsing the next symbol and extracting the least significant bit as the remaining part of the data value. The bits of the first symbol can be combined with the bits of the second symbol to reconstruct the data value.

[0232] In this current symbol as explained in Figure 11 There will also be a run flag, here the most significant bit. If this flag is set, then the next symbol will be a run symbol rather than a data symbol.

[0233] In the run symbol as explained in Figure 12 The encoding operation checks the most significant bit to indicate whether the next symbol is a run symbol or a data symbol and extracts the remaining bits as a run of zeros. That is, the maximum run is 127 zeros.

[0234] If the data symbol indicates that there is no overflow (in the overflow flag, here the least significant bit), then the symbol will also contain a run bit in the most significant bit and the encoding operation will be able to identify from this bit whether the next symbol in the set is a run symbol or a data symbol. The encoding operation can therefore extract the data value from bits 6 to 1. Here, there are 31 available data values without an overflow.

[0235] As mentioned, Figure 18 ​The state machine of FIG. 1 illustrates the process.

[0236] Starting in RLC_RESIDUAL_LSB symbol, if overflow bit = 0 and run bit = 0, the next symbol will be RLC_RESIDUAL_LSB symbol. If overflow bit = 1, the next symbol will be RLC_RESIDUAL_MSB. If overflow bit = 1, there will be no run bit. If run bit = 1 and overflow bit = 0, the next bit will be RLC_ZERO_RUN symbol.

[0237] In RLC_RESIDUAL_MSB, if run bit = 0, the next symbol will be RLC_RESIDUAL_LSB symbol. If run bit = 1, the next symbol will be RLC_ZERO_RUN symbol.

[0238] In RLC_ZERO_RUN symbol, if run bit = 0, the next bit will be RLC_RESIDUAL_LSB symbol. If run bit = 1, the next bit will be RLC_ZERO_RUN symbol.

[0239] The bits can of course be reversed (0 / 1, 1 / 0, etc.) without loss of functionality. Similarly, the position of the symbol or byte of the flags is merely illustrative.

[0240] The run-length encoding operation can identify the next symbol in the set of symbols and extract the data values or runs of zeros. The encoding operation can then combine these values with zeros to reform the residual data. The order can be in the order of extraction or, alternatively, in some predetermined order.

[0241] The encoding operation can thus output the residual data that has been encoded into a byte stream.

[0242] The above describes how multiple metadata or encoding parameters can be used in the encoding process. At the decoding side, the encoding operation can include feedback between operations. That is, the next symbol expected can be extracted from the current symbol (e.g., using the overflow and run bits) and given to the Huffman encoding operation.

[0243] The Huffman encoding operation can optionally generate a separate codebook for each type of symbol (and retrieve multiple stream heads and build multiple tables of codes and symbols).

[0244] Assuming the first symbol will be a data value, the Huffman encoding operation will decode the first code to match the type of symbol. Depending on the matched symbol, the encoding operation can identify the overflow bit and / or the run bit and identify the next type of symbol. The Huffman encoding operation can then use the corresponding codebook for the type of symbol to extract the next code in the bitstream. The process uses the current decoded symbol in this iterative fashion to extract an indication of the next symbol and change the codebook accordingly.

[0245] The above has described how the decoding operation can combine the Huffman encoding operation, preferably norming the Huffman encoding operation with the run-length encoding operation to reform the residual data from the encoded bitstream. It should be understood that the techniques of the run-length encoding operation can be applied to a stream of bytes that has not been encoded using the Huffman encoding operation. Similarly, the Huffman encoding operation can be applied without the subsequent step of the run-length encoding operation.

[0246] Examples of specific implementation steps for a decoder are described below. In the example implementation, if the stream has more than one part, the following steps are performed:

[0247] 1. Read the code lengths from the stream header (e.g., using the ReadCodeLengths function):

[0248] a. Set the code length for each symbol;

[0249] b. Assign codes from the code lengths to the symbols (e.g., using the AssignCodes function); and

[0250] c. Generate a table for searching subsets of codes with the same length. Each element of the table records the first index and corresponding code for a given length (firstldx, firstCode).

[0251] 2. Decode the RLE data:

[0252] a. Set the RLE context to RLC_RESIDUAL_LSB.

[0253] b. Decode the current code, which searches the correct code length in the generated table and indexes into the code array by: firstldx - (current_code - firstCode). This is because all codes of a given length are sequential by construction of the Huffman tree.

[0254] c. Use the RLE state machine to get the next context (see Figure 18 ). If the stream is not empty, go to b.

[0255] The run length decoder reads the run length encoded data byte by byte. By construction, the context of the first byte of data is guaranteed to be RLC_RESIDUAL LSB. The decoder determines the context of the next byte of data using the state machine shown in FIG. 6B. The context tells the decoder how to interpret the current byte of data as described above. Figure 18

[0256] It should be noted that in this particular implementation, the run length state machine is also used by the Huffman encoding and decoding process to know which Huffman code to use for the current symbol or codeword.

[0257] The methods and processes described herein can be embodied as code (e.g., software code) and / or data at both the encoder and the decoder, for example, implemented in a streaming server or client device or a client device decoding from a data store. The encoder and decoder can be implemented in hardware or software, as is well known in the art of data compression. For example, hardware acceleration using a specially programmed graphics processing unit (GPU) or a specially designed field programmable gate array (FPGA) can provide certain efficiencies. For completeness, such code and data can be stored on one or more computer-readable media, which can include any apparatus or medium that can store code and / or data for use by a computer system. When a computer system reads and executes the code and / or data stored on the computer-readable medium, the computer system performs the methods and processes embodied as data structures and code stored within the computer-readable storage medium. In certain embodiments, one or more of the steps of the methods and processes described herein can be performed by a processor (e.g., a processor of a computer system or data storage system).

[0258] In general, any of the functions described in this text or illustrated in the drawings can be implemented using software, firmware (e.g., fixed logic circuitry), programmable or non-programmable hardware, or a combination of these implementations. In general, the term "component" or "function" as used herein means software, firmware, hardware, or a combination of these. For example, in the case of software implementation, the term "component" or "function" can refer to program code that, when executed on one or more processing devices, carries out the specified tasks. The illustrated separation of components and functions into distinct units can reflect any actual or conceptual physical grouping and allocation of such software and / or hardware and tasks.

[0259] In the present application, methods for encoding and decoding a signal, in particular a video signal and / or an image signal, are described.

[0260] ​In particular, a method of encoding a signal is described, the method comprising receiving an input frame and processing the input frame to generate at least a first set of residual data, the residual data enabling a decoder to reconstruct an original frame from a reconstructed reference frame.

[0261] In one embodiment, the method comprises obtaining a reconstructed frame from a decoded frame obtained from a decoding module, wherein the decoding module is configured to generate the decoded frame by decoding a first encoded frame that has been encoded according to a first encoding method. The method further comprises down-sampling the input frame to obtain a down-sampled frame and passing the down-sampled frame to an encoding module configured to encode the down-sampled frame according to the first encoding method so as to generate the first encoded frame. Obtaining a reconstructed frame can further comprise up-sampling the decoded frame to generate the reconstructed frame.

[0262] In another embodiment, the method comprises obtaining a reconstructed frame from a combination of a second set of residual data and a decoded frame obtained from a decoding module, wherein the decoding module is configured to generate the decoded frame by decoding a first encoded frame that has been encoded according to a first encoding method. The method further comprises down-sampling the input frame to obtain a down-sampled frame and passing the down-sampled frame to an encoding module configured to encode the down-sampled frame according to the first encoding method so as to generate the first encoded frame. The method further comprises generating the second set of residual data by taking a difference between the decoded frame and the down-sampled frame. The method further comprises encoding the second set of residual data to generate a first set of encoded residual data. Encoding the second set of residual data can be performed according to a second encoding method. The second encoding method comprises transforming the second set of residual data into a transformed second set of residual data. Transforming the second set of residual data comprises selecting a subset of the second set of residual data and applying a transform to the subset to generate a corresponding subset of the transformed second set of residual data. One of the subsets of the transformed second set of residual data can be obtained by averaging the subset of the second set of residual data. Obtaining a reconstructed frame can further comprise up-sampling the combination of the second set of residual data and the decoded frame to generate the reconstructed frame.

[0263] In one embodiment, generating at least one set of residual data includes taking a difference between the reconstructed reference frame and the input frame. The method further includes encoding the first set of residual data to generate a first set of encoded residual data. Encoding the first set of residual data can be performed according to a third encoding method. The third encoding method includes transforming the first set of residual data into a transformed first set of residual data. Transforming the first set of residual data includes selecting a subset of the first set of residual data and applying a transform to the subset to generate a corresponding subset of the transformed first set of residual data. One of the subsets of the transformed first set of residual data can be obtained by a difference between an average of a subset of the input frame and a corresponding element of the combination of the second set of residual data and the decoded frame.

[0264] In particular, a method of decoding a signal is described, the method including receiving an encoded frame and at least one set of encoded residual data. The first encoded frame can be encoded using a first encoding method.

[0265] The at least one set of residual data can be encoded using a second and / or third encoding method.

[0266] The method further includes passing the first encoded frame to a decoding module, wherein the decoding module is configured to generate a decoded frame by decoding the encoded frame that has been encoded according to the first encoding method.

[0267] The method can further include decoding the at least one set of encoded residual data according to the respective encoding method used to encode it.

[0268] In one embodiment, the first set of encoded residual data is decoded by applying a second decoding method corresponding to the second encoding method to obtain a first set of decoded residual data. The method further includes combining the first set of residual data with the decoded frame to obtain a combined frame. The method further includes upsampling the combined frame to obtain a decoded reference frame.

[0269] The method further includes decoding the second set of encoded residual data by applying a third decoding method corresponding to the third encoding method to obtain a second set of decoded residual data. The method further includes combining the second set of decoded residual data with the decoded reference frame to obtain a reconstructed frame.

[0270] In another embodiment, the method includes upsampling the decoded frame to obtain a decoded reference frame.

[0271] The method further includes decoding the set of encoded residual data by applying a second or third decoding method corresponding to the second or third encoding method to obtain a set of decoded residual data. The method further includes combining the set of decoded residual data with the decoded reference frame to obtain a reconstructed frame.

[0272] The following statements describe preferred or exemplary aspects described and illustrated herein.

[0273] A method of encoding an input video into a plurality of encoded streams such that the encoded streams can be combined to reconstruct the input video, the method comprising:

[0274] receiving a full resolution input video;

[0275] down-sampling the full resolution input video to form a down-sampled video;

[0276] encoding the down-sampled video using a first codec to form an encoded base stream;

[0277] reconstructing a video from the encoded video to generate a reconstructed video;

[0278] comparing the reconstructed video to the input video; and

[0279] forming one or more further encoded streams based on the comparison.

[0280] The input video can be the down-sampled video compared to the reconstructed video.

[0281] According to an example method, comparing the reconstructed video to the input video comprises:

[0282] comparing the reconstructed video to the down-sampled video to form a first set of residuals, and wherein forming the one or more further encoded streams comprises encoding the first set of residuals to form a first level encoded stream.

[0283] The input video can be a full resolution input video compared to the reconstructed video and the reconstructed video can be up-sampled.

[0284] According to an example method, comparing the reconstructed video to the input video comprises:

[0285] up-sampling the reconstructed video to generate an up-sampled reconstructed video; and

[0286] comparing the up-sampled reconstructed video to the full resolution input video to form a second set of residuals, and wherein forming the one or more further encoded streams comprises encoding the second difference to form a second level encoded stream.

[0287] Thus, in examples, the method can generate an encoded base stream, a primary encoded stream, and a secondary encoded stream according to the example methods defined above. Each of the primary encoded stream and the secondary encoded stream can contain enhancement data used by a decoder to enhance the encoded base stream.

[0288] The residual can be the difference between two videos or frames.

[0289] The encoded stream can be accompanied by one or more headers that include parameters indicative of aspects of the encoding process to aid in decoding. For example, the headers can include the codec used, the transform applied, the quantization applied, and / or other decoding parameters.

[0290] The steps described above of ranking and selecting can be applied to the residual data, the step of subtracting the temporal coefficients can be performed, and the quantization can be adapted. Each of these steps can be predetermined and selectively applied or can be applied based on an analysis of the input video, the reduced-sampled video, the reconstructed video, the increased-sampled video, or any combination of the above to improve the overall performance of the encoder. The steps can be selectively applied based on a predetermined set of rules or deterministically applied based on an analysis or feedback of performance.

[0291] An example method, further comprising:

[0292] Transmitting the encoded base stream.

[0293] An example method, further comprising:

[0294] Transmitting the primary encoded stream.

[0295] An example method, further comprising:

[0296] Transmitting the secondary encoded stream.

[0297] According to another aspect of the disclosure, a decoding method is provided.

[0298] A method of decoding a plurality of encoded streams into a reconstructed output video, the method comprising:

[0299] Receiving a first encoded base stream;

[0300] Decoding the first encoded base stream according to a first codec to generate a first output video;

[0301] Receiving one or more additional encoded streams;

[0302] Decoding the one or more additional encoded streams to generate a set of residuals; and

[0303] Combining the set of residuals with the first video to generate a decoded video.

[0304] In an example, the method comprises retrieving a plurality of decoding parameters from the header. The decoding parameters can indicate which procedural steps were included in the encoding process.

[0305] In an example, the method can comprise receiving a primary encoded stream and receiving a secondary encoded stream. In this example, the step of decoding the one or more further encoded streams to generate a set of residuals comprises:

[0306] decoding the primary encoded stream to derive a first set of residuals;

[0307] wherein the step of combining the set of residuals with the first video to generate a decoded video comprises:

[0308] combining the first set of residuals with the first output video to generate a second output video;

[0309] upsampling the second output video to generate an upsampled second output video;

[0310] decoding the secondary encoded stream to derive a second set of residuals; and

[0311] combining the second set of residuals with the second output video to generate a reconstructed output video.

[0312] The method can further comprise displaying or outputting the reconstructed output.

Claims

1. A method for encoding a video signal, the method comprising: Receive input frame (100); processing the input frame to generate residual data (110, 119) that forms part of an enhancement stream and enables a decoder to reconstruct the input frame from a reconstructed reference frame; as well as applying a run-length encoding operation to the residual data, wherein the run-length encoding operation comprises generating a run-length encoded byte stream, the run-length encoded byte stream comprising a set of symbols representing non-zero data values ​​of the residual data and a count of consecutive zero values ​​of the residual data, wherein the run-length encoding operation comprises: encoding non-zero data values ​​of the residual data into at least a first type of symbols; encoding a count of consecutive zero values ​​into a symbol of a second type such that the residual data is encoded as a sequence of symbols of different types, and the method is characterized in that: A flag is inserted in each symbol indicating the type of symbol encoded next in the run-length encoded byte stream.

2. The method of claim 1 , wherein the run-length encoding operation comprises: encoding a data value of the residual data into a symbol of a first type and a symbol of a third type, the symbol of the first type and the symbol of the third type each comprising a portion of the data value such that the portions can be combined at a decoder to reconstruct the data value, the data value encoding further comprising: comparing the magnitude of each data value of the residual data to be encoded with a threshold; and encoding each data value into a symbol of the first type when the size is below the threshold and encoding a portion of each data value into a symbol of the first type and encoding a portion of each data value into a symbol of the third type when the size is above the threshold; and If the size is above the threshold, a flag is set in the symbol of the first type of symbols indicating that a portion of the represented data value is encoded into another symbol of the third type of symbols.

3. The method according to any one of the preceding claims, further comprising: applying a Huffman encoding operation to the set of symbols to generate Huffman-encoded data comprising a set of codes representing the run-length-encoded byte stream; comparing a data size of at least a portion of the run-length encoded byte stream with a data size of at least a portion of the Huffman-encoded data; and The run-length encoded byte stream or the Huffman-encoded data is output in an output bitstream based on the comparison.

4. The method according to claim 3, further comprising: A flag indicating whether the bitstream represents the run-length encoded byte stream or the Huffman encoded data is added to configuration metadata accompanying the output bitstream.

5. The method according to claim 3 or 4, wherein the Huffman encoding operation comprises: Separate frequency tables are generated for symbols of the first type of symbols and symbols of the second type of symbols.

6. The method of claim 3 , wherein the Huffman encoding operation is a canonical Huffman encoding operation such that the Huffman-encoded data includes a code length for each unique symbol in the set of symbols, the code length representing the length of a code used to encode the corresponding symbol, and the method further comprising: Generating a stream header including an indication of the plurality of code lengths such that a decoder can derive the code lengths and corresponding symbols for a canonical Huffman decoding operation, wherein the stream header is one of the following: a stream header of a first type comprising an indication of said symbols associated with respective ones of said plurality of code lengths; or a second type of stream header, wherein for the second type of stream header, the method further comprises ordering the plurality of code lengths in the stream header based on a predetermined order of symbols corresponding to each of the code lengths so that the code lengths can be associated with corresponding symbols at the decoder, wherein the second type of stream header further comprises a flag indicating that the predetermined order of symbols in a possible set of symbols is not present in the set of symbols in the run-length encoded byte stream, and wherein the method further comprises: The number of unique codes in the Huffman-encoded data is compared to a threshold value and the first type of stream header or the second type of stream header is generated based on the comparison.

7. A method for decoding a video signal, the method comprising: retrieving an encoded bitstream; decoding the encoded bitstream to generate residual data for an enhancement stream, and reconstructing an original frame of the video signal from the residual data and the reconstructed reference frame, The step of decoding the encoded bit stream comprises: applying a run-length encoding operation to generate said residual data, The run-length encoding operation includes: identifying a set of symbols representing non-zero data values ​​of the residual data and a set of symbols representing a count of consecutive zero values ​​of the residual data; parsing the set of symbols to derive the non-zero data values ​​and the count of consecutive zero values ​​of the residual data; generating said residual data from said counts of said non-zero data values ​​and consecutive zero values, The run-length encoding operation includes: identifying a sign of a first type of sign representing a non-zero data value of the residual data; identifying a symbol of a second type of symbol representing a count of consecutive zero values ​​such that a sequence of symbols of a different type is decoded to generate the residual data; and parsing the set of symbols according to a corresponding type of each symbol; And the method is characterized in that: A flag is retrieved from each symbol that indicates the expected subsequent type of symbol in the set of symbols.

8. The method of claim 7, wherein the run-length encoding operation comprises: identifying a symbol of a first type of symbol and a symbol of a third type of symbol, the symbol of the first type and the symbol of the third type each representing a portion of a data value; parsing the symbols of the first type of symbols and the symbols of the third type of symbols to derive portions of a data value; as well as combining the derived portions of the data value into a data value, the method further comprising: An overflow flag is retrieved from a symbol of the first type of symbols indicating whether a portion of the data value of the symbol of the first type of symbols is included in a subsequent symbol of a third type.

9. The method according to claim 7 or 8, wherein the step of decoding the encoded bitstream comprises: applying a Huffman encoding operation to the encoded bitstream to generate the set of symbols representing non-zero data values ​​of the residual data and the set of symbols representing counts of consecutive zero values ​​of the residual data, wherein the step of applying a run-length encoding operation to generate the residual data is performed on the set of symbols, and further comprising: retrieving a flag from configuration metadata accompanying the encoded bitstream indicating whether the encoded bitstream comprises a run-length encoded byte stream or Huffman encoded data; and The Huffman encoding operation is selectively applied to an encoded bitstream based on the flag.

10. The method according to claim 9, further comprising: identifying an initial type symbol expected to be derived in the encoded bitstream; applying the Huffman encoding operation to the encoded bitstream based on a set of encoding parameters associated with an expected symbol of the initial type to derive an initial symbol; retrieving from the initial symbol a flag indicating a subsequent type of symbol expected to be derived from the bitstream; as well as The Huffman encoding operation is further applied to the bitstream to derive subsequent symbols based on a set of encoding parameters associated with the subsequent type of symbols expected to be derived from the bitstream.

11. The method of claim 10 , further comprising iteratively retrieving from decoded symbols a flag indicating a subsequent type of symbol expected to be derived from the bitstream and applying the Huffman encoding operation to the bitstream to derive subsequent symbols based on a set of encoding parameters associated with the subsequent type of symbol expected to be derived from the bitstream.

12. The method of claim 9, wherein the Huffman encoding operation is a canonical Huffman encoding operation and the method further comprises: retrieving a stream header including an indication of a plurality of code lengths to be used for said canonical Huffman encoding operation; Associating each code length with a corresponding symbol according to the canonical Huffman encoding operation; as well as A Huffman encoding operation according to the specification identifies a code associated with each symbol based on the code length.

13. The method of claim 12, wherein the stream header includes an indication of symbols associated with respective ones of the plurality of code lengths and associating each code length with a symbol comprises associating each code length with a corresponding symbol in the stream header.

14. The method of claim 12, wherein the step of associating each code length with a corresponding symbol comprises associating each code length with a symbol in a predetermined set of symbols according to an order in which each code length is retrieved.

15. The method of claim 14, further comprising not associating a symbol in the predetermined set of symbols with a corresponding code length, wherein a flag of the stream header indicates that a code length is not present in the stream header for the symbol.

16. An encoding device comprising a processor configured to perform the method according to any one of claims 1 to 6.

17. A decoding device comprising a processor configured to perform the method according to any one of claims 7 to 15.

18. A computer-readable medium comprising instructions which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 15.

19. A coded bitstream comprising a coded version of residual data of an enhancement stream, The residual data can be used together with the reference reconstructed frame to reconstruct the original frame of the video signal. wherein the encoded residual data comprises the result of applying a run-length encoding operation to the residual data, The run-length encoding operation produces a set of symbols representing non-zero data values ​​of the remaining data and a set of symbols representing a count of consecutive zero values ​​of the remaining data, The run-length encoding operation at the decoder includes: identifying a sign of a first type of sign representing a non-zero data value of the residual data; identifying a symbol of a second type of symbol representing a count of consecutive zero values ​​such that a sequence of symbols of different types is decoded to generate the residual data; as well as parsing the set of symbols according to a corresponding type of each symbol; This symbol group can be parsed by a decoder to derive the non-zero data values ​​and counts of consecutive zero values ​​for the remaining data, which can be used to derive the remaining data at the decoder. The coded bit stream is characterized by: A flag in each symbol indicates the type of symbol that is encoded next in the run-length encoded byte stream.

Images