Quantization of residual error in video decoding

Abstract

According to aspects of the present invention, a method for encoding an input video into multiple encoded streams is provided, wherein the encoded streams can be combined to reconstruct the input video. A method of encoding may be provided, comprising: receiving an input video; downsampling the input video to create a downsampled video; instructing to encode the downsampled video using a base encoder to create a base encoded stream; instructing to decode the base encoded stream using a base decoder to generate a reconstructed video; comparing the reconstructed video with the downsampled video to create a first residual set; and encoding the first residual set to create a first level encoded stream, including: applying a transform to the first residual set to create a first set of coefficients; applying a quantization operation to the first set of coefficients to create a first set of quantized coefficients; and applying an encoding operation to the first set of quantized coefficients, wherein applying the quantization operation includes: adapting the quantization based on the first set of coefficients to be quantized, including changing the step width used for different coefficients in the first set of coefficients, wherein a first parameter set derived from the adaptation is signaled to a decoder to achieve dequantization of the first set of quantized coefficients.

Description

Background Art

[0001] Hybrid backward-compatible decoding techniques have been previously proposed in, for example, WO 2014 / 170819 and WO 2018 / 046940, the contents of which are incorporated herein by reference.

[0002] A method is presented which parses a data stream into a first portion of encoded data and a second portion of encoded data; implements a first decoder to decode the first portion of the encoded data into a first rendition of a signal; implements a second decoder to decode the second portion of the encoded data into reconstruction data, the reconstruction data specifying how to modify the first rendition of the signal; and applies the reconstruction data to the first rendition of the signal to produce a second rendition of the signal.

[0003] A newly added method is further proposed, wherein a set of residual elements can be used to reconstruct a reproduction of a first time sample of a signal. A set of spatiotemporally correlated elements associated with the first time sample is generated. The set of spatiotemporally correlated elements indicates a degree of spatial correlation between a plurality of residual elements, and a degree of temporal correlation between first reference data based on the reproduction and second reference data based on the reproduction of a second time sample of the signal. The set of spatiotemporally correlated elements is used to generate output data. As mentioned, the set of residuals is encoded to reduce the overall data size.

[0004] Coding applications often employ quantization operations. By means of this compression process, in which each of one or more ranges of data values ​​is compressed into a single value, the number of different values ​​in a set of video data is allowed to be reduced, thereby making the data easier to compress. In this way, quantization schemes have been used in some videos to change signals into quantities so that specific variables can only take specific discrete values. Typically, a video codec divides visual data in the form of a video frame into discrete blocks that typically have a predetermined size or number of pixels. A transform is then typically applied to the blocks so as to express the visual data in terms of a sum of frequency components. The transformed data may then be premultiplied by a quantization scale code, and then divided element by element by a quantization matrix, wherein the output element obtained by dividing each transformed premultiplied element by the matrix element is then rounded. Processing different transformed elements with a divisor (i.e., different elements of the quantization matrix) is often used to allow those frequency elements that have a greater impact on the visual appearance of the video to the viewer to be effectively allocated more data or resolution than less perceptible components.

[0005] Optimizations are sought to further reduce the overall data size while balancing the goals of not compromising the user's overall impression once the signal has been reconstructed and optimizing processing speed and complexity. Summary of the invention

[0006] According to a first aspect of the present invention, there is provided a method for encoding an input video into a plurality of encoded streams, wherein the encoded streams can be combined to reconstruct the input video, the method comprising: receiving an input video, which is typically a full resolution input video; downsampling the input video to create a downsampled video; instructing the use of a basic encoder to encode the downsampled video to create a basic encoded stream; instructing the use of a basic decoder to decode the basic encoded stream to generate a reconstructed video; comparing the reconstructed video with the downsampled video to create a first residual set; and encoding the first residual set to create a first level encoded stream, comprising: applying a transform to the first residual set to create a first coefficient set; applying a quantization operation to the first coefficient set to create a first quantized coefficient set; and applying an encoding operation to the first quantized coefficient set, wherein applying the quantization operation comprises: adapting the quantization based on the first coefficient set to be quantized, comprising changing the step width used for different coefficients in the first coefficient set, wherein a first parameter set derived from the adaptation is signaled to a decoder to achieve dequantization of the first quantized coefficient set.

[0007] The method may advantageously allow for improved efficiency of the encoding and decoding processes by altering the degree and / or manner of compression applied to coefficients during a quantization process based on the video data to be decoded according to any of a number of factors. Thus, the manner in which a normally lossy quantization procedure is performed during encoding of a video stream may be adapted so that an appropriate balance between encoding or compression efficiency and visually perceptible compression of the input video may be applied depending on the nature and content of the input video, a relationship that may vary greatly across different video frames and streams. This adaptable form of quantization may be used in conjunction with a dequantization process at a receiving decoder, for example by signaling to the decoder the manner in which quantization has been performed or the degree to which quantization has been altered from a default mode, such as via the transmission of a parameter having a value representing or indicating such information.

[0008] With reference to the method according to the first aspect, the step of quantizing may comprise adapting the quantization based on an analysis of coefficients, which coefficients may be understood as outputs of a transformation. In some embodiments, the quantization may be adapted based on an analysis of the data to be transformed as an alternative or in addition. For example, residual data such as the first residual set may be analyzed for this purpose.

[0009] The adaptation of quantization, including changing the step width for different coefficients in the first set of coefficients, typically involves using different quantization parameters for different coefficients in the coding block, or preferably using different parameters for each coefficient in the coding block. For example, each of the A, H, V, and D (average, horizontal, vertical, and diagonal) coefficients explained in more detail below may have different quantization parameters applied to it. In some embodiments, the transform may be a directional decomposition transform, which is also discussed in more detail later in this disclosure. The coding block is typically a small N×N coding block, and the transform of such a coding block may include a small kernel or matrix applied to a flattened coding unit of the residual (e.g., a 2×2 or 4×4 block of the residual). The coding units may be arranged in tiles.

[0010] The method may also generally involve, when the downsampled video has been created, encoding the downsampled video using a first codec to create a base encoded stream; reconstructing the video from the encoded video to generate a reconstructed video; comparing the reconstructed video to the input video; and creating one or more additional encoded streams based on the comparison.

[0011] The first codec is typically a hardware-based codec, and preferably the first codec is AVC, HEVC, AV1, VP8 or VP9. This means that the encoder is typically a dedicated processor using a designed decoding algorithm to perform the described process, as opposed to a software encoder, which is typically an encoding program that can be executed on a computing device. It should be understood that in the present disclosure, the terms first codec and base codec are used interchangeably. The method may further include any one or more of the following: sending a base coded stream, sending a first-level coded stream, and sending a second-level coded stream.

[0012] As described above, the step of encoding the first residual set may generally include: applying a transform to the residual set to create a set of coefficients; applying a quantization operation to the coefficients to create a set of quantized coefficients; and applying an encoding operation to the quantized coefficients.

[0013] In some embodiments, the method may involve creating and encoding a second set of residuals from the corrected reconstructed video, and the second set of residuals may be transformed and quantized for use in the encoded input video. Preferably, in such embodiments, the method comprises: decoding the first set of residuals to generate a decoded first set of residuals; correcting the reconstructed video using the decoded first set of residuals to generate a corrected reconstructed video; upsampling the corrected reconstructed video to generate an upsampled reconstructed video; comparing the upsampled reconstructed video with the input video to create a second set of residuals; and encoding the second set of residuals to create a second level coded stream, including: applying a transform to the second set of residuals to create a second set of coefficients; applying a quantization operation to the second set of coefficients to create a set of second quantized coefficients; and applying an encoding operation to the set of second quantized coefficients, wherein applying the quantization operation comprises: adapting the quantization based on the second set of coefficients to be quantized, including changing the step width used for different coefficients in the second set of coefficients, wherein a second set of parameters derived from the adaptation is signaled to the decoder to achieve dequantization of the quantized coefficients.

[0014] It should be understood that, as similarly described above, the step of encoding the second residual set generally includes: applying a transform to the second residual set to create a set of coefficients; applying a quantization operation to the coefficients to create a set of quantized coefficients; and applying an encoding operation to the quantized coefficients.

[0015] In any of these embodiments, the aforementioned signaling of the parameter set is preferably performed using a quantization matrix. A quantization matrix may be understood as an array of values ​​that are applied to the elements of the output of the transform, respectively, typically by means of a division operation. This signaling using this matrix, whether for either or both of the first and second parameter sets, typically includes transmitting a quantization matrix mode parameter that indicates how the values ​​within the quantization matrix are to be applied to one or more of the first and second coefficient sets.

[0016] The quantization matrix mode parameter can indicate one of the following modes, for example, by taking one of a set of predefined parameter values: a first mode in which the decoder will use a set of values ​​within the quantization matrix for both the first-level coding stream and the second-level coding stream; a second mode in which the decoder will use a set of values ​​within the quantization matrix for the first-level coding stream; a third mode in which the decoder will use a set of values ​​within the quantization matrix for the second-level coding stream; and a fourth mode in which two quantization matrices are signaled for each of the first-level coding stream and the second-level coding stream.

[0017] In some embodiments, the quantization matrix mode parameter may also indicate a mode in which the matrix is ​​not transmitted, which may correspond to two quality levels using a default or otherwise predetermined quantization matrix.

[0018] The method may include combining at least the first layer coded stream and the second layer coded stream into a combined coded stream; and transmitting the combined coded stream to a decoder for use in reconstructing the input video together with the received base coded stream. In some cases, the combined coded stream may include the base coded stream.

[0019] Some embodiments use a "dead zone" in the quantization step. Thus, applying a quantization operation may include quantizing coefficients using a linear quantizer, where the linear quantizer uses a variable sized dead zone.

[0020] In some embodiments, it may also be advantageous for the quantization operation to include the use of a quantization offset, which is typically a non-centered quantization (or, for the inverse of this step, a dequantization) offset. Likewise, as further described below, in a decoding process of dequantization according to the methods of the present disclosure, in some embodiments, each group of transform coefficients passed to the process belongs to a specific plane and layer, and is typically already scaled using a linear quantizer. A linear quantizer may therefore use a non-centered dequantization offset.

[0021] Typically, the method comprises adapting the distribution used in the quantization step. In some embodiments, this adaptation of the distribution is predetermined, and alternatively or additionally, the adaptation may be selectively applied based on analysis of any one or more of: input video, downsampled video, reconstructed video, and upsampled video. In this way, the overall performance of the encoder and decoder may be improved.

[0022] In some embodiments, it may be preferred to selectively apply the quantized adaptation. This may be based on a set of predetermined rules, for example. Additionally or alternatively, it may be applied deterministically based on performance, specifically analysis or feedback of decoding performance.

[0023] There may also be provided an encoder configured to implement the method of any of the above aspects of the implementation.

[0024] According to another aspect, a method for decoding an encoded stream into a reconstructed output video may be provided, the method comprising: receiving a first basic encoded stream; indicating a decoding operation of the first basic encoded stream using a basic decoder to generate a first output video; receiving a first-level encoded stream; decoding the first-level encoded stream to obtain a first residual set; and combining the first residual set with the first output video to generate a reconstructed video, wherein decoding the first-level encoded stream comprises: decoding a set of first quantized coefficients from the first-level encoded stream; obtaining a first parameter set indicating how to dequantize the set of first quantized coefficients; and dequantizing the set of first quantized coefficients using the first parameter set, wherein different quantized coefficients in the set of first quantized coefficients are dequantized using corresponding dequantization parameters.

[0025] The residual is usually obtained by decoding the received stream. Specifically, obtaining the first parameter set may include: obtaining a quantization mode parameter signaled by the first-level coding stream; in response to a first value of the quantization mode parameter, using a default quantization matrix as the first parameter set; in response to other values ​​of the quantization mode parameter, obtaining a quantization matrix signaled by the first-level coding stream and using the quantization matrix as the first parameter set.

[0026] Typically, decoding the first-level coded stream includes: applying an entropy decoding operation to the first-level coded stream before dequantizing the set of first quantized coefficients; and applying an inverse transform operation to generate a first set of residuals after dequantizing the set of first quantized coefficients.

[0027] According to the aforementioned aspects, a method for decoding multiple encoded streams into a reconstructed output video may include: receiving a first basic encoded stream; decoding the first basic encoded stream according to a first codec to generate a first output video; receiving one or more additional encoded streams; decoding the one or more additional encoded streams to generate a residual set; and combining the residual set with the first video to generate a decoded video.

[0028] In some embodiments, the method includes retrieving a plurality of decoding parameters from the header. The decoding parameters may indicate which program steps are included in the encoding process.

[0029] As described in more detail later in this disclosure, two levels of the encoded stream may be advantageously used in the encoding and decoding process. Some embodiments may accordingly further include: receiving a second-level encoded stream; decoding the second-level encoded stream to obtain a second residual set; and combining the second residual set with an upsampled version of the reconstructed video to generate a reconstruction of the original resolution input video, wherein decoding the second-level encoded stream includes: decoding a set of second quantized coefficients from the second-level encoded stream; obtaining a second set of parameters indicating how to dequantize the set of second quantized coefficients; and dequantizing the set of second quantized coefficients using the second set of parameters, wherein different quantized coefficients in the set of second quantized coefficients are dequantized using corresponding dequantization parameters.

[0030] Dequantization may involve receiving one or more quantization matrices from, for example, an encoder, and using the corresponding matrices in a dequantization step to determine appropriate quantization parameters for blocks of an encoded video frame. Thus, in some embodiments, obtaining the first and second parameter sets may include: obtaining a quantization matrix signaled in one or more of the first and second level encoding streams, and dequantizing includes, for a plurality of quantized coefficient elements within a block of quantized coefficients of a video frame (a block corresponds to an N by N grid of pixels, a frame includes a plurality of blocks covering a spatial area associated with the frame): obtaining a quantization parameter from the quantization matrix based on the position of a given quantized coefficient element; and dequantizing the given quantized coefficient element using the quantization parameter. As mentioned above, dequantization typically includes using a linear dequantization operation, and applying a non-centered dequantization offset.

[0031] A decoder for decoding an encoded stream into a reconstructed output video and configured to perform the method of any one of the above aspects or embodiments may also be provided.

[0032] According to another aspect of the present invention, a non-transitory computer-readable storage medium may be provided, which stores instructions that, when executed by a processor, cause the processor to perform any one of the methods of the above aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A high-level diagram showing the encoding process;

[0034] Figure 2 A high-level diagram showing the decoding process;

[0035] Figure 3 A high-level diagram showing the encoding process and specific encoding steps;

[0036] Figure 4 A high-level diagram showing the decoding process and specific decoding steps;

[0037] Figure 5 A high-level diagram showing the encoding process;

[0038] Figure 6 a high-level diagram showing another decoding process; and

[0039] Figure 7 A flowchart showing the concepts described in this article. DETAILED DESCRIPTION

[0040] The present invention relates to methods. More specifically, the present invention relates to methods for encoding and decoding signals. Processing data may include, but is not limited to, obtaining, deriving, outputting, receiving, and reconstructing data.

[0041] The coding techniques discussed herein are flexible, adaptable, efficient, and computationally inexpensive coding formats that combine a video coding format, a base codec (e.g., AVC, HEVC, or any other current or future codec) with coded data of enhancement layers encoded using different techniques. The techniques use a downsampled source signal that is encoded using a base codec to form a base stream. An enhancement stream is formed using an encoded set of residuals that correct or enhance the base stream, such as by increasing the resolution or by increasing the frame rate. There may be multiple enhancement layer data in a hierarchical structure. In a specific arrangement, the base stream may be decoded by a hardware decoder, while the enhancement stream may be suitable for software implementation.

[0042] It is critical that any optimization used in the new coding technique is adapted and complex according to the specific requirements or constraints of the enhancement stream. Such requirements or constraints include: the possible reduction in computing power resulting from the need for software decoding of the enhancement stream; the need for the combination of decoded residual sets and decoded frames; the possible structure of the residual data, i.e., a relatively high proportion of zero values ​​and highly variable data values ​​over a large range; subtle differences in the quantization blocks of coefficients; and the structure of the enhancement stream as a set of discrete residual frames separated into various components. It should be noted that the constraints placed on the enhancement stream mean that simple and fast entropy decoding operations are essential to enable the enhancement stream to effectively correct or enhance individual frames of the base decoded video. It should be noted that in some scenarios, the base streams are also decoded substantially simultaneously before combination, which puts a strain on resources.

[0043] In one case, the methods described herein may be applied to so-called data planes reflecting different color components of a video signal. For example, the methods described herein may be applied to different planes of YUV or RGB data reflecting different color channels. The different color channels may be processed in parallel. Thus, references to residual sets as described herein may include multiple residual sets, with each color component having a different residual set that forms part of a combined enhancement stream. The components of each stream may be aligned in any logical order, for example, each plane at the same level may be grouped together and sent together, or, residual sets of different levels in each plane may be sent together.

[0044] This document preferably meets the requirements of the following ISO / IEC documents: "Call for Proposals for Low Complexity Video Coding Enhancements", ISO / IEC JTC1 / SC29 / WG11 N17944, Macau, China, October 2018; and "Requirements for Low Complexity Video Coding Enhancements", ISO / IEC JTC1 / SC29 / WG11 N18098, Macau, China, October 2018 (which are incorporated herein by reference).

[0045] The general structure of the proposed coding scheme in which the presently described techniques may be applied uses a downsampled source signal encoded with a base codec, adds a first level of correction data to the decoded output of the base codec to generate a corrected picture, and then adds another enhancement level data to an upsampled version of the corrected picture. Thus, the streams are considered to be a base stream and an enhancement stream. This structure creates multiple degrees of freedom, allowing great flexibility and adaptability to many situations, making the coding format suitable for many use cases, including Over-the-Top (OTT) transmission, live streaming, live Ultra High Definition (UHD) broadcasting, etc. Although the decoded output of the base codec is not intended for viewing, it is a fully decoded video at a lower resolution, making the output compatible with existing decoders and also usable as a lower resolution output where deemed appropriate. In some cases, a base codec may be used to create a base stream. A base codec may include independent codecs controlled in a modular or "black box" manner. The methods described herein may be implemented by means of a computer program code that is executed by a processor and makes function calls on an underlying codec implemented in hardware and / or software.

[0046] In general, the term "residual" as used herein refers to the difference between the values ​​of a reference array or reference frame and the actual array or frame of data. The array may be a one-dimensional or two-dimensional array representing a decoding unit. For example, a decoding unit may be a 2×2 or 4×4 set of residual values ​​corresponding to a similarly sized region of an input video frame. It should be noted that this generalized example is agnostic to the encoding operation performed and the nature of the input signal. Reference to "residual data" as used herein refers to data derived from a residual set, such as the residual set itself or the output of a set of data processing operations performed on the residual set. Throughout this specification, in general, a residual set includes a plurality of residuals or residual elements, each of which corresponds to a signal element, i.e., an element of a signal or raw data. The signal may be an image or video. In these examples, the residual set corresponds to an image or frame of a video, wherein each residual is associated with a pixel of the signal, which is a signal element. The examples disclosed herein describe how these residuals can be modified (i.e., processed) to affect the encoding pipeline or the final decoded image while reducing the overall data size. The residuals or sets may be processed per residual element (or residual), or on a group basis, such as per tile or per coding unit, where a tile or coding unit is a contiguous subset of the residual set. In one case, a tile may comprise a group of smaller coding units. It should be noted that the processing may be performed on every video frame or on only a set number of frames in a sequence.

[0047] In general, each enhancement stream or both enhancement streams may be encapsulated into one or more enhancement bitstreams using a set of Network Abstraction Layer Units (NALUs). A NALU is intended to encapsulate an enhancement bitstream in order to apply the enhancement to the correct base reconstructed frame. A NALU may, for example, contain a reference index to a NALU containing the base decoder reconstructed frame bitstream to which the enhancement must be applied. In this way, the enhancement may be synchronized to the base stream, and frames of each bitstream combined to produce decoded output video (i.e., the residual of each frame of the enhancement level is combined with a frame of the base decoded stream). A group of pictures may represent multiple NALUs.

[0048] Returning to the initial process described above, where a base stream is provided together with two levels (or sub-levels) of enhancements within an enhancement stream, an example of a generalized encoding process is depicted in Figure 1. An input full resolution video 100 is processed to generate various coded streams 101, 102, 103. A first coded stream (coded base stream) is produced by feeding a downsampled version of the input video to a base codec (e.g., AVC, HEVC, or any other codec). The coded base stream may be referred to as a base layer or base level. A second coded stream (coded layer 1 stream) is produced by processing a residual obtained by taking the difference between a reconstructed base codec video and a downsampled version of the input video. A third coded stream (coded layer 2 stream) is produced by processing a residual obtained by taking the difference between an upsampled version of a corrected version of the reconstructed base decoded video and the input video. In some cases, Figure 1 The components of may provide a general low-complexity encoder. In some cases, the enhancement stream may be generated by an encoding process that forms part of the low-complexity encoder, and the low-complexity encoder may be configured to control an independent base encoder and decoder (e.g., encapsulated as a base codec). In other cases, the base encoder and decoder may be provided as part of the low-complexity encoder. In one case, Figure 1 The low-complexity encoder may be viewed as a form of wrapper for a base codec, where the functionality of the base codec may be hidden from the entity implementing the low-complexity encoder.

[0049] The downsampling operation illustrated by downsampling component 105 may be applied to an input video to produce a downsampled video to be encoded by a base encoder 113 of a base codec. Downsampling may be performed in both vertical and horizontal directions, or alternatively only in the horizontal direction. The base encoder 113 and base decoder 114 may be implemented by a base codec (e.g., as different functions of a common codec). The base codec and / or one or more of the base encoder 113 and base decoder 114 may include appropriately configured electronic circuitry (e.g., a hardware encoder / decoder) and / or computer program code executed by a processor.

[0050] Each enhancement stream encoding process may not necessarily include an upsampling step. Figure 1 In , the first enhancement stream is conceptually the correction stream, while the second enhancement stream is upsampled to provide the enhancement level.

[0051] Referring to the process of generating the enhancement stream in more detail, to generate the encoded layer 1 stream, the encoded base stream is decoded by the base decoder 114 (i.e., a decoding operation is applied to the encoded base stream to generate a decoded base stream). The decoding may be performed by a decoding function or mode of the base codec. A difference is then created between the decoded base stream and the downsampled input video at the layer 1 comparator 110 (i.e., a subtraction operation is applied to the downsampled input video and the decoded base stream to generate a first residual set). The output of the comparator 110 may be referred to as a first residual set, such as a surface or frame of residual data, where a residual value is determined for each pixel at the resolution of the output of the base encoder 113, the base decoder 114, and the downsampling block 105.

[0052] The difference is then encoded by the first encoder 115 (ie, the layer 1 encoder) to generate the encoded layer 1 stream 102 (ie, the encoding operation is applied to the first residual set to generate the first enhancement stream).

[0053] As described above, the enhancement stream may include a first enhancement level 102 and a second enhancement level 103. The first enhancement level 102 may be considered a corrected stream, e.g., a stream that provides a correction level to a base encoded / decoded video signal at a lower resolution than the input video 100. The second enhancement level 103 may be considered to convert the corrected stream into another enhancement level of the original input video 100, e.g., it applies the enhancement level or correction to a signal reconstructed from the corrected stream.

[0054] exist Figure 1 In the example of , the second enhancement layer 103 is created by encoding another set of residuals. The other set of residuals is generated by the layer 2 comparator 119. The layer 2 comparator 119 determines the difference between the upsampled version of the decoded layer 1 stream, such as the output of the upsampling component 117, and the input video 100. The input to the upsampling component 117 is generated by applying the first decoder (i.e., the layer 1 decoder) to the output of the first encoder 115. This generates a decoded layer 1 residual set. These residuals are then combined with the output of the base decoder 114 at the summing component 120. This effectively applies the layer 1 residuals to the output of the base decoder 114. It allows the losses in the layer 1 encoding and decoding process to be corrected by the layer 2 residuals. The output of the summing component 120 can be regarded as an analog signal representing the output of applying the layer 1 processing to the encoded base stream 101 and the encoded layer 1 stream 102 at the decoder.

[0055] As mentioned, the upsampled stream is compared to the input video, which creates another set of residuals (i.e., a difference operation is applied to the upsampled recreated stream to generate another set of residuals). The other set of residuals is then encoded by the second encoder 121 (i.e., the level 2 encoder) as an encoded level 2 enhancement stream (i.e., the encoding operation is then applied to the other set of residuals to generate another encoded enhancement stream).

[0056] Therefore, if Figure 1 As shown in and described above, the output of the encoding process is a base stream 101 and one or more enhancement streams 102, 103, which preferably include a first enhancement layer and another enhancement layer. The three streams 101, 102 and 103 can be combined with or without additional information such as control headers to generate a combined stream representing the video coding framework structure of the input video 100. It should be noted that Figure 1 The components shown in FIG. 1 may operate on blocks or coding units of data, which correspond, for example, to a 2×2 or 4×4 portion of a frame at a particular resolution level. The components operate without any inter-block dependencies, so they can be applied to multiple blocks or coding units within a frame in parallel. This is different from comparative video coding schemes, where there are dependencies (e.g., spatial dependencies or temporal dependencies) between blocks. The dependencies of comparative video coding schemes limit the level of parallelism and require much higher complexity.

[0057] exist Figure 2 The corresponding generalized decoding process is depicted in the block diagram of . It is said that Figure 2 Can show corresponding to Figure 1 The low complexity decoder receives the three streams 101, 102, 103 generated by the low complexity encoder together with a header 204 containing further decoding information. The encoded elementary stream 101 is decoded by a basic decoder 210 corresponding to the basic codec used in the low complexity encoder. The encoded layer 1 stream 102 is received by a first decoder 211 (i.e., a layer 1 decoder) which decodes the encoded elementary stream 101 by a basic decoder 210 corresponding to the basic codec used in the low complexity encoder. Figure 1 The first residual set encoded by the first encoder 115 is decoded. At the first summing component 212, the output of the base decoder 210 is combined with the decoded residual obtained from the first decoder 211. The combined video, which may be referred to as the layer 1 reconstructed video signal, is upsampled by the upsampling component 213. The encoded layer 2 stream 103 is received by the second decoder 214 (i.e., the layer 2 decoder). The second decoder 214 performs the decoding of the layer 2 stream 103 as received by the second decoder 214. Figure 1 The second residual set encoded by the second encoder 121 is decoded. Although the header 204 is Figure 22 is shown as being used by the second decoder 214, but it may also be used by the first decoder 211 as well as the base decoder 210. The output of the second decoder 214 is a second set of decoded residuals. These may be at a higher resolution than the first set of residuals and the input to the upsampling component 213. At a second summing component 215, the second set of residuals from the second decoder 214 is combined with the output of the upsampling component 213 (i.e., the upsampled reconstructed level 1 signal) to reconstruct the decoded video 250.

[0058] According to the low complexity encoder, Figure 2 The low complexity decoder of the video signal can operate in parallel on different blocks or decoding units of a given frame of the video signal. In addition, the decoding performed by two or more of the basic decoder 210, the first decoder 211 and the second decoder 214 can be performed in parallel. This is possible because there is no inter-block dependency.

[0059] During the decoding process, the decoder may parse the header 204 (which may contain global configuration information, picture or frame configuration information, and data block configuration information) and configure the low-complexity decoder based on those headers. In order to recreate the input video, the low-complexity decoder may decode each of the base stream, the first enhancement stream, and the other or second enhancement stream. The frames of the streams may be synchronized and then combined to derive the decoded video 250. Depending on the configuration of the low-complexity encoder and decoder, the decoded video 250 may be a lossy or lossless reconstruction of the original input video 100. In many cases, the decoded video 250 may be a lossy reconstruction of the original input video 100, where the loss has a reduced or minimal impact on the perception of the decoded video 250.

[0060] exist Figure 1 and 2 In each of the above, the layer 2 and layer 1 encoding operations may include the steps of transform, quantization, and entropy encoding (e.g., in the order described). Similarly, in the decoding stage, the residual may be passed through an entropy decoder, a dequantizer, and an inverse transform module (e.g., in the order described). Any suitable encoding and corresponding decoding operations may be used. However, preferably, the layer 2 and layer 1 encoding steps may be performed in software (e.g., as performed by one or more central or graphics processing units in the encoding device).

[0061] Transforms as described herein may use directional decomposition transforms, such as Hadamard-based transforms. Both may include a small kernel or matrix applied to the flattened coding unit of the residual (i.e., a 2×2 or 4×4 residual block). For more details on the transforms, see, for example, patent applications PCT / EP2013 / 059847 or PCT / GB2017 / 052632, which are incorporated herein by reference. The encoder may select between different transforms to be used, such as between the size of the kernel to be applied.

[0062] The transform may transform the residual information to four surfaces. For example, the transform may produce the following components: average, vertical, horizontal, and diagonal. As mentioned earlier in this disclosure, these components output by the transform may be employed in such embodiments as coefficients to be quantized according to the described method.

[0063] In summary, the methods and apparatus herein are based on an overall approach that is built upon existing encoding and / or decoding algorithms (e.g., MPEG standards, such as AVC / H.264, HEVC / H.265, etc.; and non-standard algorithms, such as VP9, ​​AV1, etc.), which are used as a baseline for enhancement layers for different encoding and / or decoding methods, respectively. The idea behind the overall approach of the example is to encode / decode video frames in a hierarchical manner, in contrast to the block-based approach used in the MPEG family of algorithms. Encoding frames in a hierarchical manner includes generating residuals for the full frame, and then generating residuals for the decimated frame, and so on.

[0064] The video compression residual data of a full-size video frame may be referred to as LoQ-2 (e.g., 1920×1080 for an HD video frame, or higher for a UHD frame), while the video compression residual data of a decimated frame may be referred to as LoQ-x, where x represents the number corresponding to the hierarchical decimation. Figure 1 and 2 In the described example of , the variable x may have values ​​1 and 2 representing the first and second enhancement streams. Thus, there are 2 hierarchical levels for which compressed residuals will be generated. Other naming schemes for the levels may also be applied without any functional changes (e.g., the level 1 and level 2 enhancement streams described herein may alternatively be referred to as level 1 and level 2 streams - meaning counting down from the highest resolution).

[0065] Figure 3A more detailed encoding process is depicted in the block diagram of . The encoding process is split into two halves as shown by the dotted lines. Below the dotted line is the base level of encoder 300, which can be usefully implemented in hardware or software. Above the dotted line is the enhancement level, which can be usefully implemented in software. Encoder 300 can include only the enhancement level process, or a combination of the base level process and the enhancement level process as desired. Encoder 300 can be usefully implemented in software, especially at the enhancement level. This arrangement allows, for example, a traditional hardware encoder that provides the base level to be upgraded using firmware (e.g., software) updates, where the firmware is configured to provide the enhancement level. In newer devices, both the base level and the enhancement level can be provided in hardware and / or a combination of hardware and software.

[0066] The encoder topology at a general level is as follows. Encoder 300 includes an input I for receiving an input signal 30. Input signal 30 may include an input video signal, where the encoder is applied frame by frame. Input I is connected to downsampler 305D and processing block 300-2. Downsampler 305D may correspond to Figure 1 The down sampling component 105, and the processing block 300-2 may correspond to Figure 1 The downsampler 305D is output to the base codec 320 at the base level of the encoder 300. The base codec 320 may implement Figure 1 The downsampler 305D also outputs to the processing block 300-1. The processing block 300-1 may correspond to Figure 1 The processing block 300-1 passes the output to the upsampler 305U, which in turn outputs to the processing block 300-2. The upsampler 305U may correspond to Figure 1 Each of the processing blocks 300-2 and 300-1 includes one or more of the following modules: a transform block 310, a quantization block 320, an entropy coding block 330, and a residual processing block 350. The residual block 350 may occur before the transform block 310 and / or control the residual processing in the processing block 300. The order of processing may be as stated in the figure.

[0067] An input signal 30, such as full (or highest) resolution video in this example, is processed by an encoder 300 to generate various encoded streams. A base encoded stream is produced by feeding a downsampled version of the input video 30 to a base codec 320 (e.g., AVC, HEVC, or any other codec) at a base level using a downsampler 305D. The base encoded stream may include the output of a base encoder of the base codec 320. A first encoded stream (an encoded layer 1 stream) is created by reconstructing the encoded base stream to create a base reconstruction, and then taking the difference between the base reconstruction and the downsampled version of the input video 30. Reconstructing the encoded base stream may include receiving a decoded base stream from the base codec (i.e., the input to the processing block 300-1 includes the following: Figure 1 300-1). The difference signal is then processed at block 300-1 to create an encoded level 1 stream. Block 300-1 includes a transform block 310-1, a quantization block 320-1, and an entropy coding block 330-1. A second encoded stream (encoded level 2 stream) is created by upsampling the corrected version of the base reconstruction using an upsampler 305U, and taking the difference between the corrected version of the base reconstruction and the input signal 30. This difference signal is then processed at block 300-2 to create an encoded level 2 stream. Block 300-2 includes a transform block 310-2, a quantization block 320-2, an entropy coding block 330-2, and a residual processing block 350-2. According to processing block 300=1, the blocks can be performed in the order shown in the figure (e.g., residual processing followed by transform followed by quantization followed by entropy coding).

[0068] A quantization scheme may be used to create the residual signal as a quantity such that a particular variable may take only particular discrete values. In one case, quantization includes dividing by a predetermined step width. This may be applied at two levels (1 and 2). For example, quantization at block 320 may include dividing the transformed residual value by the step width (e.g., where an integer quotient is used to generate the quantized value and the remainder is ignored). The step width may be predetermined, such as selected based on a desired quantization level. In one case, dividing by the step width may be converted to multiplying by an inverse step width, which may be implemented more efficiently in hardware. In this case, dequantization at block 320, for example, may include multiplying by the step width. Entropy encoding as described herein may include run length encoding (RLE) followed by processing the encoded output using a Huffman encoder. In some cases, when entropy encoding is desired, only one of these schemes may be used.

[0069] The encoded elementary stream may be referred to as a base layer stream.

[0070] As described earlier, the residual is calculated by comparing the image signal in its original form with the image signal in its reconstructed form. For example, in one case, the residual of the L-2 enhancement stream is determined by subtracting the upsampled output from the image signal in its original form (e.g., the input video as indicated in the figure). The input to the upsampling may be referred to as a reconstruction of the signal after analog decoding. In another case, the residual of the L-1 enhancement stream is determined by subtracting the image stream output by the base decoder from the original image signal in its downsampled form (e.g., the downsampled output).

[0071] exist Figure 4 The block diagram depicts the corresponding Figure 3 A decoder 400 is shown that performs a decoding process of an encoder of the present invention. The decoding process is split into two halves as shown by the dotted lines. Below the dotted line is the base level of the decoder 400, which can be usefully implemented in hardware. Above the dotted line is the enhancement level, which can be usefully implemented in software. The decoder 400 can include only the enhancement level process, or a combination of the base level process and the enhancement level process as required. The decoder 400 can be usefully implemented in software, especially at the enhancement level, and can be suitably superior to traditional decoding techniques, especially traditional hardware techniques. Traditional techniques refer to early techniques that have been previously developed and sold, are already in the market, are inconvenient to replace and / or are expensive to replace, and can still be used for the purpose of decoding signals. In other cases, the base level may include any existing and / or future video encoding tools or techniques.

[0072] The decoder topology at a general level is as follows. Decoder 400 includes an input (not shown) for receiving one or more input signals including an encoded base stream, an encoded layer 1 stream, and an encoded layer 2 stream together with an optional header containing additional decoding information. Decoder 400 includes a base decoder 420 at the base layer, and processing blocks 400-1 and 400-2 at the enhancement layer. An upsampler 405U is also provided between processing blocks 400-1 and 400-2 to provide an upsampled version of the signal output by processing block 400-1 to processing block 400-2. Base decoder 420 may correspond to Figure 2 The basic decoder 210 of FIG. 4 may correspond to the processing block 400-1. Figure 2 The first decoder 211, the processing block 400-2 may correspond to Figure 2 The second decoder 214, and the upsampler 405U may correspond to Figure 2 The upper sampler 213.

[0073] Decoder 400 receives the one or more input signals and directs the three streams generated by encoder 300. The encoded base stream is directed to and decoded by base decoder 420, which corresponds to base codec 420 used in encoder 300 and is used to reverse the encoding process at the base level. The encoded layer 1 stream is processed by block 400-1 of decoder 400 to recreate the first residual set created by encoder 300. Block 400-1 corresponds to processing block 300-1 in encoder 300 and is used to reverse or substantially reverse the processing of block 300-1 at the base level. The output of base decoder 420 is combined with the first residual set obtained from the encoded layer 1 stream. The combined signal is upsampled by upsampler 405U. The encoded layer 2 stream is processed by block 400-2 to recreate the additional residual created by encoder 300. Block 400-2 corresponds to processing block 300-2 of encoder 300 and is used to inverse or substantially inverse the processing of block 300-2 at the base level. The upsampled signal from upsampler 405U is combined with the additional residual obtained from the encoded level 2 stream to create a level 2 reconstruction of the input signal 30. The output of processing block 400-2 can be viewed as similar to Figure 2 The decoded video 250 of the decoded video.

[0074] As described above, the enhancement stream may include two streams, namely, an encoded layer 1 stream (first enhancement layer) and an encoded layer 2 stream (second enhancement layer). The encoded layer 1 stream provides a set of correction data that can be combined with a decoded version of the base stream to generate a corrected picture.

[0075] Figure 5 Show in more detail Figure 1 320E. The encoded base stream is created directly by the base encoder 320E and may be quantized and entropy encoded as needed. In some cases, these latter processes may be performed as part of the encoding performed by the base encoder 320E. In order to generate the encoded layer 1 stream, the encoded base stream is decoded at the encoder 300 (i.e., the decoding operation is applied to the encoded base stream at the base decoding block 320D). The base decoding block 320D is shown as part of the base layer of the encoder 300 and is shown as being separated from the corresponding base encoding block 320E. For example, the base decoder 320D can be a decoding component that supplements the encoding component in the form of the base encoder 320E with a base codec. In other examples, the base decoding block 320D may actually be part of the enhancement layer, and more specifically may be part of the processing block 300-1.

[0076] Return to Figure 5, creating a difference between the decoded base stream output from the base decoding block 320D and the downsampled input video (i.e., applying the subtraction operation 310-S to the downsampled input video and the decoded base stream to generate a first set of residuals). Here, the term residual is used in the same manner as is known in the art; that is, the residual represents the error or difference between a reference signal or frame and a desired signal or frame. Here, the reference signal or frame is the decoded base stream, and the desired signal or frame is the downsampled input video. Therefore, the residuals used in the first enhancement level can be considered as correction signals because they are able to 'correct' future decoded base streams to a closer approximation of the downsampled input video used in the base encoding operation. This is useful because this can correct for quirks or other characteristics of the base codec. These characteristics include, among other things, the motion compensation algorithm applied by the base codec, the quantization and entropy coding applied by the base codec, and the block adjustments applied by the base codec.

[0077] exist Figure 5 More details in Figure 3 Specifically, the first residual set is transformed, quantized, and entropy encoded to produce an encoded level 1 stream. Figure 5 , a transform operation 310-1 is applied to a first set of residuals; a quantization operation 320-1 is applied to the transformed set of residuals to generate a set of quantized residuals; and an entropy encoding operation 330-1 is applied to the quantized set of residuals to generate an encoded layer 1 stream at a first enhancement level. However, it should be noted that in other examples, only the quantization step 320-1, or only the transform step 310-1 may be performed. Entropy encoding may not be used, or may optionally be used in addition to one or both of the transform step 110-1 and the quantization step 320-1. The entropy encoding operation may be any suitable type of entropy encoding, such as a Huffman encoding operation or a run length encoding (RLE) operation, or a combination of both Huffman encoding operations and RLE operations.

[0078] The selection of entropy decoding schemes such as these entropy decoding schemes can be combined with the described quantization to have a favorable impact on decoding performance. This can be understood in view of the tendency to apply a higher degree of quantization to residual data as described in the present disclosure to produce a high proportion of zero values. Run length coding as mentioned above is particularly suitable for encoding data with this distribution, and therefore these methods can synergistically improve the efficiency of the overall process. Similarly, for embodiments in which quantization is applied with a larger step width value and the distribution of quantized data is such that there are a relatively large number of multiple integer values, the efficiency of the encoding process will generally benefit from the use of prefix / Huffman coding that is particularly suitable for such distributions. This is especially true in the case of a distribution in which higher integer values ​​have a lower frequency of residuals. These forms of distribution can be effectively encoded with Huffman coding, which works by allocating fewer bits to high-frequency symbols. In this way, quantization and entropy coding operations are complementary.

[0079] As described above, the enhancement streams may include an encoded layer 1 stream (first enhancement layer) and an encoded layer 2 stream (second enhancement layer). The first enhancement layer may be considered to achieve corrected video at the base layer, that is, for example, correcting encoder and / or decoder artifacts. The second enhancement layer may be considered to be another enhancement layer that may be used to convert the corrected video to the original input video or a close approximation thereof (e.g., adding detail or sharpness). For example, the second enhancement layer may add fine detail lost during downsampling and / or help correct errors introduced by one or more of the transform operation 310-1 and the quantization operation 320-1.

[0080] See also Figure 3 and Figure 5 To generate the encoded layer 2 stream, another enhancement level information is created at block 300-2 by generating and encoding another set of residuals. The other set of residuals is the difference between an upsampled version (via upsampler 305U) of the corrected version of the decoded base stream (reference signal or frame) and the input signal 30 (desired signal or frame).

[0081] In order to achieve reconstruction of a corrected version of the decoded base stream as would be generated at decoder 400, at least some of the processing steps of block 300-1 are reversed to emulate the process of decoder 200, and to account for at least some of the losses and quirks of the transform and quantization processes. To this end, block 300-1 includes an inverse quantization block 320-1i and an inverse transform block 310-1i. The quantized first residual set is inverse quantized at the inverse quantization block 320-1i in encoder 100 and inverse transformed at the inverse transform block 310-1i to regenerate a decoder-side version of the first residual set.

[0082] The decoded base stream from decoder 320D is combined with this improved decoder-side version of the first residual set (i.e., a summation operation 310-C is performed on the decoded base stream and the decoder-side version of the first residual set). The summation operation 310-C generates a reconstruction of a downsampled version of the input video as would likely be generated at the decoder - i.e., the reconstructed base codec video). Figure 3 and Figure 5 As shown in , the reconstructed base codec video is then upsampled by upsampler 305U.

[0083] The upsampled signal (i.e., reference signal or frame) is then compared to the input signal 30 (i.e., desired signal or frame) to create a second set of residuals (i.e., difference operation 300-S is applied to the upsampled recreated stream to generate another set of residuals). The second set of residuals is then processed at block 300-2 to become an encoded level 2 stream (i.e., encoding operations are then applied to another or second set of residuals to generate another or second encoded enhancement stream).

[0084] Specifically, the second residual set is transformed (i.e., the transform operation 310-2 is performed on the other residual set to generate another transformed residual set). The transformed residual is then quantized and entropy encoded in the manner described above with respect to the first residual set (i.e., the quantization operation 320-2 is applied to the transformed residual set to generate another quantized residual set; and the entropy encoding operation 320-2 is applied to the quantized other residual set to generate an encoded layer 2 stream containing another enhancement layer information). However, only the quantization step 20-1, or only the transformation and quantization steps may be performed. Entropy encoding may optionally be used as a supplement. Preferably, the entropy encoding operation may be a Huffman encoding operation or a run length encoding (RLE) operation or both.

[0085] Therefore, if Figure 3 and 5 As shown in and described above, the output of the encoding process is a base stream at the base layer, and one or more enhancement streams at the enhancement layer, preferably including a first enhancement layer and another enhancement layer. As discussed with reference to the previous example, Figure 5 The operations of can be applied in parallel to the coding units or blocks of color components of a frame because there are no inter-block dependencies. The encoding of each color component within a set of color components can also be performed in parallel (e.g., such that the number of copies is based on (number of frames)*(number of color components)*(number of coding units per frame)). Figure 5 It should also be noted that different color components may have different numbers of coding units per frame, for example, the luma (e.g., Y) component may be processed at a higher resolution than a set of chroma (e.g., U or V) components when human vision can detect greater changes in illumination than changes in color.

[0086] An encoded base stream and one or more enhancement streams are received at the decoder 400 . Figure 6 Show in more detail Figure 4 The decoder.

[0087] The encoded base stream is decoded at the base decoder 420 so as to produce a base reconstruction of the input signal 30 received at the encoder 300. This base reconstruction can be used in practice to provide a viewable reproduction of the signal 30 at a lower quality level. However, the main purpose of this base reconstruction signal is to provide a basis for a higher quality reproduction of the input signal 30. To this end, the decoded base stream is provided to the processing block 400-1. The processing block 400-1 also receives the encoded level 1 stream and reverses any encoding, quantization, and transformation that has been applied by the encoder 300. Block 400-1 includes an entropy decoding process 430-1, an inverse quantization process 420-1, and an inverse transformation process 410-1. Optionally, only one or more of these steps may be performed depending on the operations implemented at the corresponding block 300-1 at the encoder. By performing these corresponding steps, the decoded level 1 stream including the first residual set is made available at the decoder 400. The first residual set is combined with the decoded base stream from the base decoder 420 (i.e., a summation operation 410-C is performed on the decoded base stream and the decoded first residual set to generate a reconstruction of a downsampled version of the input video - i.e., the reconstructed base codec video). Figure 4 and Figure 6 As shown in , the reconstructed base codec video is then upsampled by upsampler 405U.

[0088] Additionally, and optionally in parallel, Figure 2 The encoded layer 2 stream is processed at block 400-2 of the encoder 300 to produce another set of decoded residuals. Similar to processing block 300-2, processing block 400-2 includes an entropy decoding process 430-2, an inverse quantization process 420-2, and an inverse transform process 410-2. Of course, these operations will correspond to the operations performed at block 300-2 in encoder 300, and one or more of these steps may be omitted as desired. Block 400-2 produces a decoded layer 2 stream including another set of residuals, and these residuals are summed at operation 400-C with the output from upsampler 405U to create a layer 2 reconstruction of the input signal 30. The layer 2 reconstruction can be viewed as, for example Figure 2 In some examples, it is also possible to obtain and view the reconstructed video passed to the upsampler 405U - this will be of the first enhancement level but may be at a lower resolution than the level 2 reconstruction.

[0089] Therefore, as shown and described above, the output of the decoding process is an (optional) base reconstruction, as well as the reconstruction of the original signal at a higher level. This example is particularly well suited for creating encoded and decoded videos at different frame resolutions. For example, the input signal 30 may be an HD video signal including frames at a resolution of 1920×1080. In some cases, both the base reconstruction and the layer 2 reconstruction may be used by the display device. For example, in the case of network traffic, the layer 2 stream may be interrupted more than the layer 1 stream and the base stream (because it may contain up to 4× amount of data, where downsampling reduces the dimension in each direction by 2). In this case, when traffic occurs, the display device may resume displaying the base reconstruction, while the layer 2 stream is interrupted (e.g., when the layer 2 reconstruction is not available), and then resume displaying the layer 2 reconstruction when network conditions improve. A similar method may be applied when the decoding device is subject to resource constraints, such as a set-top box performing a system update may have an operating base decoder 220 to output the base reconstruction, but may not have the processing capacity to calculate the layer 2 reconstruction.

[0090] The encoding arrangement also enables the video distributor to distribute video to a heterogeneous set of devices; those devices with only the base decoder 220 view the base reconstruction, while those devices with the enhancement layer can view the higher quality layer 2 reconstruction. In the comparative case, two complete video streams at separate resolutions are required to serve the two sets of devices. Since the layer 2 and layer 1 enhancement streams encode residual data, the layer 2 and layer 1 enhancement streams can be encoded more efficiently, e.g., the distribution of residual data is typically mostly around 0 quality (i.e., no difference) and typically takes a small range of values ​​about 0. This is especially true after quantization. In contrast, the complete video streams at different resolutions will have different distributions of non-zero means or medians, which require higher bit rates for transmission to the decoder.

[0091] In some examples, the residual can be considered as an error or difference at a specific quality level or resolution. In the described example, there are two quality levels or resolutions and therefore two residual sets (L-1 and L-2). Each residual set described herein models a different form of error or difference. For example, the L-1 residual typically corrects for the characteristics of the base encoder, such as correcting for artifacts introduced by the base encoder as part of the encoding process. In contrast, for example, the L-2 residual typically corrects for compound effects introduced by transitions in quality levels and differences introduced by the L-1 correction (e.g., artifacts generated by the L-1 encoding pipeline at a wider spatial scale such as an area of ​​4 or 16 pixels). This means that it is not obvious that operations performed on one residual set will necessarily provide the same effect to another residual set, for example, each residual set may have a different statistical pattern and correlation set.

[0092] In the examples described herein, the residual is encoded by an encoding pipeline. This may include transform, quantization, and entropy encoding operations. It may also include residual scaling, weighting, and filtering. These pipelines are shown in Figure 1 3A and 3B. The residuals are then transmitted to the decoder, for example as L-1 and L-2 enhancement streams, which may be combined with the base stream as a hybrid stream (or transmitted separately). In one case, a bit rate is set for a hybrid data stream comprising the base stream and the two enhancement streams, and then different adaptive bit rates are applied to the individual streams based on the data being processed to meet the set bit rate (e.g., high quality video perceived with low levels of artifacts can be constructed by adaptively assigning bit rates to different individual streams (even at a frame-by-frame level) so that constrained data can be used by the most perceptually impactful individual streams, which may change as the image data changes).

[0093] The residual set as described herein can be considered as sparse data, e.g., in many cases there is no difference for a given pixel or region, and the resulting residual value is zero. When looking at the distribution of the residuals, a lot of probability mass is assigned to small residual values ​​located close to zero, e.g., for certain video values ​​of -2, -1, 0, 1, 2, etc., which occur most frequently. In some cases, the distribution of residual values ​​is symmetric or approximately symmetric about 0. In some test video cases, it was found that the distribution of residual values ​​was shaped similar to a logarithmic or exponential distribution about 0 (e.g., symmetrically or approximately symmetrically). The exact distribution of residual values ​​may depend on the content of the input video stream.

[0094] The residual can itself be processed as a two-dimensional image, such as a delta image of differences. In this way, it can be seen that the sparsity of the data is related to features such as "dots", small "lines", "edges", "corners", etc. that are visible in the residual image. It has been found that these features are usually not completely correlated (e.g., spatially and / or temporally). The features have characteristics that are different from the characteristics of the image data from which they are derived (e.g., the pixel characteristics of the original video signal).

[0095] Because the characteristics of the residual are different from the characteristics of the image data from which the residual is derived, it is generally not possible to apply standard encoding methods, such as those found in the traditional Moving Picture Experts Group (MPEG) encoding and decoding standards. For example, many contrasting schemes use large transforms (e.g., transforms of larger pixel areas in normal video frames). Due to the characteristics of the residual, such as described above, it would be extremely inefficient to use these contrasting large transforms on the residual image. For example, it would be very difficult to encode a small point in the residual image using a large block designed for an area of ​​the normal image.

[0096] Certain examples described herein address these issues by using smaller and simpler transform kernels instead (e.g., 2×2 or 4×4 kernels—directional decomposition and directional decomposition squared as presented herein). The transforms described herein may be applied using Hadamard matrices (e.g., 4×4 matrices for flattening 2×2 coding blocks, or 16×16 matrices for flattening 4×4 coding blocks). This moves in a different direction than the contrasting video encoding methods. Applying these new methods to residual blocks generates compression efficiencies. For example, certain transforms generate uncorrelated coefficients (e.g., in space) that can be efficiently compressed. While correlations between coefficients may be exploited, for example, for lines in a residual image, these correlations may result in coding complexity, making them difficult to implement on traditional and low-resource devices, and these correlations often generate other complex artifacts that need to be corrected. Preprocessing the residual by setting certain residual values ​​to 0 (i.e., not forwarding these residual values ​​for processing) may provide a controllable and flexible way to manage bit rate and stream bandwidth, as well as resource usage.

[0097] For the sake of completeness, Figure 7 The broad principles of the concepts described herein are illustrated in flow chart form. Method 1000 includes: receiving an input video (step 1001); downsampling the input video to create a downsampled video (step 1002); instructing the downsampled video to be encoded using a basic encoder to create a basic encoded stream (step 1003); instructing the basic encoded stream to be decoded using a basic decoder to generate a reconstructed video (step 1004); comparing the reconstructed video with the downsampled video to create a first residual set (step 1005); and encoding the first residual set to create a first-level encoded stream, including: applying a transform to the first residual set to create a first coefficient set (step 1006); applying a quantization operation to the first coefficient set to create a first set of quantized coefficients (step 1007); and applying an encoding operation to the first set of quantized coefficients (step 1008), wherein applying the quantization operation includes: adapting the quantization based on the first set of coefficients to be quantized, including changing the step width used for different coefficients in the first set of coefficients, wherein a first parameter set derived from the adaptation is signaled to a decoder to achieve dequantization of the first set of quantized coefficients.

[0098] By means of a specific example, quantization of residuals and / or coefficients (transformed residuals) can be performed based on block groups with a defined step width. In this example, the number of block groups is limited by a step width value of 5. The step width can be understood as a quantization step length. The size of the step width can be selected, for example, based on a parameter value. In some cases, the size of the step width can be set dynamically, for example, based on the rate control example described above. Returning to the specific example, the aforementioned step width will produce block groups corresponding to residual values ​​in the range of 0-4, 5-9, 10-14, 15-19 (i.e., 0 to 4 includes both 0 and 4). The block group width can be configured to include or exclude endpoints as needed. In this theoretical quantization example, quantization is performed by replacing all values ​​falling into the block group with integer values ​​(e.g., residual values ​​between 0 and 4 (including the endpoints) have a quantized value of 1).

[0099] Quantization may be performed by dividing by a step width (e.g., 5 in the specific example given above), flooring the result (i.e., for positive values, the nearest integer less than a certain decimal), and then adding one (e.g., 3 / 5=0.6, floor(0.6)=0, 0+1=1; or 16 / 5=3.2, floor(3.2)=3, 3+1=4). Negative values ​​may be handled in a similar manner, such as by acting on the absolute value and then converting to a negative value after the calculation (e.g., abs(-9)=9, 9 / 5=1.8, floor(1.8)=1, 1+1=2, 2*-1=-2). In the specific case of linear quantization, all block groups may have a common step width. It should be noted that various different implementations based on this approach may be made, for example, the first block group may have a quantized value of 0 instead of 1, or may include values ​​1 to 5 (inclusive). The examples described above are provided merely for the purpose of illustrating these principles with a given step width.

[0100] In another specific example, a so-called "dead zone" (DZ) may be implemented. This may be generally understood as a zone near the zero output value of the quantizer, i.e., a frequency band containing zero signals and which may be the same or different in size from the step width. Therefore, for this frequency band of input close to zero, the signal may be effectively attenuated so that low-level signals, which may generally correspond to noise in the visual data, are not allocated data unnecessarily. In this example, residuals or coefficients having values ​​within a predefined range are set to 0. The predefined range is a range near the value 0. Values ​​less than 6 and greater than -6 are set to 0. The dead zone may be set to a fixed range (e.g., -6 to 6) or may be set based on the step width. In one case, the dead zone may be set to a predefined plurality of step widths, such as a linear function of the step width value. In the example of FIG. 8B , the dead zone is set to 2.4*step width. Therefore, in the case of a step width of 5, the dead zone extends from -6 to +6. In other cases, the dead zone may be set to a nonlinear function of the step width value.

[0101] Having a dead zone can help reduce the amount of data to be transmitted over the network, for example, helping to reduce the bit rate. When using a dead zone, residual or coefficient values ​​that fall within the dead zone are effectively ignored. This method can also help remove low-level residual noise. Having an adaptive rather than constant dead zone means that small residual or coefficient values ​​are not excessively filtered when the step width is reduced (for example, if more bandwidth is available), and the bit rate is appropriately reduced if the step width is increased. The dead zone only needs to be made at the encoder, and the decoder simply receives a quantized value of 0 for any residual or coefficient that falls within the dead zone.

[0102] Quantization offsets may be used in certain situations. Quantization offsets may be used to shift the position of a quantization block group. In an example, many coefficient values ​​may be close to zero, for example, with the count of higher values ​​decreasing as the distance from 0 increases.

[0103] Quantization can be modeled using a histogram. For ease of explanation, in a specific example, the count values ​​(both positive and negative) of the first to third block groups after the dead zone can represent a certain distribution. For example, the counts of quantized values ​​of 1, 2, 3 and -1, -2, -3 can represent this situation. Due to quantization, the distribution modeled by the histogram can be different from the actual distribution. For example, an error e that represents how the bar differs from the line can be obtained.

[0104] In order to change the nature of the error e, a quantization offset qO may be applied. The quantization offset may be understood as a parameter whose value defines whether and to what extent a quantization interval or block group will be shifted from a predetermined or default position or value set. For positive values, a positive quantization offset is used to shift each block group to the right, and a negative quantization offset is used to shift each block group to the left. The quantization offset may be applied in combination with a dead zone. In one case, a dead zone may be applied based on a first threshold set, for example, all values ​​less than (n*step width) / 2 and greater than (n*step width*=1) / 2 are set to 0.

[0105] In some examples, a quantization offset may be signaled to a decoder for use in dequantization.

[0106] In one case, at the encoder, a quantization offset may be subtracted from the residual or coefficient value prior to quantization based on the step width. Thus, in the decoder, the signaled offset may be added to the received quantization value for dequantization based on the step width. In some cases, the offset may be adjusted based on the sign of the residual or coefficient to allow symmetric operation about a value of 0. In one case, the use of the offset may be disabled by setting the quantization or dequantization offset value to 0. In one case, the applied quantization offset may be adjusted based on a defined dead zone width. In one case, the dead zone width may be calculated at the decoder, for example, based on the step width and quantization parameter received from the encoder.

[0107] In one case, the step width used for quantization may vary for different coefficients within a 2×2 or 4×4 coefficient block. For example, smaller step widths may be assigned to coefficients that are experimentally determined to affect the perception of the decoded signal to a greater extent, such as in a 2×2 or 4×4 directional decomposition (DD-squared or "DDS") as described above, the A, H, V, and D coefficients may be assigned smaller step widths, with later coefficients being assigned larger step widths. In this case, a base_stepwidth parameter that sets a default step width may be defined, and then a modifier may be applied to this parameter to calculate a modified_stepwidth for use in quantization (and dequantization), such as modified_stepwidth = base_stepwidth * modifier (where "modifier" may be set based on a specific coefficient within a block or unit, and may be derived from signaling such as the variable "qm" described below).

[0108] In some cases, the modifier may additionally or alternatively depend on the level of enhancement. For example, for a level 1 enhancement stream, the step width may be smaller because it may affect more reconstructed pixels at a higher quality level.

[0109] In some cases, modifiers may be defined based on both coefficients within a block and levels of enhancement. In one case, a quantization matrix may be defined with a set of modifiers for different coefficients and different levels of enhancement. This quantization matrix may be preset (e.g., at an encoder and / or decoder), signaled between an encoder and decoder, and / or dynamically constructed at an encoder and / or decoder. For example, in the latter case, the quantization matrix may be constructed at the encoder and / or decoder based on other stored and / or signaled parameters, such as parameters received via a configuration interface.

[0110] In one case, different quantization modes may be defined, or different schemes of quantization matrices to be applied to a given set of coefficients may be defined. In one mode, a common quantization matrix may be used for both enhancement levels; in another mode, separate matrices may be used for different levels; in yet another mode, a quantization matrix may be used for only one enhancement level, e.g., only level 2. The quantization matrix may be indexed by the position of the coefficients within the block (e.g., 0 or 1 in the x direction and 0 or 1 in the y direction for a 2x2 block, or 0 to 3 for a 4x4 block).

[0111] In one case, a base quantization matrix may be defined by a set of values. This base quantization matrix may be modified by a scaling factor that is a function of the step width of one or more of the enhancement layers. In one case, the scaling factor may be a clamped function of the step width variable. At the decoder, the step width variable may be received from the encoder for one or more of the layer 2 stream and the layer 1 stream.

[0112] With the help of some other examples of processes involving the principles described above, advantageous modes of configuring and adapting quantization in order to further improve encoding and decoding procedures may be understood by further examining the following. Data block unit enhanced payload semantics may be applied, which involves several parameters by which the properties of the quantization and dequantization steps may be signaled and configured.

[0113] In one example, the parameter dequant_offset_signalled specifies whether the value of the offset parameter will be applied when signaling dequantization. In this way, it can be signaled whether the offset is sent. In this example, if the offset is sent, it is used. If it is not sent, a default offset can be used or no offset can be used.

[0114] In an example of a method of encoding an input video into multiple encoded streams, where the encoded streams can be combined to reconstruct the input video, the method can involve receiving the input video, and downsampling the input video to create a downsampled video.

[0115] The method generally further comprises instructing to encode the downsampled video using a base encoder to create a base coded stream, the base coded stream; instructing to decode the base coded stream using a base decoder to generate a reconstructed video; comparing the reconstructed video with the downsampled video to create a first residual set; and encoding the first residual set to create a first level coded stream. This preferably comprises: applying a transform to the residual set to create a set of coefficients; applying a quantization operation to the set of coefficients to create a set of quantized coefficients; and applying an encoding operation to the quantized coefficients, wherein the quantization operation is performed using a quantization matrix, the quantization matrix being derived according to the obtained value of the quantization matrix mode parameter.

[0116] As described earlier in this disclosure, a quantization matrix mode parameter may advantageously be used to specify a quantization matrix to be used in the encoding process. In some examples, when the quantization matrix mode parameter value is equal to a predetermined value, such as when it is equal to zero 0, the method may involve using a default quantization matrix for each of the two quality levels. These levels are typically or typically correspond to level 1 and level 2 enhancement streams. When the quantization matrix mode parameter value is equal to 1, a first quantization matrix may be used for each of the two quality levels, and the first quantization matrix may be signaled, for example, from an encoder to a decoder or a device to which the encoded stream is to be transmitted. When the quantization matrix mode parameter value is equal to 2, a second quantization matrix may be used for quality level 2, and the second quantization matrix may be signaled. In this case, no quantization matrix is ​​used for quality level 1, or a default value may be used for this level. When the quantization matrix mode parameter value is equal to 3, a third quantization matrix is ​​preferably used for quality level 1 or the first level encoded stream, and the third quantization matrix is ​​signaled. In this case, no quantization matrix may be used for quality level 2, or a default value may be used for this level. When the quantization matrix mode parameter value is equal to 4, a fourth quantization matrix may be used for the first-level coded stream, and a fifth quantization matrix may be used for the second-level coded stream (for example, two matrices may be used), each of which may be equal or unequal to each other, and equal or unequal to any one of the first to third matrices described above. In this fifth mode, the fourth quantization matrix and the fifth quantization matrix may be signaled to a decoder or other device to which the coded stream is to be transmitted.

[0117] In the procedures described in the present disclosure, as mentioned above, each group of transform coefficients passed to this process typically belongs to a specific plane and layer. Typically, they have been scaled using a linear quantizer, which in some instances uses a non-centered dequantization offset. The scaling process for the transform coefficients may be applied as follows. The process may obtain position information for a block, together with a set of parameters by which the properties of the block (such as its size), the properties of the quantization operation (such as step width and offset value), and the enhancement level to which it applies may be indicated. For example, a position that may be indicated by a pair of coordinate values ​​or parameters (xTbP, yTbP) that, for example, specify the upper left sample of the current luma or chroma transform block relative to the upper left luma or chroma sample of the current picture may be used. This may be related to a specific portion of the data representing the image, and may, for example, be related to a luma plane or a chroma plane, depending on the plane to which the transform coefficient belongs.

[0118] The aforementioned parameter that specifies the size of the current transform block, which in some examples may be referred to as nTbS, may have a value that depends on the type of transform, and specifically on the values ​​of the parameters that define the transform for decoding. This type parameter may be referred to as transform_type in some examples and may have values ​​of 0, 1, or 2-3 in some applications, corresponding to a 2x2 directional decomposition transform, a 4x4 directional decomposition transform, or a value or parameter that specifies a value of zero (because those elements are not used in the bitstream), respectively. In some examples, this transform_type parameter having a value of 0 may correspond to a number of layers being equal to 4, and if transform_type is equal to 1, the corresponding number of layers value may be 16. The size nTbS parameter may have a value of 2 if transform_type is equal to zero, and may have a value of 4 if transform_type is equal to 1.

[0119] Another input to the process is typically in the form of an array of entropy decoded quantized coefficients. This may be referred to as TransCoeffQ and has a size related to the size parameter mentioned above, specifically, has dimensions (nTbS)X(nTbS), i.e., the array may include decoding units or blocks as described herein. This array may be referred to as TransCoeffQ. The step width value may be specified by a parameter called stepWidth. The index of the enhancement level may be specified by a parameter, and in some instances may be referred to as idxLevel. If a dequantization offset is to be used, this may be specified by a parameter called dQuantOffset, for example. The parameter typically specifies the value of the dequantization offset parameter to be applied. This process typically produces an array of dequantized transform coefficients. This may, for example, be in the form of an array of size (nTbS)X(nTbS) containing elements d[x][y].

[0120] The values ​​in the output array, which may be referred to as d[x][y], are typically derived based on any of the step widths as described above and / or the index of the enhancement layer as applied to the quantization matrix (which may be referred to as qm) and the entropy decoded quantized coefficients (which may be referred to as TransCoeffQ[xTbP][yTbP] for the corresponding element). Moreover, the values ​​of the output array elements may additionally be derived with the aid of applying an offset to the results of the foregoing operations. For example, a parameter, which may be referred to as appliedOffset, may be added to the value to produce the output element value.

[0121] Therefore, this can involve the following relationship to calculate each element d of the array:

[0122] d[x][y]=(TransCoeffQ[xTbP][yTbP]*(stepWidth*qm[yTbP+(idxLevel*4)][xTbP]))+appliedOffset

[0123] In the above calculation as one example of how dequantization may be performed at a decoder, idxLevel may be 0 or 1, representing level 2 and 1, respectively. In the above calculation, and as described in other examples herein, values ​​from the quantization matrix are selected based on specific coefficients and specific enhancement levels.

[0124] With respect to an offset that may be applied in some examples, as shown in the above relationship, this parameter, which may be referred to as appliedOffset, is generally derived as follows.

[0125] In some cases, appliedOffset may be conditionally applied based on whether residual filtering is to be applied. This filtering may be applied during the inverse transform (e.g., Figure 6 410-1 in , and may be a deblocking filter. In these cases, the offset may be applied only if residual filtering is applied. For example, a parameter that specifies whether a deblocking filter should be applied, which may be referred to as deblocking_signalled, has a particular value, such as equal to 1, and the offset is calculated based on the aforementioned parameter that specifies a dequantization offset, which may be referred to as dQuantOffset.

[0126] appliedOffset may also be derived from a single value of dQuantOffset and may be adapted for application to both positive and negative coefficient values. For example, appliedOffset may be configured to be a negative or positive value, typically having the same absolute value as the dequantization offset parameter and having a positive or negative sign, respectively. This may be applied if the TransCoeffQ value has a value less than zero; otherwise, if the TransCoeffQ value is greater than zero, then the applied offset may be assigned a value equal to the value of the dequantization offset parameter. If the TransCoeffQ value is equal to zero, then the applied offset may also be assigned a value of zero.

[0127] Thus, in some examples, the offsets that can be added to obtain output array elements can be derived by the following algorithm (where TransCoeffQ can be an array containing entropy decoded quantized coefficients of typical size (nTbS)X(nTbS)):

[0128] If deblocking_signalled is equal to 1

[0129]

[0130] For example, a parameter or set of parameters may be signaled as described above and used to specify how the quantization matrix is ​​to be applied to a set of coefficients. This parameter may be a quant_matrix_mode parameter that specifies which quantization matrices are to be used in the decoding process according to Table 1 below. The parameter may be one byte. The quant_matrix_mode parameter may be used to configure how the quantization matrix is ​​derived (e.g., as described later below).

[0131] Table 1 - Quantization matrix

[0132]

[0133] The quantization matrix data semantics may be applied according to the following specific example: A quantization matrix, which may be referred to as qm[y][x], is derived.

[0134] The matrix qm may have dimensions k*M by N. That is, the matrix may be defined to correspond to the quantization coefficients contained in an M by N matrix, and qm may contain such data in the form of a respective M by N matrix of quantization coefficients for each of the k enhancement levels, for example.

[0135] In the current example, two enhancement levels are involved, namely levels 1 and 2 as described in this disclosure, and k is equal to 2. The maximum transform as described with respect to these procedures is 4x4, and thus M and N may each be equal to 4.

[0136] In some examples, by defining qm with M and N values ​​corresponding to the maximum available transform (which is a 4×4 transform in these examples mentioned above, and therefore includes 16 coefficients), the application of a quantization matrix in which a smaller transform is employed can be performed by reading from qm and applying a subset of the coefficients in a corresponding M by N matrix.

[0137] In a specific example, if the value of the quantization matrix mode parameter is equal to zero, that is, if quant_matrix_mode is equal to 0, the following default quantization matrix is ​​used in the quantization operation:

[0138] qm[y][x]=

[0139] {0.500 0.500 0.500 0.617}

[0140] {0.862 0.610 1.064 0.781}

[0141] {0.500 0.500 0.500 0.617}

[0142] {3.125 1.851 1.851 1.316}

[0143] {0.500 0.500 0.500 0.617}

[0144] {0.862 0.610 1.064 0.781}

[0145] {0.862 1.064 0.610 0.781}

[0146] {3.125 1.851 1.851 1.316},

[0147] where y=0..3 are coefficients to be used for enhancement level 2, and y=4..7 are coefficients to be used for enhancement level 1; and

[0148] If the value of the quantization matrix mode parameter is equal to 1, then one matrix of modifiers is signaled for use on both enhancement levels. In some applications the matrix may be constructed with the aid of an iterative process. Thus, the qm[x][y] values ​​may be obtained row by row, whereby for each row there is a certain value in the row and each column of the matrix is ​​filled as the row is processed. The step width modifier parameter may be used in the process of obtaining these matrix values, as described below. In the current example, the number of rows to be filled may be 8 and the number of columns equal to 4, as described above in conjunction with the matrix sizes in this particular example. Specifically, the iterative process for this example may be written as:

[0149] for(y=0;y<8;y++)

[0150] for(x=0;x<4;x++)

[0151] qm[y][x]=step_width_modiffer_2[x+y*4],

[0152] wherein a step width modifier parameter, which may be referred to in this example as step_width_modifier_2, is a parameter that specifies the values ​​of the level 2 enhancement 16 coefficients to be applied at different layers of transform coefficients; and

[0153] If the value of the quantization matrix mode parameter is equal to 2, one matrix of modifiers is signaled and used on enhancement level 2. Again, specifically, an iterative procedure corresponding to the one described above may be used:

[0154] for(y=0;y<4;y++)

[0155] for(x=0;x<4;x<4)

[0156] qm[y][x]=step_width_modifier_2[x+y*4]; and

[0157] If the value of the quantization matrix mode parameter is equal to 3, one matrix of modifiers is signaled and used at enhancement level 1:

[0158] for(y=0;y<4;y++)

[0159] for(x=0;x<4;x<4)

[0160] qm[y+4][x]=step_width_modifier_2[x+y*4]; and

[0161] If the value of the quantization matrix mode parameter is equal to 4, two matrices of modifiers are signaled, the first for enhancement level 2 and the second for enhancement level 1:

[0162]

[0163] Where step_width_modifier_1 is a parameter that specifies the value of the level 1 enhancement 16 coefficients to be applied at different layers of transform coefficients.

[0164] As described above, the value of d[x][y] in the matrix can be calculated, for example, as the product of the corresponding transform coefficient at the relevant element in the matrix and the sum of the corresponding quantization matrix element values ​​at the column, wherein the column is identified by the product of the parameter specifying the size of the current transform block and the levelIdxSwap parameter, and in the corresponding row, the step width modifier parameter value corresponds to the element, wherein the offset described above is typically also applied additively to the product.

[0165] The dequantization process described above may be performed according to the following ordered steps. The dequantization process according to the foregoing example may be invoked with as inputs the luma position (xTbY, yTbY), the transform size set equal to nTbS (i.e., the size of the current transform block), an array TransCoeffQ as described above of size (nTbS)X(nTbS), and a step width parameter. The output may thus be an array of dequantized coefficients (e.g., dequantized transformed residuals) of size (nTbS)X(nTbS), which in this example may be referred to as dequantCoeff.

[0166] The methods and processes described herein may be embodied as code (e.g., software code) and / or data at both encoders and decoders implemented, for example, in a streaming server or client device or a client device that decodes from a data repository. Encoders and decoders may be implemented in hardware or software, as is well known in the field of data compression. For example, hardware acceleration using a specially programmed graphics processing unit (GPU) or a specially designed field programmable gate array (FPGA) may provide certain efficiencies. For completeness, such code and data may be stored on one or more computer-readable media, which may include any device or medium that can store code and / or data for use by a computer system. When a computer system reads and executes the code and / or data stored on a computer-readable medium, the computer system executes the methods and processes embodied as data structures and codes stored in a computer-readable storage medium. In certain embodiments, one or more of the steps of the methods and processes described herein may be executed by a processor (e.g., a processor of a computer system or a data storage system).

[0167] In general, any of the functions described herein or shown in the drawings may be implemented using software, firmware (e.g., fixed logic circuitry), programmable or non-programmable hardware, or a combination of these embodiments. In general, the terms "component" or "function" as used herein refer to software, firmware, hardware, or a combination of these. For example, in the case of a software embodiment, the term "component" or "function" may refer to program code that performs a specified task when executed on one or more processing devices. The illustrated separation of components and functions into distinct units may reflect any actual or conceptual physical grouping and allocation of such software and / or hardware and tasks.

Claims

1. A method for encoding an input video into a plurality of encoded streams, wherein the encoded streams can be combined to reconstruct the input video, the method comprising: Receive input video; downsampling the input video to create a downsampled video; Instructing to encode the downsampled video using a base encoder to create a base encoded stream; Instructing to decode the base coded stream using a base decoder to generate a reconstructed video; comparing the reconstructed video to the downsampled video to create a first set of residuals; encoding the first residual set to create a first-level coded stream; decoding the first residual set to generate a decoded first residual set; Correcting the reconstructed video using the decoded first residual set to generate a corrected reconstructed video; upsampling the corrected reconstructed video to generate an upsampled reconstructed video; comparing the upsampled reconstructed video to the input video to create a second set of residuals; Encoding the first residual set and the second residual set to create a first-level coded stream and a second-level coded stream, the encoding comprising: applying respective transforms to the first set of residuals and the second set of residuals to create respective first set of coefficients and second set of coefficients; applying respective quantization operations to the first set of coefficients and the second set of coefficients to create a first set of quantized coefficients and a second set of quantized coefficients; as well as applying corresponding encoding operations to the first set of quantized coefficients and the second set of quantized coefficients, Wherein applying the quantization operation comprises: adapting the quantization accordingly based on the first set of coefficients and the second set of coefficients to be quantized, including changing the step width used for different coefficients in the first set of coefficients and the second set of coefficients, wherein a first set of parameters and a second set of parameters derived from the adapting are signaled to a decoder to enable dequantization of the first set of quantized coefficients and the second set of quantized coefficients, wherein one or more of the first set of parameters and the second set of parameters are signaled using a quantization matrix, The method further comprises: transmitting a quantization matrix mode parameter indicating how values ​​within the quantization matrix are to be applied to one or more of the first set of coefficients and the second set of coefficients, The quantization matrix mode parameter indicates a first mode in which the decoder is to use a set of values ​​within the quantization matrix for both the first layer coded stream and the second layer coded stream. 2 . The method of claim 1 , wherein the first and second parameter sets include signaling to indicate that a default set of one or more quantization matrices is to be used at the decoder.

3. The method according to claim 1 or 2, comprising: combining at least the first layer coded stream and the second layer coded stream into a combined coded stream; as well as The combined encoded stream is transmitted to the decoder for use along with the received base encoded streams to reconstruct the input video. The method of claim 3 , wherein the combined encoded stream comprises the base encoded stream.

5. The method of claim 1 or 2, wherein applying the quantization operation comprises quantizing coefficients using a linear quantizer, wherein the linear quantizer uses a variable size dead zone. The method according to claim 1 , wherein the quantization operation further comprises using a quantization offset. The method of claim 6 , wherein the quantization offset is selectively signaled to the decoder.

8. A method according to claim 1 or 2, comprising adapting a distribution used in the quantizing step.

9. The method of claim 1 or 2, wherein adapting the quantization is predetermined and / or selectively applied based on analysis of any one or more of: the input video, the downsampled video, the reconstructed video, and the upsampled video.

10. The method of claim 1 or 2, wherein adapting the quantization is selectively applied based on a set of predetermined rules and / or deterministically applied based on analysis or feedback of decoding performance.

11. The method of claim 1 or 2, wherein encoding the residual comprises applying the encoding to blocks of the residual associated with a frame of the input video, wherein each block is encoded without using image data from another block in the frame so that each block can be encoded in parallel, wherein each element position in the block has a corresponding quantization parameter for varying the step width.

12. A method of decoding an encoded stream into a reconstructed output video, the method comprising: Receiving a first basic coded stream; Instructing to use a basic decoder to perform a decoding operation on the first basic coded stream to generate a first output video; Receiving a first-layer coded stream and a second-layer coded stream; Decoding the first-layer coded stream and the second-layer coded stream to obtain a first residual set and a second residual set; as well as, combining the first set of residuals with the first output video to generate a reconstructed video, and combining the second set of residuals with an upsampled version of the reconstructed video to generate a reconstruction of the original resolution input video; The decoding of the first-level coded stream and the second-level coded stream comprises: Decoding a set of first quantized coefficients from the first-level coded stream, and decoding a set of second quantized coefficients from the second-level coded stream; Obtaining a first set of parameters and a second set of parameters indicating how to dequantize the first set of quantized coefficients and the second set of quantized coefficients, respectively; and dequantizing the first set of quantized coefficients and the second set of quantized coefficients, respectively, using the first set of parameters and the second set of parameters, wherein different quantized coefficients in the first set of quantized coefficients and the second set of quantized coefficients are dequantized using corresponding dequantization parameters, Wherein obtaining the first parameter set comprises: obtaining a quantization mode parameter signaled along with the first-level coded stream; and In response to the first value of the quantization mode parameter, a default quantization matrix is ​​used as the first parameter set and the second parameter set.

13. The method according to claim 12, wherein In response to other values ​​of the quantization mode parameter, a quantization matrix signaled with the first-level coding stream is obtained and used as the first parameter set.

14. The method according to claim 12 or 13, wherein decoding the first-level coded stream comprises: Applying an entropy decoding operation to the first level coded stream prior to dequantizing the first set of quantized coefficients; as well as After dequantizing the first set of quantized coefficients, an inverse transform operation is applied to generate the first set of residuals.

15. The method according to claim 12 or 13, wherein obtaining the first and second parameter sets comprises: obtaining a quantization matrix signaled with one or more of the first and second level coded streams, and Dequantizing includes a plurality of quantized coefficient elements within a block of quantized coefficients for a video frame, the block corresponding to an N by N grid of picture elements, the frame including a plurality of blocks covering a spatial region associated with the frame: obtaining a quantization parameter from the quantization matrix based on a position of a given quantized coefficient element; as well as The given quantized coefficient element is dequantized using the quantization parameter.

16. The method of claim 12 or 13, wherein dequantizing comprises using a linear dequantization operation and applying a non-centered dequantization offset.

17. An encoder for encoding an input video, the encoder being configured to perform the method according to any one of claims 1 to 11.

18. A decoder for decoding an encoded stream into a reconstructed output video, the decoder being configured to perform the method according to any one of claims 12 to 16.

19. A non-transitory computer-readable storage medium storing instructions which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 16.

Images