Quantization of residuals in video coding

Abstract

A method of decoding an encoded stream into a reconstructed output signal is provided, the method comprising: receiving a first output signal decoded from a first base encoded stream according to a first codec; receiving a hierarchical encoded stream; decoding the hierarchical encoded stream to obtain a set of residuals; and combining the set of residuals with the first output signal to generate a reconstructed signal, wherein the decoding the hierarchical encoded stream comprises: decoding a set of quantized coefficients from the hierarchical encoded stream; dequantizing the set of quantized coefficients.

Description

Background Technology

[0001] Hybrid backward-compatible decoding techniques have previously been proposed in, for example, WO 2014 / 170819 and WO 2018 / 046940 (the contents of which are incorporated herein by reference).

[0002] One proposed method involves parsing a data stream into a first portion of encoded data and a second portion of encoded data; implementing a first decoder to decode the first portion of encoded data into a first reproduction of a signal; implementing a second decoder to decode the second portion of encoded data into reconstructed data, the reconstructed data specifying how to modify the first reproduction of the signal; and applying the reconstructed data to the first reproduction of the signal to produce a second reproduction of the signal.

[0003] A further proposed method involves using a set of residual features to reconstruct a reproduction of a first time-series sample of the signal. A set of spatiotemporal correlation features associated with the first time-series sample is generated. This set of spatiotemporal correlation features indicates the degree of spatial correlation between multiple residual features and the degree of temporal correlation between the first reference data based on the reproduction and the second reference data based on the reproduction of a second time-series sample of the signal. The set of spatiotemporal correlation features is used to generate output data. As mentioned, the residual set is encoded to reduce the overall data size.

[0004] Encoding applications typically employ quantization operations. This compression process, by compressing each of one or more ranges of data values ​​into a single value, allows for a reduction in the number of distinct values ​​in the set of video data, thereby making the data easier to compress. In this way, quantization schemes can be used in some videos to transform signals into quanta, allowing specific variables to take only specific discrete values. Typically, video codecs divide visual data in the form of video frames into discrete blocks, usually of a predetermined size or number of pixels. A transformation is then typically applied to these blocks to express the visual data based on the sum of frequency components. The transformed data can then be pre-multiplied by a quantization bit code and then divided element-wise by the quantization matrix, with each pre-multiplied element divided by a matrix element to obtain an output element that is then rounded. Dividing the different transformed elements by a divisor (i.e., different elements of the quantization matrix) is often used to allow those frequency elements that have a greater impact on the viewer's visual appearance of the video to be efficiently allocated more data or resolution compared to less perceptible components.

[0005] The goal is to optimize for further reducing the overall data size, while balancing the objective of not compromising the user's overall impression once the signal has been reconstructed with optimizing processing speed and complexity.

[0006] Quantization is a fundamental subprocess of virtually any encoding method. More precisely, quantization is the procedure of constraining the values ​​of signal properties (e.g., the brightness of pixels in an image) from a generally continuous set of values ​​(e.g., a set of real or high-bit-depth digital values, the “original set”) to a relatively small discrete set (e.g., a finite and low-bit-depth set of integer values, the “quantized set”).

[0007] An important aspect of quantization is the corresponding process of "inverse quantization" or "dequantization," which is used to obtain values ​​belonging to the original (i.e., larger) set of values ​​from the quantized values ​​(i.e., values ​​belonging to a small discrete set). Each value in the small discrete set actually corresponds to multiple (and possibly infinite) values ​​in the larger set of values, so the key issue in dequantization is limiting which of the multiple values ​​should be selected.

[0008] Traditional encoding methods quantize values ​​in the original set without knowing how those values ​​are statistically distributed. Therefore, a typical decoding method selects a value from the original set that roughly falls within the middle of a defined range in the original set, based on each quantized value in the quantized set. For example, if the quantized value Q1 corresponds to the range (10, 19) in the original set, a typical dequantizer would assign the value 14 to Q1. Clearly, it would be preferable to make a different dequantization decision if the dequantizer possessed information indicating that 90% of the values ​​in the original set belonging to the range (10, 19) are actually either 10 or 11. Unfortunately, information about the statistical distribution of the original values ​​is generally not readily available in the encoding methods used in this technique so far.

[0009] Another important characteristic of quantization methods is the limitation of each quantization range (i.e., the range of original values ​​in the original set corresponding to each quantized value in the quantization set). Again, in the absence of known and reliable data on the statistical distribution of values ​​in the original set, most methods choose a uniform quantization step size, i.e., a range of equal size.

[0010] In summary, the quantization and dequantization methods available in this technique are inevitably suboptimal in terms of both the quality of signal reproduction (for lossy coding) and compression performance because they cannot be based on and / or driven by the statistical distribution of values ​​in the original set. Summary of the Invention

[0011] According to one aspect, a method is provided for encoding an input signal into a plurality of encoded streams, wherein the encoded streams can be combined to reconstruct the input signal, the method comprising: receiving the input signal; downsampling the input signal to create a downsampled signal; instructing the encoding of the downsampled signal using a basic encoder to create a basic encoded stream; instructing the decoding of the basic encoded stream using a basic decoder to generate a reconstructed signal; comparing the reconstructed signal with an input video to create a set of residuals; and encoding the set of residuals to create a hierarchical encoded stream, comprising: applying a transform to the set of residuals to create a set of transformed coefficients; applying a quantization operation to the set of transformed coefficients to create a set of quantized coefficients; and applying an encoding operation to the quantized coefficients.

[0012] According to another aspect, a method is provided for decoding an encoded stream into a reconstructed output signal, the method comprising: receiving a first output signal decoded from a first base encoded stream according to a first codec; receiving a hierarchical encoded stream; decoding the hierarchical encoded stream to obtain a set of residuals; and combining the set of residuals with the first output signal to generate a reconstructed signal, wherein decoding the hierarchical encoded stream comprises: decoding a set of quantized coefficients from the hierarchical encoded stream; and dequantizing the set of quantized coefficients. The combination may include a combination with an upsampled format of the first output signal.

[0013] The hierarchical coded stream can be a first-level coded stream; the set of quantized coefficients can be a first set of quantized coefficients; and the residual set can be a first residual set, and the method may further include: receiving a second-level coded stream; decoding the second-level coded stream to obtain a second residual set; and combining the second residual set with an upsampling pattern of the reconstructed signal to generate a reconstruction of the original resolution input signal, wherein decoding the second-level coded stream includes: decoding a second set of quantized coefficients from the second-level coded stream; and dequantizing the second set of quantized coefficients.

[0014] The method advantageously allows for improved efficiency in the encoding and decoding processes by adjusting the degree and / or manner of compression applied to coefficients during quantization based on any one of several factors, according to the video data to be decoded. Thus, the manner in which a typically lossy quantization procedure is performed during the encoding of a video stream can be adapted to apply an appropriate balance between encoding or compression efficiency and visually perceptible compression of the input video, depending on the nature and content of the input video—a relationship that can vary considerably across different video frames and streams. This adaptable form of quantization can be used in conjunction with the dequantization process at the receiving decoder, for example, by signaling to the decoder, via a transmission of parameters having values ​​representing or indicating the information, how quantization has been performed or the degree to which quantization has been changed from the default mode.

[0015] In some embodiments, the block folding process can also be used to enhance decoding efficiency. Specifically, in these cases, the quantization operation further includes applying a block folding operation to a set of coefficients, wherein each coefficient having a value exceeding a predetermined maximum value is quantized to have a quantized value corresponding to a first quantized block group among a plurality of quantized block groups having a defined step width, the maximum value being defined by an upper limit of the first quantized block group. This can be done to place all residual or coefficient values ​​residing above a selected quantized block group into the selected block group. The first block group can be considered to correspond to an upper value, or a block group corresponding to the highest (absolute) quantized value, relative to what can be understood as the endpoint of the range of values ​​involved. Block folding can be implemented at either the upper or lower endpoints. A similar process can be performed for negative values ​​within the range. Block folding can be configured to adjust or reduce the bit rate based on at least one of network conditions and underlying stream processing. Thus, the block folding process can be configurable itself, for example, in various embodiments, any one or two of conditions and underlying stream processing or parameters derived therefrom are used to configure block folding, such as parameters defining block folding. In some embodiments, the quantization operation further includes using a dequantization offset. This may include applying the dequantization offset to a plurality of quantization block groups having a defined step size to adjust each of one or more corresponding values ​​in the plurality of quantization block groups by the value of the dequantization offset.

[0016] In these embodiments, the value corresponding to each of the plurality of quantization block groups may be adjusted. Alternatively, any one or both of the values ​​corresponding to the beginning of the first block group and the end of the last block group may be left unadjusted by a quantization offset; that is, they remain unadjusted by a quantization offset. The first block group can be understood as corresponding to the numerical minimum value or the minimum value of the range. Similarly, the last block group can be understood as representing the maximum value of the range or the numerical maximum value. These adjustments and non-adjustments may be combined with quantization operations involving dead zones and block group folding, as described later in this disclosure.

[0017] Typically, the dequantization offset value is adjustable or configurable. In some embodiments, the dequantization offset value may vary based on data indicating the operating conditions under which encoding is performed.

[0018] In some embodiments, the method further includes signaling a quantization offset value to a decoder that will receive the encoded stream. This signaling may be performed, for example, in an embodiment where the quantization offset value changes dynamically during encoding.

[0019] Dequantization typically involves subtracting the dequantization offset from the residual or coefficient value before quantization based on the quantization step width.

[0020] In some embodiments, the value of the dequantization offset is adjusted based on the sign of the residuals or coefficients. This can be implemented to allow symmetric operations with respect to zero values.

[0021] The method can be executed such that when the dequantization offset value is set to a first predetermined value, the application of the offset to the block group value is deactivated. For example, this can be done by setting the quantization or dequantization offset value to zero.

[0022] In some embodiments, the dequantization offset value is adjusted based on the width defined by the dead zone. In these embodiments, the dead zone can be used to perform quantization operations, as detailed later in this disclosure. The dequantization offset can be configured or adjusted based on the width defined by the dead zone.

[0023] In some embodiments, the method includes deriving a corresponding dequantization offset δ for each of the encoded coefficient groups, typically from a single reference dequantization offset δ corresponding to the transmitted signal reference StepWidth of a given data layer. i Offset value δ i Typically, it depends at least in part on the reference offset δ and the quantization StepWidth of the coefficient set i. i One or both of them.

[0024] In some embodiments, the offset of all quantization steps can be derived from the reference step according to the following relationship:

[0025] Dequantization_Offsetk = f (StepWidth k ),

[0026] and the following relationships:

[0027] Actual_Dequantization_Offset i =

[0028] = Reference_Dequantization_Offset + f (StepWidth i ) - f (Reference_StepWidth).

[0029] In these cases, f(StepWidth) is usually logarithmic.

[0030] In some embodiments, the method may involve varying the step width used in the quantization operation according to a step width parameter. Specifically, the step width may be varied for each of one or more coefficients in a set of coefficients, for example, for different coefficients within a 2×2 or 4×4 block of coefficients. For instance, the step width may be varied such that a smaller step width value is used for one or more coefficients that are pre-determined to have a greater impact on the perception of the decoded signal. The degree of impact is typically determined experimentally, thereby obtaining information indicating which coefficients have a greater impact on the viewer's perception of the decoded signal.

[0031] The step width is typically assigned a default value based on a base step width parameter. One or more modified step widths can be obtained based on the base step width and the step width modifier parameter. For example, this can be done by obtaining the modified step width according to the formula modified_stepwidth = base_stepwidth * modifier, where the modifier can be set based on a specific coefficient within a block or cell. In these embodiments, the corresponding step width modifier parameter can be used to modify the step width for each of one or more of the coefficients.

[0032] In some embodiments, a corresponding step width value may be used for, or associated with, each of two or more coded streams or enhancement layers, which include a base coded stream and one or more enhancement layer coded streams.

[0033] In some embodiments, the stride modifier parameter is varied according to the enhancement level, which depends on the enhancement level employed. The stride modifier can be varied such that a smaller stride is used for the first-level coded stream and a larger stride is used for the base coded stream.

[0034] In some preferred embodiments, the quantization operation uses a quantization matrix defined by a set of step width modifier parameter values ​​for different coefficients and different enhancement levels. Therefore, the method may involve a corresponding step width modifier parameter value for each coefficient and each enhancement level. The method can be performed by an encoder and a corresponding decoding process by a decoder, and the quantization matrix can be obtained by various means in different embodiments. Specifically, the quantization matrix can be preset at at least one of the encoder and decoder, or the quantization matrix can be transmitted between the encoder and decoder, and alternatively, the quantization matrix can be dynamically constructed at at least one of the encoder and decoder.

[0035] The method may further include constructing the quantization matrix as a function of at least one of one or more stored and one or more transmitted parameters.

[0036] In some embodiments, the scaled transform coefficients d[x][y] can be derived according to the following formula, where x=0...nTbS−1, y=0...nTbS-1, and a given quantization matrix qm[x][y]:

[0037] d[x][y] = (TransformCoeffQ[x][y] * ((qm[x + (levelIdxSwap * nTbS)][y]+ stepWidthModifier[x][y]) + appliedOffset [x][y]),

[0038] Where TransformCoeffQ is an array of (nTbS) x (nTbS) sizes of quantized transform coefficients containing entropy decoding.

[0039] `levelIdx` is a variable that specifies the index of the enhanced sublayer, and `appliedOffset[x][y]` and `stepWidthModifier[x][y]` are variables. Furthermore, `appliedOffset[x][y]` can correspond to the dead zone, as described elsewhere in this document.

[0040] For example, levelIdx can be set to 1 for enhancement sublayer 1, and lbe can be set to 2 for enhancement sublayer 2.

[0041] Typically, stepWidthModifier[x][y] is derived from the following variable:

[0042] If dequant_offset_signalled_flag=0, then stepWidthModifier[x][y]=((((Floor (−Cconst*Ln(qm[x+(levelIdxSwap*nTbS)][y])))+Dconst)*(qm[x+(levelIdxSwap*nTbS)][y]2))) / 32768), where Cconst and Dconst are constants and can have values ​​of 5242 and 99614 respectively in a single instance.

[0043] In some embodiments, the method includes transmitting the plurality of encoded streams along with signaling information indicating whether a dequantization offset will be applied during a dequantization operation. Thus, signaling information indicating whether a quantization offset has been applied during a quantization operation can be provided; in some embodiments, the dequantization offset corresponds to or is equal to the quantization offset. Typically, the offset is at least one function of the quantization step width.

[0044] In these cases, the offset is typically a function of the probability distribution of the values ​​before quantization, serving as a complement or alternative to the step width function.

[0045] In some embodiments, the method includes applying a modification to the quantization step width to generate a modified quantization step width, and applying the modified quantization step width to quantize one or more coefficients. Preferably, the modification is determined based on a probability distribution of the values ​​before quantization.

[0046] Typically, quantization operations involve quantizing the coefficients using a linear quantizer, which employs a variable-size dead zone.

[0047] In these embodiments, the size of the dead zone can be set to a predetermined multiple of the step width used in the quantization operation, for example, as a linear function of the step width value. Alternatively, a nonlinear function of the step width value can be used.

[0048] In some preferred embodiments, the step size used in the quantization operation is variable, and the size of the dead zone is more preferably adjusted according to the variable step size.

[0049] The size of the dead zone can be set by multiplying the step width used in the quantization operation by a multiplier parameter, wherein the value of the multiplier parameter varies based on data indicating the operating conditions under which encoding is performed (e.g., the available bit rate). Therefore, in some embodiments, the multiplier can also be adaptive.

[0050] In some embodiments, the size of the dead zone is calculated as a multiple of the quantization step width according to the formula Dead Zone = DZmultiplier x StepWidth, where the dead zone multiplier DZmultiplier is calculated as a linear function of StepWidth according to DZmultiplier = a * StepWidth + b, where a and b are constants, typically suitable constants known to both the encoder and decoder that respectively perform the method and the corresponding decoding method.

[0051] Quantization is typically performed based on one or more quantization parameters. These parameters are typically set to control and provide a desired bit rate in one or more encoded streams. That is, they can be set to control and / or provide a desired bit rate in one or more encoded streams. The desired bit rate is a common bit rate for all streams to generate a common encoded stream, or it may provide different bit rates for different encoded streams.

[0052] In these embodiments, the one or more quantization parameters are preferably set to provide the desired quality level or maximize the quality level within a set of predefined bit rate constraints. This can be understood as adjusting the parameters to provide the highest or largest possible quality level.

[0053] The method may include determining quantization parameters by receiving the state of a buffer that receives one or more encoded streams and a base encoded stream, and by using the state to determine quantization parameters.

[0054] The buffer is preferably used to store and / or combine the encoded base stream and the encoded enhancement stream, and is configured to receive input at a variable bit rate while reading output at a constant rate. The rate controller can read the buffer's state to ensure that the buffer does not overflow or become empty, and that data is always available for reading at its output. The buffer's state can also be used to generate the one or more quantization parameters. The one or more quantization parameters can be controlled based on the amount of data within the buffer.

[0055] Typically, the value of the quantization parameter is inversely related to the amount of data in the buffer.

[0056] In some embodiments, quantization parameters are determined for each frame, residual, and / or residual group, that is, for at least one of: each frame, residual, and residual group. Typically, the quantization parameters of a frame can be determined using a previous set of quantization parameters based on the target data size of the frame and the current data size of the frame. In any of these embodiments, the quantization parameters may be based on a previous set of quantization parameters.

[0057] The method may include defining a set of curves to map a normalized size to one or more quantization parameters, wherein each curve includes one or more of a multiplier and an offset depending on the nature of the current frame. The set of curves may be defined to map a normalized size to quantization parameters. Each curve may have one or more of a multiplier and an offset, the offset of which may depend on the nature of the current frame (e.g., it may depend on the complexity of the information to be encoded intra-frame). The multiplier and offset may define the shape of the curve. The multiplier may be applied to a size normalization function, which is a function of the quantization parameter Q. In one case, the current size (i.e., in Q...) t-1 (size of encoded frame t) and Q t-1 A point can be used to define a space within the set of curves. This point can be used to select a set of curves closest to the given curves. These curves can be curves above and below the point, or the highest or lowest curve of the point. The set of closest curves, along with the point, can be used in an interpolation function to determine a new curve associated with the point. Once this new curve is determined, a multiplier and offset can be determined for the new curve. These values ​​can then be used, along with the received target size, to determine the value of Qt (e.g., the curve can define a function of size and Q). Therefore, typically, a multiplier is applied to a size normalization function, which is a function of the quantization parameter Q. t-1 The current size of the encoded frame t and Q t-1Points within a space that can be used to define a set of curves, wherein said points are used to select a set of curves that are closest to the given curves. The closest curves can be the curves above and below the point, or the highest or lowest curve of the point.

[0058] In these cases, the set of closest curves is typically used together with the points in the interpolation function to determine a new curve associated with those points, and the multiplier and offset of the determined new curve can be determined, further including using the values ​​of the multiplier and offset of the determined new curve values ​​together with the received target size to determine Q. t The value of .

[0059] The set of curves can be stored in accessible memory and updated based on the set of curves determined for a previous frame. In some cases, adaptive quantization can be applied in different ways for different coefficient positions within a decoding unit or block, such as for different elements in an array of 4 or 16 coefficients (e.g., for a 2×2 or 4×4 transform).

[0060] In some embodiments, quantization is performed using a quantization matrix derived from the obtained values ​​of quantization matrix mode parameters. Preferably, in these cases, the quantization matrix mode parameters specify the quantization matrix to be used in the encoding process.

[0061] Typically, correspondingly different quantization matrices are used for each of two or more layers in the encoded stream. More preferably, different quantization matrices are used for each encoded stream, wherein a default quantization configuration is predetermined, and variations relative to the default configuration are transmitted between the encoder and decoder.

[0062] In these embodiments, the method may include causing different quantization matrices to be used for the corresponding encoded streams by means of at least one of the following: default configuration, and causing a common quantization matrix to be used for the corresponding encoded streams by means of signaling overriding the default configuration.

[0063] In some embodiments, the quantization matrix is ​​used only for a subset of multiple enhancement layers or only for one of multiple enhancement layers. The quantization matrix is ​​typically indexed according to the position of the coefficients within a block in which the coefficients are arranged.

[0064] In some embodiments, the base quantization matrix is ​​defined by a set of values ​​and modified according to a scaling factor, which is a function of the stride width of one or more enhancement layers. For example, the scaling factor may be computed as a clamping function of the stride width parameter. In some embodiments, each entry in the quantization matrix is ​​scaled using an exponential function of the scaling factor.

[0065] In some embodiments, the linear quantizer uses a non-centered dequantization offset. More specifically, the linear quantizer may use a dead zone of a different size relative to the quantization step size and the non-centered dequantization offset.

[0066] Typically, the dequantization offset value is received from the encoder. The received offset value can be added to the received quantization value before step-width-based dequantization.

[0067] According to another aspect, a method is provided for decoding an encoded stream into a reconstructed output video, the method comprising: receiving a first output signal decoded from a first base encoded stream according to a first codec; receiving a hierarchical encoded stream; decoding the hierarchical encoded stream to obtain a residual set; and combining the residual set with the first output signal to generate a reconstructed signal, wherein the received encoded stream includes a group of transform coefficients scaled using a linear quantizer, and the decoding includes applying a dequantization operation, wherein the dequantization operation is performed using a dead zone, wherein the size of the dead zone is obtained according to the step width used in the dequantization operation and quantization parameters received from the encoder. The combination may include a combination with an upsampled format of the first output signal.

[0068] In some preferred embodiments, the method includes determining from signaling information whether a dequantization offset will be applied when performing a dequantization operation. The signaling information may be received directly or indirectly from an encoder. The information may be within a bitstream associated with a signal. The dequantization offset can be understood as the dequantization offset previously described in this disclosure.

[0069] In these embodiments, the dequantization offset is typically a function of at least one of the quantization step widths. Preferably, as a supplement to or alternative to the step width function, the dequantization offset is typically a function of the probability distribution of the values ​​before quantization.

[0070] In some embodiments, the method includes applying a modification to a quantization step to generate a modified quantization step, and applying the modified quantization step to dequantize one or more quantized values. In these embodiments, the modification is typically determined based on a probability distribution of the values ​​prior to quantization. Preferably, the method includes encoding the input video or signal prior to decoding, as described, for example, with respect to the first aspect. The method may include quantizing a value in a manner different from dequantization.

[0071] The method may include encoding a signal prior to decoding, the method including applying a quantization step width to quantize one or more values ​​to generate the one or more quantized values, such that quantization and dequantization are asymmetric.

[0072] In these embodiments involving asymmetric decoding, the method may further include modifications during signal encoding to the decoder or decoding operation.

[0073] Asymmetric encoder and decoder levels can be implemented in several ways. In some cases, this can be both on the wider encoder side, meaning that the encoders at L1 and L2 can be asymmetric with the decoder that forms the part of the L1 path.

[0074] The following describes several additional non-limiting implementations of asymmetric quantization and dequantization, which can also be implemented in combination with the features related to the above aspects described above.

[0075] A method is also provided for encoding an input video into a plurality of encoded streams and decoding at least one of the plurality of encoded streams, wherein the encoding includes applying a first quantization operation using a first set of parameters, and the decoding includes asymmetricly applying a second dequantization using a different set of parameters than the first quantization operation. Decoding may be performed at a decoder or as part of the reconstruction path of the encoding method.

[0076] A method is also provided for encoding an input video into a plurality of encoded streams and decoding at least one of the plurality of encoded streams, wherein the encoding includes applying a first quantization operation and transmitting a plurality of parameters to the decoding process, and the decoding includes receiving the plurality of parameters and applying a second dequantization using parameters different from the first quantization operation based on the received plurality of parameters.

[0077] A method is also provided for encoding an input video into multiple encoded streams, wherein the encoded streams can be combined to reconstruct the input video. The method includes receiving the input video, which is typically a full-resolution input video. The method may further include applying a quantization operation to a set of coefficients to create a set of quantized coefficients; and applying an encoding operation to the set of quantized coefficients, wherein applying the quantization operation includes adapting the quantization based on a first set of coefficients to be quantized, including changing the step width for different coefficients in the first set of coefficients. The method further includes signaling modified parameters, and wherein the adapted first set of parameters is signaled to a decoder to modify the dequantization of the first set of quantized coefficients, wherein the modified parameters are used by the decoder to modify the first set of parameters such that the dequantization parameters used are different from the quantization parameters used.

[0078] Also provided is a method for decoding an encoded stream into a reconstructed output video, the method comprising: retrieving dequantization modification parameters; decoding a set of quantized coefficients from a first-level encoded stream; obtaining a parameter set indicating how to dequantize the quantized coefficients; modifying the first parameter set using the retrieved dequantization modification parameters; and dequantizing the set of quantized coefficients using the parameter set, wherein dequantizing different quantized coefficients in the set of quantized coefficients using the corresponding dequantization parameters results in dequantization being asymmetric to quantization.

[0079] According to another aspect, an encoder for encoding input video is provided, the encoder being configured to perform the method according to the first aspect.

[0080] According to another aspect, a decoder is provided for decoding an encoded stream into a reconstructed output video, the decoder being configured to perform the method according to the decoding aspect described above.

[0081] According to another aspect, a system including an encoder and a decoder is provided. The system is adapted to perform asymmetric quantization as previously described in this disclosure.

[0082] According to another aspect, a non-transitory computer-readable storage medium is provided that stores instructions which, when executed by a processor, cause the processor to perform the method.

[0083] More generally, a method is provided for encoding an input signal into multiple encoded streams. This method can be implemented using any of the implementation details described above.

[0084] The video decoding techniques described in this disclosure provide more efficient and effective quantization and dequantization in the context of hierarchical coding methods compared to conventional approaches. This can be further understood by examining the embodiments described below, which may utilize any of the aspects described above.

[0085] First, as will be understood in this disclosure, in the context of a hierarchical decoding format, a signal is decomposed into multiple “tiers” (also referred to as “hierarchical levels”) of data, each tier corresponding to a “quality level” (“LoQ”) of the signal, ranging from the highest tier at the sampling rate of the original signal to the lowest tier, which typically has a sampling rate lower than that of the original signal. In a non-limiting example, when the signal is an image, the lowest tier could be a thumbnail of the original image, or even just a single pixel. The other tiers contain information about corrections to be applied to the reconstruction to produce the final output. The decoded signal at a given quality level is reconstructed by first decoding the lowest tier (thus reconstructing the signal at a first lowest quality level), then predicting the reconstruction of the signal at a second next higher quality level, then decoding the corresponding second tier of the reconstructed data (also referred to as “residual data” at the second quality level), then combining the prediction with the reconstructed data to reconstruct the reconstruction of the signal at the second higher quality level, and so on, until a given quality level is reconstructed.

[0086] Different decoding formats can be used to decode different data echelons, and different quality levels can have different sampling rates (e.g., resolution for image or video signals). Subsequent echelons can refer to the same signal resolution (i.e., sampling rate) or to progressively higher signal resolution.

[0087] The methods and embodiments described herein also include ways to appropriately utilize the predictable statistical characteristics of residual data in hierarchical encoding methods. Specifically, in the context of a hierarchical structure, residual data is constructed by means of a distribution based on a known probability distribution pattern (e.g., by means of non-restrictive instances, Laplace distribution, Poisson distribution, etc.): thus it is possible to quickly identify the key parameters of the probability distribution and indicate them to the decoder by means of a relatively limited set of metadata.

[0088] Furthermore, the non-limiting embodiments herein also include methods for appropriately adapting quantization and dequantization operations based on the known probability distribution of the original symbols, in order to simultaneously improve compression efficiency, reconstruction quality, and processing speed. The non-limiting embodiments herein also include methods for minimizing signaling overhead by deriving, using default formulas, how the optimal dequantization offset for each coefficient group should change based on the relative magnitude of its quantization step width.

[0089] The embodiments described herein deviate from conventional systems and methods, thereby providing a novel approach to dynamically adapt quantization and dequantization operations based on appropriate and concise indications of the characteristics of the probability distribution of the original unquantized values.

[0090] In its broadest aspect, the present invention is a method for rapidly evaluating the probability distribution of raw unquantized values, generating appropriate signaling metadata based on said distribution, and adapting quantization and dequantization operations based on said metadata. The invention also includes decoder-side methods to improve the statistical properties of the dequantization offset even in the absence of signaling information from the encoder.

[0091] These methods are particularly valuable and effective in the context of hierarchical coding methods because the hierarchical structure results in implicitly defining the type of probability distribution of the residual values. This, in turn, allows each specific probability distribution to be characterized with relatively few and quickly identifiable parameters, and exhibits several properties of the dequantization offset (e.g., its relationship to the magnitude of the quantization step width). According to the non-limiting embodiments described herein, the novel dequantization methods described herein are used in loops within the encoding and decoding layers of hierarchical coding methods.

[0092] For simplicity, the non-limiting embodiments described herein refer to the signal as a sequence of samples (i.e., two-dimensional images, video frames, video fields, audio frames, etc.). In the description, the terms "image," "picture," or "plane" (in the broadest sense of a "hyperplane," i.e., an array of elements having any number of dimensions and a given sampling raster) will often be used to identify a digital reproduction of a signal along a sample sequence, wherein each plane has a given resolution in each of its dimensions (e.g., X and Y) and comprises a set of planar elements (or "elements" or "pixels" or display elements, often referred to as "pixels" in a two-dimensional image, often referred to as "volumetric elements," etc.), characterized as one or more "values" or "settings" (e.g., by means of non-limiting examples, color settings in a suitable color space, settings indicating density levels, settings indicating temperature levels, settings indicating audio pitch shift, settings indicating amplitude, etc.). Each planar element is identified by a suitable set of coordinates indicating the integer position of the element within the sampling raster of the image. The signal dimension may include only the spatial dimension (e.g., in the case of an image) or also include the temporal dimension (e.g., in the case of a signal that evolves over time).

[0093] As a non-limiting example, the signal can be an image, an audio signal, a multi-channel audio signal, a telemetry signal, a video signal, a multi-view video signal (e.g., 3D video), a volumetric signal (e.g., medical imaging, scientific imaging, holographic imaging, etc.), a volumetric video signal, or even a signal with more than four dimensions.

[0094] For simplicity, the non-limiting embodiments described herein often refer to signals displayed as a defined 2D plane (e.g., a 2D image in a suitable color space), such as video signals. The terms "frame" or "field" will be used interchangeably with the term "image" to indicate a temporal sample of a video signal: any concepts and methods described for video signals composed of frames (progressive video signals) can also be readily applied to video signals composed of fields (interlaced video signals), and vice versa. Although the embodiments described herein focus on image and video signals, those skilled in the art will readily understand that the same concepts and methods are also applicable to any other type of multidimensional signal (e.g., audio signals, volumetric signals, stereoscopic video signals, 3DoF / 6DoF video signals, all-optical signals, etc.).

[0095] According to some embodiments described herein, a signal processor (e.g., computer processor hardware) is configured to receive data to be quantized and a corresponding reference quantization step width to be used for quantizing said data. Based at least in part on the quantization step width, the signal processor identifies a range of non-quantized values ​​corresponding to a quantized value of zero (“Dead Zone”). The use of dead zones has been previously described in this disclosure. In some non-limiting embodiments, the encoder calculates the Dead Zone based on a formula known to both the encoder and decoder, according to the reference step width. This allows for variation of the relative size of the Dead Zone quantization block group without any signaling overhead. In a non-limiting embodiment, the Dead Zone is calculated as a multiple of the quantization step width (i.e., Dead Zone = DZmultiplier * StepWidth), where the Dead Zone multiplier DZmultiplier is calculated as a linear function of StepWidth (i.e., DZmultiplier = a * StepWidth + b, where a and b are suitable constants known to both the encoder and decoder). In other non-limiting embodiments, the relationship between DZmultipliers is non-linear. In some non-limiting embodiments, the DZmultiplier is represented by a fixed-point integer number to avoid using hardware-specific floating-point calculations.

[0096] According to some other embodiments described herein, the encoder identifies a dequantization offset value δ based at least in part on a metric calculated from the data to be quantized, and adds it to the central limit of each non-quantized value range (“quantized block group”) to obtain a dequantized value (R_i) corresponding to Qi. The dequantization of the quantized symbol Q is performed by both the encoder (for analog decoding) and the decoder as follows.

[0097] If the quantized sign Qi is positive:

[0098]

[0099] If the symbol Qi is actually negative:

[0100]

[0101] In some other embodiments, the decoder generates a dequantization offset value δ by analyzing the signal-encoded byte stream. If the signaling bit corresponding to the dequantization offset is set to 1, the decoder processes subsequent bytes in the byte stream to generate a reference dequantization offset.

[0102] According to some other embodiments, for multiple encoded data sets (e.g., groups of encoded transform coefficients), the encoder signals a single parameter corresponding to a reference quantization StepWidth to the decoder, and the decoder uses a quantization matrix to process the parameter corresponding to the reference StepWidth and derives a distinct quantization StepWidth for each of the groups of transform coefficients. To limit the number of dequantization offsets to be transmitted as meta-data in the byte stream, both the encoder and decoder automatically derive a specific dequantization offset value δi for each of the encoded coefficient groups from a single reference dequantization offset value δ, which is predetermined to correspond to the offset of the signaled reference StepWidth of the data layer. This offset value δi depends at least in part on the signaled dequantization offset δ and the quantization StepWidthi of coefficient set i.

[0103] In some embodiments, the encoder and decoder calculate the offsets of all quantized StepWidths derived from the reference StepWidth by means of the following assumed relationship:

[0104] Dequantization_Offsetk = f (StepWidthk)

[0105] And therefore:

[0106] Actual_Dequantization_Offseti =

[0107] = Reference_Dequantization_Offset + f (StepWidthi) - f (Reference_StepWidth)

[0108] In some embodiments, f(StepWidth) is a logarithmic relation.

[0109] In other embodiments, the encoder derives the reference dequantization offset value δ using an RDO (“rate distortion optimization”) algorithm. By way of a non-limiting example, in one embodiment, the encoder calculates the reference dequantization offset value δ by performing a bipartite search optimization algorithm on the data to be quantized using a suitable metric (e.g., by way of a non-limiting example, MSE, MAE, PSNR) to determine the fidelity of the reproduction produced by dequantizing with a given dequantization offset value δ. In a non-limiting embodiment, the signal processor selects a value that minimizes the mean square error between the original unquantized value and the dequantized value (R_i). In other non-limiting embodiments, the encoder performs the optimization only considering the residuals corresponding to regions of the signal that have been identified as having relatively high priority according to a suitable priority mapping.

[0110] In other embodiments, the encoder selects a dequantization offset and detects coefficient values ​​quantized at dequantization points that are likely far from the corresponding quantization buckets. Accordingly, the encoder assigns quantized symbols from adjacent quantization buckets to the coefficients to reduce the measure of the difference between the unquantized value of the coefficient and the dequantized value of the quantized symbol assigned to the coefficient. In a non-limiting embodiment, the encoder detects at least one coefficient with respect to an edge and assigns it a different quantized symbol to enhance edge transitions.

[0111] In another embodiment, the encoder performs quantization in two steps: a first pass of quantization with a relatively large step width and a relatively small dead zone, the purpose of which is to group adjacent residual values ​​into clusters; and then, actual quantization with a relatively small target quantization step width, to detect the reference solution quantization offset of the coefficient group.

[0112] In some embodiments, the decoder is configured to generate progressively larger dequantization offsets as the quantization bucket moves from the center (zero symbol / dead zone) of the residual distribution to the two tails of the distribution. In some non-limiting embodiments, the decoder produces a StepWidth_Modifieri value for each quantization StepWidthi, which depends on StepWidthi. The decoder then produces Decoding_StepWidthi = (StepWidthi + StepWidth_Modifieri). Decoding_StepWidthi is then used instead of StepWidthi to perform dequantization.

[0113] In some embodiments, the encoder signals the dequantization offset value δ as encoded metadata within the encoded bitstream. In one embodiment, the decoder detects whether a dequantization offset is signaled for a given group of quantized coefficients using a given signaling bit in the byte stream: if the bit is set to 1, subsequent bytes contain information corresponding to the magnitude of the reference dequantization offset. In other non-limiting embodiments, when no dequantization offset is signaled, the decoder continues to compute the StepWidth_Modifier for each group of transform coefficients. In other non-limiting embodiments, based on a second signaling bit, the decoder processes the byte stream to generate a reference dequantization offset and simultaneously generates one or more StepWidth_Modifiers to be added to the StepWidth to progressively increase the dequantization offset. The StepWidth_Modifier can be used to avoid artifacts such as mosquito noise.

[0114] In some embodiments, processing operations and intermediate variables are performed using fixed-point integers, thereby avoiding the execution of processing operations that depend on floating-point numbers.

[0115] In some other non-limiting embodiments, the offset value δi for each quantized block group is generated based on parameters of the probability distribution identifying the unquantized values. In non-limiting embodiments, it is assumed that the offset value is calculated at least in part based on a Laplace (i.e., double exponential) probability distribution identified from the metadata. In other non-limiting embodiments, the offset value is decoded directly from the metadata. In non-limiting embodiments, the offset value δ is the same for all block groups except those centered around zero; the offset value δ is added to the very center edge of the quantized block group for block groups corresponding to positive unquantized values ​​(thus reducing the amount of the dequantized value relative to the average of the block group range), and subtracted to the average of the block group for block groups corresponding to negative unquantized values ​​(thus again reducing the absolute value of the dequantized value relative to the average of the block group range).

[0116] In other embodiments, the signal processor is configured to perform a random dequantization process (“statistical jitter” or “statistical dequantization”) to reproduce a probability distribution similar to the probability distribution of the original unquantized values ​​quantized into the same quantized block group i for the dequantized values. The signal processor receives metadata indicating the probability distribution of the original unquantized values. Based at least in part on the parameters of the probability distribution, on the minimum and maximum values ​​of each block group, and on randomly generated values, the signal processor generates a dequantized value (R_i) for each quantized value Qi. In these embodiments, multiple occurrences of the same quantized value Qi can be dequantized into multiple different dequantized values ​​(R_i). In a non-limiting embodiment, the parameters in the metadata indicating the probability distribution of the original unquantized values ​​are assumed to be parameters of a Laplace distribution.

[0117] In other embodiments, the signal processor processes received metadata corresponding to the quantized data and generates a parameter StepWidthi indicating the size of the quantized block group and information corresponding to the probability distribution of the original unquantized data. Based at least in part on the parameter and the information, the signal processor generates a range of all quantized block groups to be used for dequantizing the received quantized data. In a non-limiting embodiment, the size of the quantized block group is non-uniform. In other non-limiting embodiments, the size of the quantized block group is non-uniform and is obtained at least in part based on a lookup table known to the signal processor. In other non-limiting embodiments, dequantization follows a Dead Zone plus uniform step width (DZ+USW) scheme, where both DZ and USW are functions of the parameter StepWidthi. In a non-limiting embodiment, the dequantization offset is also a function of the parameter StepWidth and is calculated as f(StepWidth).

[0118] According to other embodiments, the signal processor pre-calculates a set of parameters based on the received metadata and stores the parameters in a lookup table to accelerate the dequantization process of the quantized data.

[0119] According to some embodiments, the information corresponding to the probability distribution of the original unquantized values ​​includes a parameter corresponding to the diversity parameter (“b”) of the Laplace distribution. In other non-limiting embodiments, the information corresponding to the probability distribution of the original unquantized values ​​includes a parameter corresponding to the probability of some quantized block groups (“mini-histograms”).

[0120] In some embodiments, in the context of a hierarchical (layered) decoding method, the quantized data corresponds to residual data. A signal processor configured as a decoder decodes the reconstructed signal at a first (lower) quality level. The decoder then processes the reconstructed signal at the first quality level to produce a preliminary reconstruction of the signal at a second (higher) quality level. The decoder then decodes the received quantized data according to embodiments herein and produces dequantized reconstructed data. Based at least in part on the dequantized reconstructed data, the decoder produces residual data. The decoder then combines the residual data with the preliminary reconstruction of the signal at the second quality level to produce a reconstructed signal at the second quality level.

[0121] Some embodiments include a signal processor configured to produce a set of values ​​(“quantized data”) corresponding to the difference between a set of elements of a signal at a given higher quality level and a corresponding predicted signal at the same quality level. The method includes, within an encoder, receiving: a reproduction of the signal at the higher quality level; generating a reproduction of the signal at a lower quality level; generating a predicted reproduction of the signal at the lower quality level; generating a set of residual data corresponding to the difference between the reproduction of the signal at the lower quality level and the predicted reproduction of the signal at the lower quality level; generating a set of unquantized reconstructed data based at least in part on the set of residual data; quantizing the set of unquantized reconstructed data according to a non-limiting method described herein, thereby producing quantized data and corresponding dequantized metadata; and according to the method described herein... The non-restrictive dequantization method described herein processes a set of quantized data and corresponding dequantized metadata to generate dequantized reconstructed data; processes the dequantized reconstructed data to generate dequantized residual data; combines the dequantized residual data with a predicted reproduction of the signal at a lower quality level to generate a reconstructed reproduction of the signal at a lower quality level; processes the reconstructed reproduction of the signal at a lower quality level to generate a predicted reproduction of the signal at a higher quality level; generates a set of residual data corresponding to the difference between the reproduction of the signal at a higher quality level and the predicted reproduction of the signal at a higher quality level; generates a set of unquantized reconstructed data based at least in part on the set of residual data; and quantizes the set of unquantized reconstructed data according to the non-restrictive method described herein to generate quantized data and corresponding dequantized metadata.

[0122] On the other hand, the decoder and encoder can be asymmetric. That is, the decoding process can use parameters different from or modified than those used in the corresponding encoding process. This applies to both decoding operations performed in the encoding method and decoding operations performed in the decoding method. In an example, the quantization operation performed during the encoding operation can use a step width formula, while the quantization operation performed during the decoding operation can modify or modulate the step width formula. Several further non-limiting examples, aspects, and implementations of asymmetric quantization and dequantization are described below.

[0123] According to another aspect of the invention, a method is provided for encoding an input video into a plurality of encoded streams and decoding at least one of the plurality of encoded streams, wherein the encoding includes applying a first quantization operation using a first set of parameters, and the decoding includes asymmetricly applying a second dequantization using a different set of parameters than the first quantization operation. Decoding may be performed at a decoder or as part of the reconstruction path of the encoding method.

[0124] According to another aspect of the present invention, a method is provided for encoding an input video into a plurality of encoded streams and decoding at least one of the plurality of encoded streams, wherein the encoding includes applying a first quantization operation and transmitting a plurality of parameters to the decoding process, and the decoding includes receiving the plurality of parameters and applying a second dequantization using parameters different from the first quantization operation based on the received plurality of parameters.

[0125] According to another aspect of the present invention, a method is provided for encoding an input video into a plurality of encoded streams, wherein the encoded streams can be combined to reconstruct the input video, the method comprising: receiving the input video, which is typically a full-resolution input video; downsampling the input video to create a downsampled video; instructing to encode the downsampled video using a base encoder to create a base encoded stream; instructing to decode the base encoded stream using a base decoder to generate a reconstructed video; comparing the reconstructed video with the downsampled video to create a first residual set; and encoding the first residual set to create a first-level encoded stream, comprising: applying a transform to the first residual set to... The method involves: creating a first set of coefficients; applying a quantization operation to the first set of coefficients to create a first set of quantized coefficients; and applying an encoding operation to the first set of quantized coefficients, wherein applying the quantization operation includes: adapting quantization based on the first set of coefficients to be quantized, including changing the step width used for different coefficients in the first set of coefficients, wherein the method further includes signaling modified parameters, and wherein a first set of parameters derived from the adaptation is signaled to a decoder to modify the dequantization of the first set of quantized coefficients, wherein the modified parameters are used by the decoder to modify the first set of parameters such that the dequantization parameters used are different from the quantization parameters used.

[0126] According to another aspect, a method for decoding an encoded stream into a reconstructed output video can be provided, the method comprising: retrieving dequantization modification parameters; receiving a first base encoded stream; instructing a decoding operation on the first base encoded stream using a base decoder to generate a first output video; receiving a first-level encoded stream; decoding the first-level encoded stream to obtain a first set of residuals; and combining the first set of residuals with the first output video to generate a reconstructed video, wherein decoding the first-level encoded stream comprises: decoding a first set of quantized coefficients from the first-level encoded stream; obtaining a first set of parameters indicating how to dequantize the first set of quantized coefficients; modifying the first set of parameters using the retrieved dequantization modification parameters; and dequantizing the first set of quantized coefficients using the first set of parameters, wherein different quantized coefficients in the first set of quantized coefficients are dequantized using corresponding dequantization parameters such that dequantization and quantization are asymmetric. Attached Figure Description

[0127] Figure 1 A high-level diagram illustrating the coding process;

[0128] Figure 2 A high-level diagram illustrating the decoding process;

[0129] Figure 3 A high-level diagram illustrating the encoding process and specific encoding steps;

[0130] Figure 4 A high-level diagram illustrating the decoding process and specific decoding steps;

[0131] Figure 5 A high-level diagram illustrating the coding process;

[0132] Figure 6 A high-level diagram illustrating another decoding process;

[0133] Figure 7 A flowchart illustrating the concepts described in this article;

[0134] Figures 8A-8D Demonstrates how quantization can be performed during the coding process based on a specific instance;

[0135] Figure 9 Display the residual distribution of the instance;

[0136] Figure 10 This demonstrates examples of generating and using dequantization modifiers;

[0137] Figure 11 An example of dequantization offset is shown;

[0138] Figure 12 Another example diagram showing an instance of dequantization offset is presented, similar to... Figure 11 ;as well as,

[0139] Figure 13 A block diagram illustrating an instance computer system that provides data processing based on instance methods;

[0140] Figure 14 It is a schematic diagram of an encoder; and,

[0141] Figure 15 This is a schematic diagram of a decoder. Detailed Implementation

[0142] This invention relates to methods. More specifically, this invention relates to methods for encoding and decoding signals. Data processing may include, but is not limited to, acquiring, exporting, outputting, receiving, and reconstructing data.

[0143] The decoding technique discussed in this paper is a flexible, adaptable, efficient, and computationally inexpensive decoding format that combines a video decoding format, a base codec (e.g., AVC, HEVC, or any other current or future codec), and decoded data with enhancement levels encoded using different techniques. The technique uses a downsampled source signal, which is encoded using the base codec to form a base stream. Enhancement streams are formed using, for example, a set of encoded residuals from the base stream, either by increasing resolution or by improving frame rate correction or enhancement. Multiple enhancement levels can exist in a hierarchical structure. In a particular arrangement, the base stream can be decoded by a hardware decoder, while the enhancement streams can be adapted for software implementations.

[0144] Any optimizations used in the new decoding technique, tailored to the specific requirements or constraints of the enhancement stream and with low complexity, are crucial. Such requirements or constraints include: the potential reduction in computational power resulting from the need for software decoding of the enhancement stream; the need for the combination of the decoded set of residuals with the decoded frames; the possible structure of the residual data, i.e., a relatively high proportion of zero values ​​and a large range of highly variable data values; subtle differences in the input quantization blocks of the coefficients; and the structure of the enhancement stream for a set of discrete residual frames separated into various components. It should be noted that the constraints placed on the enhancement stream imply simple and fast entropy decoding operations as the basis so that the enhancement stream can effectively correct or enhance individual frames of the underlying decoded video. It should also be noted that in some scenarios, the underlying stream is decoded substantially simultaneously before combination, which puts pressure on resources.

[0145] In one scenario, the method described herein can be applied to so-called data planes that reflect different color components of a video signal. For example, the method described herein can be applied to different planes that reflect YUV or RGB data for different color channels. Different color channels can be processed in parallel. Therefore, a reference to the residual set as described herein can include multiple residual sets, where each color component has a different residual set forming part of a combined enhancement stream. Components of each first-order stream can be compared in any logical order; for example, each plane at the same level can be grouped together and sent together, or residual sets from different levels within each plane can be sent together.

[0146] This current document preferably meets the requirements of the following ISO / IEC documents: "Call for Proposals for Low Complexity Video Coding Enhancements" ISO / IEC JTC1 / SC29 / WG11 N17944, Macau, China, October 2018, and "Requirements for Complexity Video Coding Enhancements" ISO / IEC JTC1 / SC29 / WG11 N18098, Macau, China, October 2018 (which are incorporated herein by reference).

[0147] The general structure of the proposed coding scheme, to which the currently described technology can be applied, uses a downsampled source signal encoded by a base codec, adds first-level correction data to the decoded output of the base codec to generate a corrected picture, and then adds another level of enhancement data to the upsampled form of the corrected picture. Thus, the stream is considered as a base stream and an enhancement stream. This structure creates multiple degrees of freedom, allowing for great flexibility and adaptability to many situations, making the decoding format suitable for many use cases, including over-the-top (OTT) transmission, live streaming, live ultra-high-definition (UHD) broadcasting, etc. Although the decoded output of the base codec is not intended for viewing, it is a fully decoded video at a lower resolution, thus making the output compatible with existing decoders and usable as a lower-resolution output where appropriate. In some cases, the base codec can be used to create a base stream. The base codec may include a standalone codec controlled in a modular or "black box" manner. The methods described herein can be implemented using computer program code, which is executed by a processor and makes function calls on a hardware and / or software-implemented codec.

[0148] Generally, as used herein, the term "residual" refers to the difference between the values ​​of a reference array or reference frame and the actual array or frame of data. The array can be a one-dimensional or two-dimensional array representing a decoding unit. For example, a decoding unit can be a 2×2 or 4×4 set of residual values ​​corresponding to a region of similar size to an input video frame. It should be noted that this generalized example is unknown to the nature of the encoding operation performed and the input signal. References to "residual data" as used herein refer to data derived from the residual set, such as the residual set itself or the output of a set of data processing operations performed on the residual set. Throughout this specification, generally, a residual set contains multiple residuals or residual elements, each corresponding to a signal element, i.e., an element of the signal or original data. The signal can be an image or video. In these instances, the residual set corresponds to an image or frame of video, where each residual is associated with a pixel of the signal, which is a signal element. The examples disclosed herein describe how these residuals can be modified (i.e., processed) to affect the encoding pipeline or the final decoded image while reducing the overall data size. Residuals or sets can be processed on a per-residual-feature (or residual) basis, or on a group basis, such as per-patch or per-decoding-unit basis, where a patch or decoding unit is a neighboring subset of the residual set. In one case, a patch may comprise a group of smaller decoding units. It should be noted that processing can be performed on each video frame or on only a specified number of frames in the sequence.

[0149] Generally, each enhancement stream or two enhancement streams can be encapsulated into one or more enhancement bitstreams using a set of Network Abstraction Units (NALUs). NALUs are intended to encapsulate enhancement bitstreams so that enhancements are applied to the correct underlying reconstructed frames. A NALU may, for example, contain a reference index to the NALU containing the underlying decoder reconstructed frame bitstream to which the enhancements must be applied. In this way, enhancements can be synchronized to the underlying stream, and frames from each bitstream are combined to produce the decoded output video (i.e., the residual of each frame from the enhancement layer combined with frames from the underlying decoded stream). A group of images can represent multiple NALUs.

[0150] Returning to the initial process described above, where the base stream is provided along with enhancements at two levels (or sub-levels) within the enhancement stream, an instance of the generalized coding process is depicted in... Figure 1In the block diagram, the input full-resolution video 100 is processed to generate various coded streams 101, 102, and 103. A first coded stream (coded base stream) is generated by feeding a downsampled version of the input video to a base codec (e.g., AVC, HEVC, or any other codec). The coded base stream may be referred to as the base layer or base level. A second coded stream (coded level 1 stream) is generated by processing the residual obtained by the difference between the downsampled version of the reconstructed base codec video and the input video. A third coded stream (coded level 2 stream) is generated by processing the residual obtained by the difference between the upsampled version of the reconstructed base decoded video and the input video. In some cases, Figure 1 The components can provide a general low-complexity encoder. In some cases, enhanced streams can be generated through an encoding process that forms part of the low-complexity encoder, and the low-complexity encoder can be configured to control independent base encoders and decoders (e.g., encapsulated as a base codec). In other cases, the base encoder and decoder can be supplied as part of the low-complexity encoder. In one case, Figure 1 A low-complexity encoder can be viewed as a form of envelope for a base codec, where the functionality of the base codec is hidden from the entity implementing the low-complexity encoder.

[0151] The downsampling operation illustrated by downsampling component 105 can be applied to input video to produce downsampled video to be encoded by the base encoder 113 of the base codec. Downsampling can be performed in both the vertical and horizontal directions, or alternatively only in the horizontal direction. The base encoder 113 and the base decoder 114 can be implemented by the base codec (e.g., as different functions of a common codec). The base codec and / or one or more of the base encoder 113 and base decoder 114 may include appropriately configured electronic circuitry (e.g., hardware encoder / decoder) and / or computer program code executed by a processor.

[0152] Each enhanced stream coding process may not necessarily include an upsampling step. For example, in Figure 1 In this context, the first enhancement flow is conceptually a correction flow, while the second enhancement flow is upsampled to provide enhancement levels.

[0153] For more details, see the process of generating the enhanced stream. To generate the encoded Level 1 stream, the encoded base stream is decoded by the base decoder 114 (i.e., a decoding operation is applied to the encoded base stream to generate the decoded base stream). Decoding can be performed by the decoding function or mode of the base codec. Then, at the Level 1 comparator 110, the difference between the decoded base stream and the undersampled input video is created (i.e., a subtraction operation is applied to the undersampled input video and the decoded base stream to generate a first set of residuals). The output of comparator 110 may be referred to as the first set of residuals, such as a surface or frame of residual data, where a residual value is determined for each pixel at the resolution of the outputs of the base encoder 113, the base decoder 114, and the undersampled block 105.

[0154] The difference is then encoded by the first encoder 115 (i.e., the level 1 encoder) to generate an encoded level 1 stream 102 (i.e., the encoding operation is applied to the first residual set to generate the first enhanced stream).

[0155] As described above, the enhancement stream may include a first enhancement level 102 and a second enhancement level 103. The first enhancement level 102 may be considered as a corrected stream, for example, a stream that provides a correction level to the underlying coded / decoded video signal at a resolution lower than that of the input video 100. The second enhancement level 103 may be considered as another enhancement level that converts the corrected stream into the original input video 100, for example, by applying an enhancement level or correction to the signal reconstructed from the corrected stream.

[0156] exist Figure 1 In this example, a second enhancement layer 103 is created by encoding another set of residuals. This other set of residuals is generated by a layer 2 comparator 119. The layer 2 comparator 119 determines the difference between the upsampled form of the decoded layer 1 stream, such as the output of the upsampling component 117, and the input video 100. The input to the upsampling component 117 is generated by applying a first decoder (i.e., a layer 1 decoder) to the output of the first encoder 115. This generates a decoded set of layer 1 residuals. These residuals are then combined with the output of the base decoder 114 at the summing component 120. This effectively applies the layer 1 residuals to the output of the base decoder 114. This allows losses during layer 1 encoding and decoding to be corrected by layer 2 residuals. The output of the summing component 120 can be considered as an analog signal representing the output of the encoded base stream 101 and the encoded layer 1 stream 102 at the decoder after applying layer 1 processing.

[0157] As mentioned, the upsampled stream is compared with the input video, which creates another set of residuals (i.e., the difference operation is applied to the recreated upsampled stream to generate another set of residuals). The other set of residuals is then encoded by the second encoder 121 (i.e., the level 2 encoder) into an encoded level 2 enhanced stream (i.e., the encoding operation is then applied to the other set of residuals to generate another encoded enhanced stream).

[0158] Therefore, as Figure 1 As shown and described above, the output of the encoding process is a base stream 101 and one or more enhancement streams 102, 103, which preferably include a first enhancement level and another enhancement level. The three streams 101, 102, and 103 can be combined, with or without additional information such as control headers, to generate a combined stream representing the video coding framework structure of the input video 100. It should be noted that... Figure 1 The components shown operate on blocks or decoding units of data, such as 2×2 or 4×4 portions of a frame at a specific resolution level. These components operate without any inter-block dependencies, thus allowing them to be applied in parallel to multiple blocks or decoding units within a frame. This differs from contrasting video coding schemes, where dependencies (e.g., spatial or temporal) exist between blocks. These dependencies limit the level of parallelism and require significantly higher complexity.

[0159] exist Figure 2 The block diagram depicts the corresponding generalized decoding process. It can be said that... Figure 2 The display corresponds to Figure 1 The low-complexity decoder is a low-complexity encoder. The low-complexity decoder receives three streams 101, 102, and 103 generated by the low-complexity encoder, along with a header 204 containing additional decoding information. The encoded base stream 101 is decoded by a base decoder 210 corresponding to the base codec used in the low-complexity encoder. The encoded level 1 stream 102 is received by a first decoder 211 (i.e., a level 1 decoder), which decodes streams generated by the low-complexity encoder. Figure 1 The first residual set encoded by the first encoder 115 is decoded. At the first summing component 212, the output of the base decoder 210 is combined with the decoded residual obtained from the first decoder 211. The combined video, which may be referred to as the layer 1 reconstructed video signal, is upsampled by the upsampling component 213. The encoded layer 2 stream 103 is received by the second decoder 214 (i.e., the layer 2 decoder). The second decoder 214 decodes the data as shown by the first encoder 115. Figure 1 The second encoder 121 decodes the second residual set encoded by the second encoder 121. Although header 204 is in Figure 2The video is shown as being used by the second decoder 214, but it can also be used by the first decoder 211 and the base decoder 210. The output of the second decoder 214 is a second set of decoded residuals. These can be at a higher resolution than the first set of residuals and the input to the upsampling component 213. At the second summing component 215, the second set of residuals from the second decoder 214 is combined with the output of the upsampling component 213 (i.e., the upsampled reconstructed level 1 signal) to reconstruct the decoded video 250.

[0160] According to a low-complexity encoder, Figure 2 The low-complexity decoder can operate in parallel on different blocks or decoding units of a given frame of the video signal. Furthermore, decoding performed by two or more of the base decoder 210, the first decoder 211, and the second decoder 214 can be executed in parallel. This is possible because there is no inter-block dependency.

[0161] During decoding, the decoder can parse header 204 (which may contain global configuration information, image or frame configuration information, and data block configuration information) and configure the low-complexity decoder based on those headers. To recreate the input video, the low-complexity decoder can decode each of the base stream, the first enhancement stream, and another or a second enhancement stream. Frames of the streams can be synchronized and then combined to derive decoded video 250. Depending on the configuration of the low-complexity encoder and decoder, decoded video 250 can be a lossy or lossless reconstruction of the original input video 100. In many cases, decoded video 250 can be a lossy reconstruction of the original input video 100, wherein the loss has a reduced or minimal impact on the perception of decoded video 250.

[0162] exist Figure 1 and 2 In each of these operations, the Level 2 and Level 1 encoding operations may include transform, quantization, and entropy encoding steps (e.g., in the order described). Similarly, at the decoding stage, the residual may be passed through an entropy decoder, a dequantizer, and an inverse transform module (e.g., in the order described). Any suitable encoding and corresponding decoding operations can be used. However, preferably, the Level 2 and Level 1 encoding steps may be performed in software (e.g., by one or more central or graphics processing units in the encoding device).

[0163] The transforms described herein can use directional decomposition transforms, such as Hadamard-based transforms. Both can include small kernels or matrices applied to the flattened decoding units of the residuals (i.e., 2×2 or 4×4 residual blocks). Further details regarding the transforms can be found, for example, in patent applications PCT / EP2013 / 059847 or PCT / GB2017 / 052632, which are incorporated herein by reference. The encoder can select between different transforms to be used, such as between kernel sizes to be applied.

[0164] The transformation can transform the residual information onto four surfaces. For example, the transformation can produce the following components: average, vertical, horizontal, and diagonal. As mentioned earlier in this disclosure, these components output by the transformation can be used as coefficients to be quantized according to the described method in these embodiments.

[0165] In summary, the methods and apparatus described in this paper are based on a general approach built upon existing encoding and / or decoding algorithms (e.g., MPEG standards such as AVC / H.264, HEVC / H.265, etc.; and non-standard algorithms such as VP9, ​​AV1, etc.) as a baseline for corresponding enhancement layers used for different encoding and / or decoding methods. The concept behind the general approach is to encode / decode video frames hierarchically, in contrast to the block-based approach used in MPEG family algorithms. Hierarchical frame encoding involves generating residuals for the entire frame, followed by generating residuals for extracted frames, and so on.

[0166] The video compression residual data for a full-size video frame can be referred to as LoQ-2 (e.g., 1920×1080 for HD video frames, or even higher for UHD frames), while the video compression residual data for a decimated frame can be referred to as LoQ-x, where x represents the number of decimations corresponding to the hierarchical decimation. Figure 1 and 2 In the described instance, the variable x may have values ​​1 and 2 representing the first and second enhancement streams. Therefore, there exist two hierarchical levels that will generate compressed residuals. Other naming schemes for the levels can also be applied without any functional changes (e.g., the level 1 and level 2 enhancement streams described herein can instead be referred to as level 1 and level 2 streams – indicating a countdown from the highest resolution).

[0167] Figure 3The block diagram depicts a more detailed encoding process. The encoding process is divided into two halves, as indicated by the dashed lines. Below the dashed lines is the base level of the encoder 300, which can be usefully implemented in hardware or software. Above the dashed lines is the enhancement level, which can be usefully implemented in software. The encoder 300 may include enhancement level-only processes, or a combination of base level and enhancement level processes, as needed. The encoder 300 can be usefully implemented in software, particularly at the enhancement level. This arrangement allows for upgrades, for example, using firmware (e.g., software) updates to provide the base level, where the firmware is configured to provide the enhancement level. In newer devices, both the base level and the enhancement level may be provided in hardware and / or a combination of hardware and software.

[0168] The encoder topology in a typical horizontal configuration is as follows. Encoder 300 includes input I for receiving input signal 30. Input signal 30 may include an input video signal, wherein the encoder is applied frame by frame. Input I is connected to downsampler 305D and processing block 300-2. Downsampler 305D may correspond to... Figure 1 The downsampling component 105, and the processing block 300-2 can correspond to Figure 1 The second encoder 121. The downsampler 305D outputs at the base level of the encoder 300 to the base codec 320. The base codec 320 can be implemented... Figure 1 The basic encoder 113 and basic decoder 114. The downsampler 305D also outputs to the processing block 300-1. The processing block 300-1 can correspond to Figure 1 The first encoder 115. Processing block 300-1 passes the output to upsampler 305U, which in turn outputs to processing block 300-2. Upsampler 305U can correspond to Figure 1 The upsampling component 117. Each of processing blocks 300-2 and 300-1 includes one or more of the following modules: transform block 310, quantization block 320, entropy encoding block 330, and residual processing block 350. Residual block 350 may occur before transform block 310 and / or control residual processing in processing block 300. The order of processing can be illustrated as shown in the figure.

[0169] Input signal 30, such as full (or highest) resolution video in this example, is processed by encoder 300 to generate various encoded streams. A basic encoded stream is generated by feeding a downsampled form of the input video 30 to a base codec 320 (e.g., AVC, HEVC, or any other codec) at the base level using downsampler 305D. The basic encoded stream may include the output of the base encoder of the base codec 320. A first encoded stream (encoded level 1 stream) is created by reconstructing the encoded basic stream to create a basic reconstruction, and then taking the difference between the basic reconstruction and the downsampled form of the input video 30. Reconstructing the encoded basic stream may include receiving the decoded basic stream from the base codec (i.e., the input to processing block 300-1 includes, for example, the decoded basic stream). Figure 1 The base decoded stream is shown. Next, the difference signal is processed at block 300-1 to create an encoded level 1 stream. Block 300-1 includes transform block 310-1, quantization block 320-1, and entropy coding block 330-1. A second encoded stream (encoded level 2 stream) is created by upsampling the corrected form of the base reconstruction using upsampler 305U and taking the difference between the corrected form of the base reconstruction and the input signal 30. This difference signal is then processed at block 300-2 to create an encoded level 2 stream. Block 300-2 includes transform block 310-2, quantization block 320-2, entropy coding block 330-2, and residual processing block 350-2. Following processing block 300-1, the blocks can be executed in the order shown in the figure (e.g., residual processing followed by transform followed by quantization followed by entropy coding).

[0170] Quantization schemes can be used to create quantities from residual signals, allowing specific variables to take only specific discrete values. In one case, quantization involves dividing by a predetermined step width. This can be applied at two levels (1 and 2). For example, quantization at block 320 could involve dividing the transformed residual value by the step width (e.g., where an integer quotient is used to generate the quantized value and the remainder is ignored). The step width can be predetermined, for example, chosen based on the desired quantization level. In one case, dividing by the step width can be converted to multiplying by the inverse step width, which can be implemented more efficiently in hardware. In this case, dequantization at, for example, block 320 could involve multiplying by the step width. Entropy coding as described herein can include run-length encoding (RLE), followed by processing the encoded output using a Huffman encoder. In some cases, when entropy coding is required, only one of these schemes can be used.

[0171] The encoded basic stream can be called the basic layer stream.

[0172] As previously described, the residual is calculated by comparing the original image signal with the reconstructed image signal. For example, in one case, the residual of the L-2 enhancement stream is determined by subtracting the upsampled output from the original image signal (e.g., the input video indicated in the figure). The upsampled input can be referred to as the reconstruction of the signal after analog decoding. In another case, the residual of the L-1 enhancement stream is determined by subtracting the image stream output by the base decoder from the downsampled original image signal (e.g., the downsampled output).

[0173] exist Figure 4 The block diagram depicts the corresponding Figure 3 The decoder 400 performs the decoding process of the encoder. The decoding process is divided into two halves, as shown by the dashed lines. Below the dashed lines is the base layer of the decoder 400, which can be usefully implemented in hardware. Above the dashed lines is the enhancement layer, which can be usefully implemented in software. The decoder 400 may include only the enhancement layer process, or a combination of the base layer process and the enhancement layer process, as needed. The decoder 400 can be usefully implemented in software, especially at the enhancement layer, and may suitably outperform conventional decoding techniques, especially conventional hardware techniques. Conventional techniques refer to earlier technologies that were previously developed and marketed, are inconvenient to replace and / or expensive to replace, and are still usable for the purpose of decoding signals. In other cases, the base layer may include any existing and / or future video coding tools or technologies.

[0174] The decoder topology at a typical level is as follows. Decoder 400 includes inputs (not shown) for receiving one or more input signals, including an encoded base stream, an encoded level 1 stream, and an encoded level 2 stream, along with an optional header containing additional decoding information. Decoder 400 includes a base decoder 420 at the base level and processing blocks 400-1 and 400-2 at the enhancement level. An upsampler 405U is also disposed between processing blocks 400-1 and 400-2 to provide processing block 400-2 with an upsampled version of the signal output from processing block 400-1. The base decoder 420 may correspond to... Figure 2 The basic decoder 210, processing block 400-1 can correspond to Figure 2 The first decoder 211, processing block 400-2 can correspond to Figure 2 The second decoder 214, and the upsampler 405U can correspond to Figure 2 The upper sampler 213.

[0175] Decoder 400 receives the one or more input signals and directs three streams generated by encoder 300. The encoded base stream is directed to and decoded by base decoder 420, which corresponds to the base codec 420 used in encoder 300 and is used to reverse the encoding process at the base level. The encoded level 1 stream is processed by block 400-1 of decoder 400 to recreate the first set of residuals created by encoder 300. Block 400-1 corresponds to processing block 300-1 in encoder 300 and is used at the base level to reverse or substantially reverse the processing of block 300-1. The output of base decoder 420 is combined with the first set of residuals obtained from the encoded level 1 stream. The combined signal is sampled by upsampler 405U. The encoded level 2 stream is processed by block 400-2 to recreate additional residuals created by encoder 300. Block 400-2 corresponds to processing block 300-2 of encoder 300 and is used at the base level to reverse or substantially reverse the processing of block 300-2. The upsampled signal from upsampler 405U is combined with additional residuals obtained from the encoded layer 2 stream to create a layer 2 reconstruction of the input signal 30. The output of processing block 400-2 can be considered similar to Figure 2 250 decoded videos.

[0176] As described above, the enhancement stream may include two streams: a coding level 1 stream (first enhancement level) and a coding level 2 stream (second enhancement level). The coding level 1 stream provides a set of correction data, which can be combined with the decoded form of the base stream to generate a corrected image.

[0177] Figure 5 More detailed display Figure 1 The encoder 300. The encoded base stream is created directly by the base encoder 320E and can be quantized and entropy encoded as needed. In some cases, these subsequent processes can be performed as part of the encoding performed by the base encoder 320E. To generate the encoded level 1 stream, the encoded base stream is decoded at the encoder 300 (i.e., the decoding operation is applied to the encoded base stream at the base decoder block 320D). The base decoder block 320D is presented as part of the base level of the encoder 300 and is presented as separate from the corresponding base encoder block 320E. For example, the base decoder 320D can be a decoding component that supplements the encoding components in the form of the base encoder 320E with the base codec. In other instances, the base decoder block 320D can actually be part of an enhancement level, and more precisely, part of processing block 300-1.

[0178] Return to Figure 5The difference between the decoded base stream output from the base decoder block 320D and the downsampled input video is created (i.e., subtraction operation 310-S is applied to the downsampled input video and the decoded base stream to generate a first set of residuals). Here, the term residual is used in the same manner as known in this art; that is, a residual represents the error or difference between a reference signal or frame and a desired signal or frame. Here, the reference signal or frame is the decoded base stream, and the desired signal or frame is the downsampled input video. Therefore, the residuals used in the first enhancement layer can be considered as correction signals because they are able to 'correct' the decoded base stream in the future to a closer approximation of the downsampled input video used in the base coding operation. This is useful because it can correct for quirks or other characteristics of the base codec. These characteristics include, in particular, the motion compensation algorithm applied by the base codec, the quantization and entropy coding applied by the base codec, and the block adjustment applied by the base codec.

[0179] exist Figure 5 More detailed display Figure 3 The components of block 300-1. Specifically, the first set of residuals is transformed, quantized, and entropy-encoded to produce an encoded level 1 stream. Figure 5 In this process, transform operation 310-1 is applied to the first residual set; quantization operation 320-1 is applied to the transformed residual set to generate a quantized residual set; and entropy coding operation 330-1 is applied to the quantized residual set to generate a coded level 1 stream at the first enhancement level. However, it should be noted that in other instances, only quantization step 320-1 or only transform step 310-1 may be performed. Entropy coding may not be used, or it may optionally be used as a supplement to one or both of transform step 110-1 and quantization step 320-1. The entropy coding operation can be any suitable type of entropy coding, such as a Huffman coding operation or a run-length encoding (RLE) operation, or a combination of both Huffman coding and RLE operations.

[0180] For example, the choice of entropy decoding schemes, such as iso-entropy decoding schemes, can have a beneficial effect on decoding performance in combination with the described quantization. This can be understood from the tendency to apply a higher degree of quantization to residual data as described in this disclosure to produce a high proportion of zero values. Run-length encoding, as mentioned above, is particularly suitable for encoding data with this distribution, and thus these methods can synergistically improve the efficiency of the overall process. Similarly, for embodiments where quantization is applied with a larger step width and the distribution of the quantized data results in a relatively large number of integer values, the efficiency of the encoding process will generally benefit from the use of prefix / Huffman coding, which is particularly suitable for these distributions. This is especially true in the case where the higher integer values ​​have lower frequency residuals. These forms of distributions can be efficiently encoded using Huffman coding, which works by allocating fewer bits to high-frequency symbols. In this way, quantization and entropy coding operations are complementary.

[0181] As described above, the enhancement stream may include a coded level 1 stream (first enhancement level) and a coded level 2 stream (second enhancement level). The first enhancement level can be viewed as implementing the corrected video at the base level, that is, for example, correcting encoder and / or decoder artifacts. The second enhancement level can be viewed as another enhancement level that can be used to convert the corrected video into the original input video or an approximation thereof (e.g., adding detail or sharpness). For example, the second enhancement level may add fine details lost during undersampling and / or help correct errors introduced by one or more of the transform operation 310-1 and quantization operation 320-1.

[0182] See Figure 3 and Figure 5 In order to generate the encoded level 2 stream, another enhancement level information is created at block 300-2 by generating and encoding another set of residuals. The other set of residuals is the difference between the upsampled form of the corrected form of the decoded base stream (reference signal or frame) (via upsampler 305U) and the input signal 30 (desired signal or frame).

[0183] To achieve the reconstruction of the corrected form of the decoded base stream generated at decoder 400, at least some processing steps of block 300-1 are reversed to simulate the process of decoder 200, taking into account at least some losses and oddities in the transform and quantization processes. For this purpose, block 300-1 includes an inverse quantization block 320-1i and an inverse transform block 310-1i. The quantized first residual set is inverse quantized at inverse quantization block 320-1i in encoder 100 and inverse transformed at inverse transform block 310-1i to regenerate the decoder-side form of the first residual set.

[0184] This improved decoder-side format combination of the decoded base stream from decoder 320D and the first residual set (i.e., performing a summation operation 310-C on the decoder-side format of the decoded base stream and the first residual set). The summation operation 310-C generates a reconstruction, as is likely to be performed on the subsampled format of the input video generated at the decoder—i.e., the reconstructed base codec video). Figure 3 and Figure 5 As shown, the reconstructed base codec video is then upsampled using upsampler 305U.

[0185] The upsampled signal (i.e., the reference signal or frame) is then compared with the input signal 30 (i.e., the desired signal or frame) to create a second residual set (i.e., the difference operation 300-S is applied to the recreated stream after upsampling to generate another residual set). The second residual set is then processed at block 300-2 to become an encoded level 2 stream (i.e., the encoding operation is then applied to another or a second residual set to generate another or a second encoded enhanced stream).

[0186] Specifically, the second residual set is transformed (i.e., transformation operation 310-2 is performed on another residual set to generate another transformed residual set). The transformed residuals are then quantized and entropy encoded in the manner described above with respect to the first residual set (i.e., quantization operation 320-2 is applied to the transformed residual set to generate another quantized residual set; and entropy encoding operation 320-2 is applied to the other quantized residual set to generate an encoded level 2 stream containing another level of enhanced information). However, only quantization step 20-1, or only transformation and quantization steps, can be performed. Entropy encoding can optionally be used as a supplement. Preferably, the entropy encoding operation can be a Huffman coding operation or a run-length encoding (RLE) operation, or both.

[0187] Therefore, as Figure 3 and 5 As shown and described above, the output of the encoding process is the base stream at the base layer, and one or more enhancement streams at enhancement layers, preferably including a first enhancement layer and another enhancement layer. As discussed with reference to previous examples, Figure 5 The operations can be applied in parallel to the decoding units or blocks of the color components of a frame because there is no inter-block dependency. Encoding of each color component within the set of color components can also be performed in parallel (e.g., such that the copying is based on (number of frames) * (number of color components) * (number of decoding units per frame)). Figure 5 (The operation). It should also be noted that different color components may have different numbers of decoding units per frame, for example, the luminance (e.g., Y) component may be processed at a high resolution of the colorimetric (e.g., U or V) component set when the change in illuminance is greater than the change in color that can be detected by human vision.

[0188] At decoder 400, the encoded base stream and one or more enhancement streams are received. Figure 6 More detailed display Figure 4 The decoder.

[0189] At base decoder 420, the encoded base stream is decoded to produce a base reconstruction of the input signal 30 received at encoder 300. This base reconstruction can be used in practice to provide a visually reproducible representation of signal 30 at a lower quality level. However, the primary purpose of this base reconstruction is to provide a basis for a higher quality reproduction of the input signal 30. For this purpose, the decoded base stream is provided to processing block 400-1. Processing block 400-1 also receives the encoded level 1 stream and reverses any encoding, quantization, and transform applied by encoder 300. Block 400-1 includes an entropy decoding process 430-1, an inverse quantization process 420-1, and an inverse transform process 410-1. Optionally, only one or more of these steps may be performed depending on the operations performed at the corresponding block 300-1 at encoder. By performing these corresponding steps, the decoded level 1 stream, including a first set of residuals, becomes available at decoder 400. The first residual set is combined with the decoded base stream from the base decoder 420 (i.e., a summation operation 410-C is performed on the decoded base stream and the decoded first residual set to generate a reconstructed, subsampled form of the input video—i.e., the reconstructed base codec video). For example... Figure 4 and Figure 6 As shown, the reconstructed base codec video is then upsampled by the upsampler 405U.

[0190] Furthermore, and optionally in parallel, in Figure 2 At block 400-2, the encoded level 2 stream is processed to produce another set of decoded residuals. Similar to processing block 300-2, processing block 400-2 includes an entropy decoding process 430-2, an inverse quantization process 420-2, and an inverse transform process 410-2. Of course, these operations will correspond to the operations performed at block 300-2 in encoder 300, and one or more of these steps may be omitted as needed. Block 400-2 produces a decoded level 2 stream including another set of residuals, and these residuals are summed at operation 400-C with the output from upsampler 405U to create a level 2 reconstruction of the input signal 30. Level 2 reconstruction can be viewed as, for example... Figure 2 The decoded video output from the 250 level. In some instances, it is also possible to obtain and view the reconstructed video passed to the upsampler 405U - this will have a first enhancement level but may be at a lower resolution than the level 2 reconstruction.

[0191] Therefore, as shown and described above, the output of the decoding process is (optionally) a base reconstruction, as well as a reconstruction of the original signal at a higher level. This example is particularly well-suited for creating encoded and decoded video at different frame resolutions. For instance, the input signal 30 could be an HD video signal comprising frames at a resolution of 1920×1080. In some cases, both the base reconstruction and the Layer 2 reconstruction can be used by the display device. For example, in the case of network traffic, the Layer 2 stream may be interrupted more often than the Layer 1 stream and the base stream (because it may contain up to 4× data, where undersampling reduces the dimension by 2 in each direction). In this case, when traffic occurs, the display device can resume displaying the base reconstruction, while the Layer 2 stream is interrupted (e.g., when the Layer 2 reconstruction is unavailable), and then resume displaying the Layer 2 reconstruction when network conditions improve. A similar approach can be applied when the decoding device is under resource constraints; for example, a set-top box performing a system update may have an operating base decoder 220 to output the base reconstruction, but may not have the processing capacity to compute the Layer 2 reconstruction.

[0192] The encoding arrangement also allows the video distributor to distribute video to a set of heterogeneous devices; those devices with only the basic decoder 220 examine the basic reconstruction, while those with enhancement layers examine the higher-quality layer 2 reconstruction. In the comparative case, two complete video streams at separate resolutions are needed to serve the two sets of devices. Because the layer 2 and layer 1 enhancement streams encode residual data, the layer 2 and layer 1 enhancement streams can be encoded more efficiently, for example, the distribution of residual data is typically mostly around 0 quality (i.e., no difference) and usually takes a small range of values ​​around 0. This is especially true after quantization. In contrast, the complete video streams at different resolutions will have different distributions of non-zero mean or median, which require higher bit rates to transmit to the decoder.

[0193] In some instances, residuals can be viewed as errors or differences at a specific quality level or resolution. In the described instance, there are two quality levels or resolutions and therefore two sets of residuals (L-1 and L-2). Each set of residuals described herein models a different form of error or difference. For example, L-1 residuals typically correct for characteristics of the base encoder, such as artifacts introduced by the base encoder as part of the encoding process. In contrast, L-2 residuals typically correct for the combined effects introduced by changes in quality levels and the differences introduced by L-1 correction (e.g., artifacts generated by the L-1 encoding pipeline at a wider spatial scale, such as a region of 4 or 16 pixels). This means that the following is not self-evident: an operation performed on one set of residuals will necessarily provide the same effect to the other set of residuals; for example, each set of residuals may have different statistical patterns and correlation sets.

[0194] In the example described in this paper, the residuals are encoded by an encoding pipeline. This may include transform, quantization, and entropy encoding operations. It may also include residual grading, weighting, and filtering. These pipelines are shown in... Figure 1 The residual is then transmitted to the decoder, for example as L-1 and L-2 enhancement streams, which can be combined with the base stream as a hybrid stream (or transmitted separately). In one case, a bit rate is set for the hybrid data stream comprising the base stream and the two enhancement streams, and then different adaptive bit rates are applied to individual streams based on the data being processed to meet the set bit rate (e.g., high-quality video perceived with low artifact levels can be constructed by adaptively assigning bit rates to different individual streams (even at the frame-by-frame level) so that constrained data can be used by the individual stream that is most perceptibly influential, which may change as the image data changes).

[0195] The residual set described in this paper can be considered sparse data, for example, in many cases there is no difference for a given pixel or region, and the resulting residual value is zero. When looking at the distribution of the residuals, many probability masses are assigned to small residual values ​​located close to zero, such as for some video values ​​of -2, -1, 0, 1, 2, etc., which occur most frequently. In some cases, the distribution of residual values ​​is symmetrical or approximately symmetrical about 0. In some test video cases, the distribution of residual values ​​is found to have a shape similar to a logarithmic or exponential distribution about 0 (e.g., symmetrical or approximately symmetrical). The exact distribution of the residual values ​​can depend on the content of the input video stream.

[0196] The residual can be viewed itself as a two-dimensional image, such as a difference image of the difference. In this way, the sparsity of the data can be seen to involve features visible in the residual image, such as “points,” small “lines,” “edges,” and “corners.” These features have been found to be generally not perfectly correlated (e.g., spatially and / or temporally). These features have properties that differ from those of the image data from which they originate (e.g., the pixel characteristics of the original video signal).

[0197] Because the characteristics of residuals differ from those of the image data from which they originate, it is generally impossible to apply standard coding methods, such as those found in the traditional Moving Picture Experts Group (MPEG) coding and decoding standards. For example, many contrast schemes use large transforms (e.g., transforms of large pixel regions in a normal video frame). Due to the characteristics of residuals, such as those described above, using these large transforms for residual images would be extremely inefficient. For example, encoding small points in a residual image using large blocks of regions designed for normal images would be very difficult.

[0198] Some of the examples described in this paper address these issues by alternatively using smaller and simpler transform kernels (e.g., 2×2 or 4×4 kernels – directed decomposition and directed decomposition squared, as presented in this paper). The transforms described in this paper can be applied using Hadamard matrices (e.g., 4×4 matrices for flattening 2×2 decoded blocks, or 16×16 matrices for flattening 4×4 decoded blocks). This shifts in a different direction from the contrasting video coding methods. Applying these new methods to residual blocks yields compression efficiencies. For example, some transforms generate uncorrelated coefficients (e.g., in space) that can be efficiently compressed. While correlations between coefficients can be utilized, for example, for lines in the residual image, these correlations can lead to coding complexity, making implementation difficult on conventional and low-resource devices, and these correlations often generate other complex artifacts that require correction. Preprocessing residuals by setting certain residual values ​​to 0 (i.e., not forwarding these residual values ​​for processing) provides a controllable and flexible way to manage bit rates and stream bandwidth, as well as resource usage.

[0199] For the sake of completeness, Figure 7 The broad principles of the concepts described in this article are illustrated in the form of flowcharts. Method 1000 includes: receiving an input video (step 1001); downsampling the input video to create a downsampled video (step 1002); instructing the encoding of the downsampled video using a base encoder to create a base encoded stream (step 1003); instructing the decoding of the base encoded stream using a base decoder to generate a reconstructed video (step 1004); comparing the reconstructed video with the downsampled video to create a first residual set (step 1005); and encoding the first residual set to create a first-level encoded stream, including: applying a transform to the first residual set to create a first set of coefficients (step 1006); applying a quantization operation to the first set of coefficients to create a first set of quantized coefficients (step 1007); and applying an encoding operation to the first set of quantized coefficients (step 1008), wherein applying the quantization operation includes: adapting the quantization based on the first set of coefficients to be quantized, including changing the step width for different coefficients in the first set of coefficients, wherein a first set of parameters derived from the adaptation is transmitted to the decoder to achieve dequantization of the first set of quantized coefficients.

[0200] Figure 8A Provides examples of how to perform quantization of residuals and / or coefficients (transformed residuals) based on block groups with defined step widths. Figure 8A The x-axis represents the residual or coefficient value. In this example, the number of block groups is limited by a step width of 5. The step width can be understood as the quantization step size, as shown in the diagram. The size of the step width can be selected, for example, based on parameter values. In some cases, the size of the step width can be set dynamically, for example, based on the rate control example described above. Figure 8AIn this example, the step width generates blocks corresponding to residual values ​​in the ranges of 0-4, 5-9, 10-14, and 15-19 (i.e., 0 to 4 includes both 0 and 4). The block width can be configured to include or exclude endpoints as needed. In this example, quantization is performed by replacing all values ​​falling within the block with integer values ​​(e.g., residual values ​​between 0 and 4 (including endpoints) have a quantization value of 1). Figure 8A In quantization, the following operations can be performed: divide by the step size (e.g., 5), take the base of the result (i.e., for positive values, the nearest integer less than a certain decimal), and then add one (e.g., 3 / 5 = 0.6, floor(0.6) = 0, 0 + 1 = 1; or 16 / 5 = 3.2, floor(3.2) = 3, 3 + 1 = 4). Negative values ​​can be handled in a similar way, for example, by applying it to the absolute value and then converting it to a negative value after calculation (e.g., abs(-9) = 9, 9 / 5 = 1.8, floor(1.8) = 1, 1 + 1 = 2, 2 * -1 = -2). Figure 8A This example demonstrates a linear quantization scenario where all block groups share a common step width. It should be noted that various different implementations based on this approach can be developed; for example, the first block group may have a quantization value of 0 instead of 1, or may include values ​​from 1 to 5 (including the endpoints). Figure 8A This is merely an illustration of quantization based on a block group with a given step width.

[0201] Figure 8B This demonstrates how the so-called "dead zone" (DZ) can be implemented. This can be understood as the area near the zero output value of the quantizer; that is, a band containing zero signals and whose size can be the same as or different from the step width. Therefore, for this band of input close to zero, the signal can be effectively attenuated, so that low-level signals, which typically correspond to noise in visual data, are not unnecessarily allocated to the data. Figure 8B In this context, residuals or coefficients with values ​​within a predefined range are set to 0. Figure 8B In this context, the predefined range is the range around the value 0. Figure 8B In this configuration, values ​​less than 6 and greater than -6 are set to 0. The dead zone can be set to a fixed range (e.g., -6 to 6) or based on the step size. In one case, the dead zone can be set to multiple predefined step sizes, for example, as a linear function of the step size values. Figure 8B In this example, the dead zone is set to 2.4 times the step width. Therefore, with a step width of 5, the dead zone extends from -6 to +6. In other cases, the dead zone can be set as a non-linear function of the step width value.

[0202] In one case, the dead time is set based on a dynamic step width, which can be adaptive, for example. In this case, the dead time can change with the step width. For example, if the step width is updated to 3 instead of 5, the dead time of 2.4 * step width can change from the range -6 to +6 to the range -3.6 to 3.6; or if the step width is updated to 10, the dead time can change to extend from -12 to 12. In one case, the step width multiplier can be between 2 and 4. In another case, the multiplier can also be adaptive, for example based on operating conditions such as the available bit rate.

[0203] Having a dead time helps reduce the amount of data to be transmitted over the network, for example, by helping to reduce the bit rate. When using a dead time, residual or coefficient values ​​that fall within the dead time are effectively ignored. This method also helps remove low-level residual noise. Having an adaptive rather than constant dead time means that smaller residual or coefficient values ​​are not over-filtered when the step size decreases (e.g., if more bandwidth is available), and the bit rate decreases appropriately if the step size increases. The dead time only needs to be specified at the encoder; the decoder simply receives a quantized value of 0 for any residual or coefficient that falls within the dead time.

[0204] Figure 8C This demonstrates how a method called block group folding can be applied. Figure 8C In some instances, block group folding is used in conjunction with dead zones, but in others, block group folding can be used without dead zones and / or in conjunction with other quantization methods. Figure 8C In this context, block folding is used to place all residual or coefficient values ​​residing above a selected quantization block into the selected block. This can be viewed as a form of quantization limiting.

[0205] Figure 8C In the next step, a step width of 5 is applied again. A dead zone with a range of 2.4 * step width is also applied, such that values ​​between -6 and 6 are set to 0. This can also be seen as following into the larger first quantization block group (with a value of 0). Then, two quantization block groups with a width of 5 are defined for positive and negative values. For example, the block group with quantization value 1 is defined between 6 and 11 (e.g., with a step width of 5), and the block group with quantization value 2 is defined between 11 and 16. In this example, to perform block group folding, all residuals or coefficients with values ​​that would normally fall above the second block group (e.g., with a value greater than 16) are "folded" into the second block group, for example, limited to have a quantization value of 2. This can be done by setting all values ​​greater than a threshold to the maximum block group value (e.g., 2). A similar process occurs for negative values. Figure 8C The large arrow indicates this.

[0206] Block group folding can be an optional processing option at the encoder. It does not need to be specified during dequantization at the decoder (e.g., a value of 2 that has been "folded" or "limited" is simply dequantized as if it were in a second block group). Block group folding can be performed to reduce the number of bits sent to the decoder via the network. Block group folding can be configured based on network conditions and / or underlying streaming processing to reduce the bit rate.

[0207] Figure 8D This demonstrates how quantization offsets can be used in specific situations. Quantization offsets can be used to shift the position of quantized block groups. Figure 8D The diagram shows a line indicating the possible real-world counts of the residual or coefficient values ​​along the x-axis. In this example, many values ​​are close to zero, with the count of higher values ​​decreasing as we move away from 0. If the counts are normalized, the line can also indicate the probability distribution of the residual or coefficient values.

[0208] Figure 8D The left-hand side bars and the dashed lines on the right-hand side illustrate the histogram modeling the quantization. For clarity, the counts of the first to third blocks after the dead zone are shown (for both positive and negative values, the latter are striped to represent bars). For example, the bars show the counts of quantized values ​​1, 2, 3 and -1, -2, -3. Due to quantization, the distribution modeled by the histogram differs from the actual distribution shown by the lines. For example, the error 'e' is shown, illustrating the degree to which the bars differ from the lines.

[0209] To modify the nature of the error e, a quantization offset qO can be applied. The quantization offset can be understood as a parameter whose value determines whether and to what extent the quantization interval or block group will be shifted from a predetermined or default position or set of values. For positive values, a positive quantization offset is used to shift each block group to the right, and a negative quantization offset is used to shift each block group to the left. The quantization offset can be applied in combination with a dead zone. In one case, the dead zone can be applied based on a first set of thresholds, for example, all values ​​less than (n * step width) / 2 and greater than (n * step width * -1) / 2 are set to 0.

[0210] In some instances, the quantization offset can be signaled to the decoder for use during dequantization.

[0211] In one case, at the encoder, the quantization offset can be subtracted from the residual or coefficient value before step-width-based quantization. Therefore, in the decoder, the transmitted offset can be added to the received quantized value for step-width-based dequantization. In some cases, the offset can be adjusted based on the sign of the residual or coefficient to allow symmetric operation about 0 values. In one case, the use of the offset can be disabled by setting the quantization or dequantization offset value to 0. In one case, the applied quantization offset can be adjusted based on a defined dead-time width. In one case, the dead-time width can be calculated at the decoder, for example, based on the step width and quantization parameters received from the encoder.

[0212] In one scenario, the step width used for quantization can vary for different coefficients within a 2×2 or 4×4 coefficient block. For example, a smaller step width can be assigned to coefficients that are experimentally determined to have a greater impact on the perception of the decoded signal. For instance, in a 2×2 or 4×4 directional decomposition (DD-squared or "DDS") as described above, smaller step widths can be assigned to the A, H, V, and D coefficients, with larger step widths assigned to later coefficients. In this case, the base_stepwidth parameter can be used to define the default step width, and a modifier can then be applied to this parameter to calculate modified_stepwidth for use in quantization (and dequantization), for example, modified_stepwidth = base_stepwidth * modifier (where the "modifier" can be set based on specific coefficients within a block or cell and can be derived from signaling such as the variable "qm" described below).

[0213] In some cases, the modifier may additionally or alternatively depend on the enhancement level. For example, for a level 1 enhancement stream, the step size can be smaller because it can affect multiple reconstructed pixels at a higher quality level.

[0214] In some cases, modifiers can be defined based on both the coefficients within a block and the enhancement level. In one case, the quantization matrix can be defined with a set of modifiers for different coefficients and different enhancement levels. This quantization matrix can be preset (e.g., at the encoder and / or decoder), signaled between the encoder and decoder, and / or dynamically constructed at the encoder and / or decoder. For example, in the latter case, the quantization matrix can be constructed at the encoder and / or decoder based on other stored and / or signaled parameters, such as parameters received via a configuration interface.

[0215] In one scenario, different quantization modes can be defined, or different schemes of quantization matrices to be applied to a given set of coefficients can be defined. In one mode, a common quantization matrix can be used for two enhancement levels; in another mode, separate matrices can be used for different levels; and in yet another mode, a quantization matrix can be used for only one enhancement level, such as only for level 2. The quantization matrix can be indexed by the position of the coefficients within the block (e.g., 0 or 1 along the x-direction and 0 or 1 along the y-direction for a 2×2 block, or 0 to 3 for a 4×4 block).

[0216] In one case, the set of values ​​can define the underlying quantization matrix. This underlying quantization matrix can be modified by a scaling factor, which is a function of the stride width of one or more enhancement layers. In another case, the scaling factor can be a clamping function of the stride width variable. At the decoder, the stride width variable can be received from the encoder for one or more of the layer 2 and layer 1 streams.

[0217] With the aid of other examples of processes involving the principles described above, advantageous patterns for configuring and adapting quantization to further improve encoding and decoding procedures can be understood by further examining the following text. Data block units can be applied to enhance payload semantics, involving several parameters that can be used to signal and configure the properties of the quantization and dequantization steps.

[0218] In one example, the parameter `dequant_offset_signalled` specifies whether the offset parameter value will be applied during signal dequantization. This allows you to determine whether the signal should send an offset. In this example, if an offset is sent, it is used. If it is not sent, the default offset can be used, or no offset may be used.

[0219] In an example of a method for encoding an input video into multiple encoded streams, wherein the encoded streams can be combined to reconstruct the input video, the method may involve receiving the input video and downsampling the input video to create an downsampled video.

[0220] The method typically further includes instructing the encoding of the downsampled video using a base encoder to create a base encoded stream; instructing the decoding of the base encoded stream using a base decoder to generate a reconstructed video; comparing the reconstructed video with the downsampled video to create a first set of residuals; and encoding the first set of residuals to create a first-level encoded stream. This preferably includes: applying a transform to the residual set to create a set of coefficients; applying a quantization operation to the set of coefficients to create a set of quantized coefficients; and applying an encoding operation to the quantized coefficients, wherein the quantization operation is performed using a quantization matrix derived from obtained values ​​of quantization matrix mode parameters.

[0221] As previously described in this disclosure, the quantization matrix mode parameter can be advantageously used to specify the quantization matrix to be used in the encoding process. In some instances, when the quantization matrix mode parameter value is equal to a predetermined value, such as when it is equal to zero, the method may involve using a default quantization matrix for each of two quality levels. These levels are typically or generally correspond to level 1 and level 2 enhancement streams. When the quantization matrix mode parameter value is equal to 1, a first quantization matrix can be used for each of the two quality levels, and the first quantization matrix can be signaled, for example, from the encoder to the decoder or to a device to which the encoded stream will be transmitted. When the quantization matrix mode parameter value is equal to 2, a second quantization matrix can be used for quality level 2, and the second quantization matrix can be signaled. In this case, a quantization matrix is ​​not used for quality level 1, or a default value can be used for this level. When the quantization matrix mode parameter value is equal to 3, a third quantization matrix is ​​preferably used for quality level 1 or the first level encoded stream, and the third quantization matrix is ​​signaled. In this case, a quantization matrix is ​​not used for quality level 2, or a default value can be used for this level. When the quantization matrix mode parameter value is equal to 4, a fourth quantization matrix can be used for the first-level encoded stream, and a fifth quantization matrix can be used for the second-level encoded stream (for example, two matrices can be used), each of which can be equal to or unequal to each other, and equal to or not equal to any of the aforementioned first to third matrices. In this fifth mode, the fourth and fifth quantization matrices can be transmitted to the decoder or other devices to which the encoded stream will be transmitted.

[0222] In the procedures described in this disclosure, as mentioned above, each group of transform coefficients passed to this process typically belongs to a specific plane and layer. Typically, they have been scaled using a linear quantizer, which in some instances uses a non-centered dequantization offset. A scaling procedure can be applied to the transform coefficients as follows. This procedure can acquire the block's location information, along with a set of parameters that can indicate the block's properties (e.g., its size), the nature of the quantization operation (e.g., stride and offset values), and the enhancement layer to which it is applied. For example, the location can be indicated by a pair of coordinate values ​​or, for example, parameters (xTbP, yTbP) specifying the top-left sample of the current luma or chroma transform block relative to the top-left luma or chroma sample of the current image. This can be associated with a specific portion of the data representing the image and, for example, with the luma plane or chroma plane, depending on the plane to which the transform coefficients belong.

[0223] The aforementioned parameter specifying the size of the current transform block (in some instances, it may be called nTbS) may have a value that depends on the type of transform, and more specifically, on the value of the parameter defining the transform used for decoding. This type parameter may be called transform_type in some instances and may have values ​​of 0, 1, or 2-3 in some applications, corresponding to a 2×2 directional decomposition transform, a 4×4 directional decomposition transform, or a value or parameter specifying zero (because those elements are not used in the bitstream). In some instances, this transform_type parameter with a value of 0 may correspond to a layer number equal to 4, and if transform_type equals 1, the corresponding layer number value may be 16. The size nTbS parameter may have a value of 2 when transform_type equals zero, and a value of 4 when transform_type equals 1.

[0224] The other input to the process is typically in the form of an array of entropy-decoded quantized coefficients. This may be called TransCoeffQ and has a size related to the size parameter mentioned above, specifically, a size of (nTbS) x (nTbS), meaning the array may include decoding units or blocks as described herein. This array may be called TransCoeffQ. The step width value can be specified using a parameter called stepWidth. The index of the enhancement level can be specified by a parameter, and in some instances may be called idxLevel. If a dequantization offset will be used, this can be specified using a parameter, for example, called dQuantOffset. This parameter typically specifies the value of the dequantization offset parameter to be applied. This process typically produces an array of dequantized transform coefficients. This may, for example, be in the form of an array of size (nTbS) x (nTbS) containing elements d[x][y].

[0225] The values ​​in the output array, denoted as d[x][y], are typically derived from either the step size described above, or the index of the enhancement level applied to the quantization matrix (which may be denoted as qm), and the quantized coefficients of the entropy decoding (which may be denoted as TransCoeffQ[xTbP][yTbP] of the corresponding feature). Furthermore, the values ​​of the output array features can be derived additionally by applying an offset to the result of the aforementioned operations. For example, a parameter, denoted as appliedOffset, can be added to the aforementioned value to produce the output feature values.

[0226] Therefore, this can involve the following relationship to calculate each element d of the array:

[0227] d[ x ][ y ]=(TransCoeffQ[ xTbP ][ yTbP ]*(stepWidth * qm[yTbP +(idxLevel * 4)][xTbP]))+ appliedOffset

[0228] In the above calculation, which serves as an example of how dequantization can be performed at the decoder, idxLevel can be 0 or 1, representing level 2 and level 1, respectively. In the above calculation, and as described in other examples herein, the value from the quantization matrix is ​​selected based on specific coefficients and a specific enhancement level.

[0229] Relative to the offset that can be applied in some instances, as shown in the relationships above, this parameter, which may be called appliedOffset, is typically derived as follows.

[0230] In some cases, appliedOffset can be conditionally applied based on whether residual filtering will be applied. This filtering can be applied in the inverse transform (e.g., Figure 6 The offset is applied after 410-1 in the above, and can be a deblocking filter. In these cases, the offset can be applied only if the residual filter is applied. For example, if the parameter, which may be called deblocking_signalled, that specifies whether a deblocking filter should be applied has a specific value, such as equal to 1, then the offset is calculated based on the aforementioned parameter that specifies the dequantization offset, which may be called dQuantOffset.

[0231] The appliedOffset can also be derived from a single value of dQuantOffset and is suitable for application to both positive and negative coefficient values. For example, appliedOffset can be configured to be negative or positive, typically having the same absolute value as the dequantization offset parameter and a positive or negative sign, respectively. This can be applied if the TransCoeffQ value is less than zero; otherwise, if the TransCoeffQ value is greater than zero, a value equal to the dequantization offset parameter can be assigned to the applied offset. If the TransCoeffQ value is zero, zero can also be assigned to the applied offset.

[0232] Therefore, in some instances, the offsets of the output array elements that can be summed can be derived using the following algorithm (where TransCoeffQ can be an array of entropy-decoded quantized coefficients of size (nTbS)x(nTbS):

[0233] If deblocking_signalled equals 1

[0234] If TransCoeffQ[ xTbP ][ yTbP ] < 0

[0235] appliedOffset = (dQuantOffset * -1)

[0236] else If TransCoeffQ [ xTbP ] [ yTbP ] > 0

[0237] appliedOffset = dQuantOffset

[0238] else

[0239] appliedOffset = 0

[0240] else

[0241] appliedOffset = 0

[0242] For example, a parameter or set of parameters can be signaled as described above and used to specify how the quantization matrix will be applied to the set of coefficients. This parameter can be the `quant_matrix_mode` parameter, which specifies which quantization matrices will be used in the decoding process according to Table 1 below. The parameter can be one byte. The `quant_matrix_mode` parameter can be used to configure how the quantization matrix is ​​derived (for example, as described later below).

[0243] Table 1 - Quantization Matrix

[0244]

[0245] The quantization matrix data semantics can be applied based on the following specific instance. Derive a quantization matrix that can be called qm[y][x].

[0246] The matrix qm can have a size of k*M*N. That is, the matrix can be limited to the quantization coefficients contained in the M*N matrix, and qm can contain this data in the form of, for example, the corresponding M*N matrix of the quantization coefficients of each of the k enhancement levels.

[0247] In the current instance, two enhancement levels are involved, namely levels 1 and 2 as described in this disclosure, and k equals 2. The maximum transformation described with respect to these procedures is 4×4, and therefore M and N can each be equal to 4.

[0248] In some instances, by limiting qm with M and N values ​​corresponding to the largest available transformation (which is a 4×4 transformation as mentioned above, thus including 16 coefficients), the application of a quantization matrix employing a smaller transformation can be performed by reading from qm and applying a subset of the coefficients in the corresponding M×N matrix.

[0249] In a specific instance, if the value of the quantization matrix mode parameter is equal to zero, i.e., if quant_matrix_mode equals 0, then the following default quantization matrix is ​​used in the quantization operation:

[0250] qm[ y][ x ] =

[0251] { 0.500 0.500 0.500 0.617}

[0252] { 0.862 0.610 1.064 0.781}

[0253] { 0.500 0.500 0.500 0.617}

[0254] { 3.125 1.851 1.851 1.316}

[0255] { 0.500 0.500 0.500 0.617}

[0256] { 0.862 0.610 1.064 0.781}

[0257] { 0.862 1.064 0.610 0.781}

[0258] { 3.125 1.851 1.851 1.316},

[0259] Where y = 0..3 are the coefficients to be used for enhancement level 2, and y = 4..7 are the coefficients to be used for enhancement level 1; and

[0260] If the quantization matrix mode parameter is equal to 1, a matrix of modifiers is transmitted and used at both enhancement levels. In some applications, the matrix can be constructed using an iterative process. Thus, the qm[x][y] values ​​can be obtained row by row, whereby for each row, a certain value is present, and each column of the matrix is ​​filled as the row is processed. The step width modifier parameter can be used in the process of obtaining these matrix values, as described below. In the current example, the number of rows to be filled can be 8, and the number of columns can be equal to 4, as described above in conjunction with the matrix size in this specific example. Specifically, the iterative process for this example can be written as:

[0261] for (y = 0; y < 8; y++)

[0262] for (x = 0; x < 4; x++)

[0263] qm[y][x] = step_width_modifier_2[x + y*4],

[0264] In this example, the step width modifier parameter, which can be called step_width_modifier_2, specifies the values ​​of the 16 level 2 enhancement coefficients to be applied at different levels of the transform coefficients; and

[0265] If the value of the quantization matrix mode parameter is equal to 2, then a matrix of the modifier is transmitted and used at enhancement level 2. Similarly, more precisely, an iterative procedure corresponding to the iterative procedure described above can be used:

[0266] for (y = 0; y < 4; y++)

[0267] for (x = 0; x < 4; x < 4)

[0268] qm[ y][ x ] = step_width_modifier_2[x + y*4]; and

[0269] If the value of the quantization matrix mode parameter is equal to 3, then a matrix of the modifier is transmitted and used on enhancement level 1:

[0270] for (y = 0; y < 4; y++)

[0271] for (x = 0; x < 4; x < 4)

[0272] qm[y + 4][ x ] = step_width_modifier_2[x + y*4]; and

[0273] If the quantization matrix mode parameter is equal to 4, then the two matrices of the modifier are transmitted, the first for enhancing level 2 and the second for enhancing level 1:

[0274] for (y = 0; y < 4; y++)

[0275] for (x = 0; x < 4; x < 4)

[0276] qm[ y][ x ] = step_width_modifier_2[x + y*4]

[0277] for (y = 0; y < 4; y++)

[0278] for (x = 0; x < 4; x < 4)

[0279] qm[ y][ x ] = step_width_modifier_1[x + y*4],

[0280] The step_width_modifier_1 parameter specifies the level 1 enhancement values ​​of the 16 coefficients to be applied at different levels of the transformation coefficients.

[0281] As described above, the value of d[x][y] in the matrix can be calculated, for example, as the product of the corresponding transformation coefficient at the relevant element in the matrix and the sum of the corresponding quantized matrix element values ​​at the column, which is identified by the product of a parameter specifying the size of the current transform block and the levelIdxSwap parameter, and in the corresponding row, the step width modifier parameter value corresponds to the element, wherein the offset described above is also typically applied additively to the product.

[0282] The dequantization process described above can be performed according to the following ordered steps. The dequantization process according to the aforementioned example can be invoked, with the luminance position (xTbY, yTbY), a transform size set to nTbS (i.e., the size of the current transform block), an array TransCoeffQ of size (nTbS) x (nTbS) as described above, and a step size parameter as input. The output can therefore be an array of dequantization coefficients (e.g., the dequantized transform residuals) of size (nTbS) x (nTbS), which in this example may be referred to as dequantCoeff.

[0283] Another example of an approach employing asymmetric quantization is now described. For simplicity, the examples described typically employ the use case of video sequences, i.e., time-based signals consisting of sequences of 2D images (often referred to as “frames” or “fields” in the case of interleaved video signals), where each element (often referred to as a “pixel” in this non-limiting example) is characterized as a set of color settings in a suitable color space (e.g., YUV, RGB, HSV, etc.). Different color planes (e.g., the luminance-Y plane and the two chrominance-U and V planes) are often encoded separately and often at different resolutions (due to the low sensitivity of the human eye to chrominance information).

[0284] In other cases, the signal is represented as a sequence of N-dimensional samples, and instances refer to the fact that a signal can be fully represented as an (N+1)-dimensional signal (e.g., if one dimension is time, this corresponds to a sequence representing a spatial reproduction with a single spatiotemporal reproduction). These will be considered as non-limiting examples of possible kinds of signals that can be filtered and / or compressed using the innovative methods described herein. For signals other than images or videos, those skilled in the art can readily apply the methods described herein by appropriately adapting them to the use cases described for image and video signals. In non-limiting instances, the time samples of the signal can also be hyperplanes with a different number of dimensional elements than 2 (e.g., one-dimensional time samples, three-dimensional time samples, etc.), and / or dimensions that may be applied differently from the time methods described herein for the time dimension.

[0285] Each sample of the signal in time is represented by a hyperplane (or more simply, a “plane,” which in its broadest sense is defined as “a collection of elements organized as an array of one or more dimensions”): for example, a multi-channel audio signal, a 2D HD video frame, or a 3D volumetric medical image can all be represented by an array of planar elements (more precisely, a 1D plane of elements for multi-channel audio signals, a 2D plane of elements for HD video frames, and a 3D hyperplane of elements for volumetric medical images).

[0286] Along time, samples occur at a given (local) sample rate. In conventional methods, the sample rate is always constant. In contrast, the innovative method described in this paper also allows for a variable sample rate. Specifically, for quality levels below the top level, the duration of a time sample at a given quality level (i.e., the sample rate at that given quality level) can be variable, thus effectively representing a variable sample rate. To allow for appropriate signal replay with time samples of different durations, each time sample is also characterized as a "picture number" or "picture identifier," indicating the time at which the time sample should begin to be displayed.

[0287] The methods and embodiments described herein can be used in combination with each other and / or with other methods. The techniques and algorithms described in the many preferred embodiments herein aim to achieve compression, i.e., to encode and / or decode a suitable reproduction of a signal with a minimum number of bits. This is also a non-limiting example: other embodiments can achieve different purposes, such as multi-scale encoding and decoding, functional degradation and recovery, adaptive streaming, robust and efficient filtering, signal denoising (e.g., image denoising, video denoising, etc.), signal enhancement (e.g., signal supersampling, deinterlacing, etc.), generation of signal quality metrics, content recognition, machine vision, signal encryption (e.g., secure communication), etc.

[0288] As mentioned above, quantization at the encoder and dequantization at the decoder can be performed asymmetrically. That is, the quantization or dequantization steps proposed in this disclosure can be performed differently or according to different or modified parameters. These solutions provide specific practical tools to alter the detail in the reconstructed data and provide more detail in specific regions of the statistical distribution. Specifically, this asymmetry provides more accurate quantization of the information responsible for the detail in the reconstructed dataset (e.g., where the encoded data is residual data, which is combined with image data to reconstruct the image, the residual data responsible for the detail at the edges can be more accurately dequantized).

[0289] This asymmetric quantization can have specific utility tools, such as those found in the hierarchical decoding schemes described earlier in this disclosure. In some instances, the decoding process can be performed on both the encoder and decoder sides of the reconstruction path when the encoded information is reconstructed. In other instances, parameters are signaled through the encoder process to the decoder process to indicate or facilitate asymmetric modifications to the dequantization process. In a particular instance where the encoder quantizes with a specific stride, the corresponding decoding can utilize this stride and be modified according to the signaled parameters received by the decoder from the encoder.

[0290] As mentioned, the quantization or dequantization steps proposed in this disclosure can be performed in different ways or with different or modified parameters. Specifically, the dequantization offset or stride modifier proposed above can be provided only at the decoder, such that the offset and / or modifier are not applied during encoding. For example, the stride can be used in the encoded quantization and is not modified according to the examples provided in the above disclosure.

[0291] The following section provides an explanation of asymmetric quantification and a set of non-restrictive examples.

[0292] In one instance, asymmetry means that the encoder quantizes with a stepwidth (SWenc), and the decoder dequantizes with a slightly different stepwidth (SWenc + SWmodifier). SWmodifier is not fixed, but adaptively depends on SWenc, at least in part on a transmissible parameter (or in part on the default value of said parameter), depending on the nonlinear function. Examples of stepwidth modifiers are illustrated in the accompanying drawings.

[0293] In this specific implementation of the example, each orientation step width (calculated based on the main step width according to the adaptive quantization matrix logic) becomes a different modifier and can be modulated according to the optional transmitted signal parameters. Asymmetric dequantization allows for statistically "undershooting" of the dequantization of coefficients closer to the center (zero) of the Laplace distribution in the bucket, while more dequantization occurs towards the center and even the very tail of the bucket. That is, the mosquito noise risk of the transition between dead zones and + / -1 buckets is lower, and the dequantization of residuals responsible for sharper / higher contrast edges is more accurate.

[0294] The possibility of the encoder "adjusting" the knob of the adaptive SWmodifier allows the encoder to reconstruct the critical residuals more accurately. Encoders based on the principles proposed in this paper also have the possibility of another instance mode of signaling. In this mode, the dequantization step width is symmetric to the encoding and uses a dequantization offset signaled by the encoder. However, things are not so simple in this case: the dequantization offset can be referenced to the main step width setting and can be adaptively modulated based on each directional step width derived from the main step width. That is, each step width in the quantization matrix can obtain a different offset, and the modulation operates in a manner that follows the assumption of a Laplace distribution (i.e., an exponential decrease in the residual distribution). Instance encoders based on the principles proposed in this paper have many ways to advantageously utilize this feature. For example, the encoder can ignore this feature and use the default quantization matrix and default dequantization offset / SWmodifier derived, or the encoder can be slightly smarter and try different signaled dequantization logics during encoding, ultimately selecting the dequantization logic that produces the best metrics (e.g., the basis for MSE, PSNR). This is because quantization can remain the same, and the encoder can easily try dequantization according to different alternatives. A simple binary quick lookup can quickly converge to the optimal mode and / or optimal transmission parameters for the frame. (i.e., simple distortion optimization at the same rate, DO).

[0295] If the encoder also begins to employ a quantization matrix, these methods can optionally be made more complex in some non-limiting instances. In this case, the encoder can signal the quantization matrix to the decoding process based on the statistics of the residuals. Then, the optimizations mentioned above are performed to further improve the accuracy of the reconstruction. Another prudent alternative is that, instead of signaling different dequantization parameters, the encoder can modulate the quantization step width to be optimal for a given default dequantization method. That is, in this instance, the encoder can encode several times with slightly different step widths, selecting a step width that optimizes the trade-off between rate and distortion. In some implementations, it is proposed that the combination of a custom quantization matrix and custom dequantization parameters can enhance compression by approximately 5-10% or more compared to a good average default value.

[0296] Further, after selecting the optimal average dequantization parameters for the signal to be transmitted, the encoder might want to change the values ​​of some "unlucky" coefficients that are far from their buckets in the dequantization process. For example, if the instance encoder is quantizing with Stepwidth=1,000, DeadZone=2.4 x Stepwidth (i.e., deadZoneOffset=0.2 x Stepwidth), and for frame averaging, the good dequantization point for bucket 10 is 10% (i.e., 10 x Stepwidth + deadZoneOffset + 10% x Stepwidth) and a coefficient is at 98% of the bucket, then the encoder might want to decide to gradually push that coefficient into bucket 11 so that it will be dequantized closer to its original value.

[0297] Another proposed non-restrictive instance procedure allows the dead zone to be adaptive with respect to the coefficient Stepwidth (i.e., the width of the dead zone (bucket 0)) and depends on Stepwidth based on a certain formula. Interestingly, the asymmetric solution quantization is entangled with the dead zone, since the dead zone becomes dependent on (Stepwidth + SWmodifier).

[0298] Examples of related processes can be summarized by the following statement:

[0299] ● Asymmetric solution quantization means that the solution quantization point can be different for each bucket, with the central bucket having a "conservative" solution quantization point and the tail buckets having solution quantization points that gradually move away from the center.

[0300] ● Dequantization of coefficients with larger stepwidths occurs differently than dequantization of coefficients with smaller stepwidths. That is, the quantization matrix also affects the dequantization modifier.

[0301] ● How the encoder can dequantize (mode 0 or 1) and how this affects the adaptive formula.

[0302] ●It is all closely linked to the quantization matrix.

[0303] Knowing the dequantization method to be used by the decoder, the encoder can further optimize the reconstruction of these coefficients by gradually pushing some coefficients into adjacent buckets using dequantization points that are closer to the original values ​​of the coefficients.

[0304] Example Implementation Plan

[0305] For a specific instance, the scaled transformation coefficients can be given as d[x][y] and determined according to the following formula:

[0306] d[x][y] = (TransformCoeffQ[x][y] * ((qm[x + (levelIdxSwap * nTbS)][y]+ stepWidthModifier[x][y]) + appliedOffset [x][y])

[0307] in:

[0308] TransformCoeffQ[x][y] represents an array of coefficients;

[0309] qm[x][y] represents the quantization matrix of quantization parameters or step width;

[0310] stepWidthModifier[x][y] represents the array of modified parameters; and,

[0311] appliedOffset[x][y] is an array that serves as the dead zone as described elsewhere in this document. That is, in some communication scenarios, the dequantization offset can be used to change the dead zone or can be 0.

[0312] If flag signaling does not use dequantization offset, the stride modifier can be a function of the stride, for example:

[0313] stepWidthModifier [x][y] = ((((Floor(−Cconst * Ln (qm[x +(levelIdxSwap * nTbS)][y]))) + Dconst) * (qm[x + (levelIdxSwap * nTbS)][y] 2 ))) / 32768) >> 16

[0314] If flag signaling uses dequantized offsets, the step size modifier can be based on a set of offset values, for example:

[0315] stepWidthModifier [x][y] = (Floor((dQuantOffsetActual [x][y]) * (qm[x+ (levelIdxSwap * nTbS)][y])) / 32768)

[0316] The offset value is a function of the step width (i.e., qm[x][y]), for example:

[0317] dQuantOffsetActual [x][y] = ((Floor(−Cconst * Ln(qm[x + (levelIdxSwap* nTbs)][y]) + (dQuantOffset << 9) + Floor(Cconst * Ln(StepWidth)))) * (qm[x+ (levelIdxSwap * nTbs)][y])) >> 16

[0318] Dequantization offset

[0319] The methods and embodiments described herein also include ways to appropriately utilize the predictable statistical characteristics of residual data in hierarchical encoding methods. Specifically, in the context of a hierarchical structure, residual data is constructed by means of a distribution based on a known probability distribution pattern (e.g., by means of non-restrictive instances, Laplace distribution, Poisson distribution, etc.): thus it is possible to quickly identify the key parameters of the probability distribution and indicate them to the decoder by means of a relatively limited set of metadata.

[0320] Furthermore, the non-limiting embodiments herein also include methods for appropriately adapting quantization and dequantization operations based on the known probability distribution of the original symbols, in order to simultaneously improve compression efficiency, reconstruction quality, and processing speed. The non-limiting embodiments herein also include methods for minimizing signaling overhead by deriving, using default formulas, how the optimal dequantization offset for each coefficient group should change based on the relative magnitude of its quantization step width.

[0321] The embodiments described herein deviate from conventional systems and methods, thereby providing a novel approach to dynamically adapt quantization and dequantization operations based on appropriate and concise indications of the characteristics of the probability distribution of the original unquantized values.

[0322] In a broader sense, a method is disclosed for rapidly evaluating the probability distribution of raw unquantized values, generating appropriate signaling metadata based on said distribution, and adapting quantization and dequantization operations based on said metadata. This disclosure also includes decoder-side methods to improve the statistical properties of the dequantization offset even in the absence of signaling information from the encoder.

[0323] The concept is particularly valuable and effective in the context of hierarchical coding methods because the hierarchical structure implicitly defines the type of probability distribution of the residual values. This, in turn, allows each specific probability distribution to be characterized with relatively few and quickly identifiable parameters, and presents several properties of the dequantization offset (e.g., its relationship to the magnitude of the quantization step width). According to the non-limiting embodiments described herein, the novel dequantization method described herein is used in loops within the encoding and decoding layers of a hierarchical coding method.

[0324] For simplicity, the non-limiting embodiments described herein refer to the signal as a sequence of samples (i.e., two-dimensional images, video frames, video fields, audio frames, etc.). In the description, the terms "image," "picture," or "plane" (in the broadest sense of a "hyperplane," i.e., an array of elements having any number of dimensions and a given sampling raster) will often be used to identify a digital reproduction of a signal along a sample sequence, wherein each plane has a given resolution in each of its dimensions (e.g., X and Y) and comprises a set of planar elements (or "elements" or "pixels" or display elements, often referred to as "pixels" in a two-dimensional image, often referred to as "volumetric elements," etc.), characterized as one or more "values" or "settings" (e.g., by means of non-limiting examples, color settings in a suitable color space, settings indicating density levels, settings indicating temperature levels, settings indicating audio pitch shift, settings indicating amplitude, etc.). Each planar element is identified by a suitable set of coordinates indicating the integer position of the element within the sampling raster of the image. The signal dimension may include only the spatial dimension (e.g., in the case of an image) or also include the temporal dimension (e.g., in the case of a signal that evolves over time).

[0325] As a non-limiting example, the signal can be an image, an audio signal, a multi-channel audio signal, a telemetry signal, a video signal, a multi-view video signal (e.g., 3D video), a volumetric signal (e.g., medical imaging, scientific imaging, holographic imaging, etc.), a volumetric video signal, or even a signal with more than four dimensions.

[0326] For simplicity, the non-limiting embodiments described herein often refer to signals displayed as a defined 2D plane (e.g., a 2D image in a suitable color space), such as video signals. The terms "frame" or "field" will be used interchangeably with the term "image" to indicate a temporal sample of a video signal: any concepts and methods described for video signals composed of frames (progressive video signals) can also be readily applied to video signals composed of fields (interlaced video signals), and vice versa. Although the embodiments described herein focus on image and video signals, those skilled in the art will readily understand that the same concepts and methods are also applicable to any other type of multidimensional signal (e.g., audio signals, volumetric signals, stereoscopic video signals, 3DoF / 6DoF video signals, all-optical signals, etc.).

[0327] According to the non-limiting embodiments described herein, a signal processor (e.g., computer processor hardware) is configured to receive data to be quantized and a corresponding reference quantization step width to be used for quantizing the data. Based at least in part on the quantization step width, the signal processor identifies a range of unquantized values ​​corresponding to a quantized value of zero (“Dead Zone”). In some non-limiting embodiments, the encoder calculates the Dead Zone based on a formula known to both the encoder and decoder, according to the reference step width. This allows for variation of the relative size of the Dead Zone quantization block group without any signaling overhead. In non-limiting embodiments, the Dead Zone is calculated as a multiple of the quantization step width (i.e., Dead Zone = DZmultiplier * StepWidth), where the Dead Zone multiplier DZmultiplier is calculated as a linear function of StepWidth (i.e., DZmultiplier = a * StepWidth + b, where a and b are suitable constants known to both the encoder and decoder). In other non-limiting embodiments, the relationship between DZmultipliers is non-linear. In some non-limiting embodiments, DZmultipliers are represented using fixed-point integer numbers to avoid using hardware-specific floating-point calculations.

[0328] According to the non-limiting embodiments described herein, the encoder identifies a dequantization offset value δ based at least in part on a metric calculated from the data to be quantized, to be added to the central limit of each unquantized value range (“quantization block group”) in order to obtain the value corresponding to Q. i Quantization value of the solution The dequantization of the quantized symbol Q is performed by both the encoder (for analog decoding) and the decoder as follows.

[0329] If the quantized sign Qi is positive:

[0330]

[0331] If the symbol Q is actually i Negative:

[0332]

[0333] In some non-limiting embodiments, the decoder generates a dequantization offset value δ by analyzing the signal-encoded byte stream. If the signaling bit corresponding to the dequantization offset is set to 1, the decoder processes subsequent bytes in the byte stream to generate a reference dequantization offset.

[0334] According to a third non-limiting embodiment, for a plurality of encoded data sets (e.g., groups of encoded transform coefficients), the encoder transmits a single parameter corresponding to a reference quantization StepWidth to the decoder, and the decoder uses a quantization matrix to process the parameter corresponding to the reference StepWidth and derives a distinct quantization StepWidth for each of the groups of transform coefficients. To limit the number of dequantization offsets to be transmitted as meta-data in the byte stream, both the encoder and decoder automatically derive a specific dequantization offset value δ for each of the groups of encoded coefficients from a single reference dequantization offset value δ, which is predetermined to correspond to the offset of the transmitted reference StepWidth of the data layer. i The offset value δ i It depends at least in part on the dequantization offset δ of the transmitted signal and the quantization StepWidth of the coefficient set i. i .

[0335] In a non-limiting embodiment, the encoder and decoder calculate the offsets of all quantized StepWidths derived from the reference StepWidth by assuming the following relationship:

[0336] Dequantization_Offset k = f (StepWidth k )

[0337] And therefore:

[0338] Actual_Dequantization_Offset i =

[0339] = Reference_Dequantization_Offset + f (StepWidth i - f(Reference_StepWidth)

[0340] In some non-limiting embodiments, f(StepWidth) is a logarithmic relationship.

[0341] In other non-limiting embodiments, the encoder derives the value of the reference dequantization offset δ using an RDO (“rate distortion optimization”) algorithm. As a non-limiting example, in one embodiment, the encoder calculates the reference dequantization offset δ by performing a bisection search optimization algorithm on the data to be quantized using appropriate metrics (e.g., MSE, MAE, PSNR, as non-limiting examples) to determine the fidelity of the reproduction produced by dequantizing with a given dequantization offset δ. In a non-limiting embodiment, the signal processor selects the original unquantized value and the dequantized value... The value that minimizes the mean square error between the two. In other non-limiting embodiments, the encoder performs the optimization only considering the residuals corresponding to regions of the signal that have been identified as having relatively high priority according to a suitable priority mapping.

[0342] In other non-limiting embodiments, the encoder selects a dequantization offset and detects coefficient values ​​quantized at dequantization points that are likely far from the corresponding quantization buckets. Accordingly, the encoder assigns quantized symbols from adjacent quantization buckets to the coefficients to reduce the measure of the difference between the unquantized value of the coefficient and the dequantized value of the quantized symbol assigned to the coefficient. In a non-limiting embodiment, the encoder detects at least one coefficient with respect to an edge and assigns it a different quantized symbol to enhance edge transitions.

[0343] In other non-limiting embodiments, the encoder performs quantization in two steps: a first pass of quantization with a relatively large step width and a relatively small dead zone, the purpose of which is to group adjacent residual values ​​into clusters; and then, actual quantization with a relatively small target quantization step width, to detect the reference solution quantization offset of the coefficient group.

[0344] According to another non-limiting embodiment, the decoder is configured to generate progressively larger dequantization offsets as the quantization bucket moves from the center (zero symbol / dead zone) of the residual distribution to the two tails of the distribution. In some non-limiting embodiments, the decoder is configured for each quantization StepWidth. i The generation depends on StepWidth i StepWidth_Modifier i The decoder then produces the Decoding_StepWidth value. i =(StepWidth i +StepWidth_Modifier i Then use Decoding_StepWidth. i Replace StepWidth i In order to perform dequantization.

[0345] In some non-limiting embodiments, the encoder signals the dequantization offset value δ as coded metadata within the coded bitstream. In one embodiment, the decoder detects whether a dequantization offset is signaled for a given group of quantized coefficients using a given signaling bit in the byte stream: if the bit is set to 1, subsequent bytes contain information corresponding to the magnitude of the reference dequantization offset. In other non-limiting embodiments, when no dequantization offset is signaled, the decoder continues to compute the StepWidth_Modifier for each group of transform coefficients. In other non-limiting embodiments, based on a second signaling bit, the decoder processes the byte stream to generate a reference dequantization offset and simultaneously generates one or more StepWidth_Modifiers to be added to the StepWidth to progressively increase the dequantization offset. The StepWidth_Modifier can be used to avoid artifacts such as mosquito noise.

[0346] In some non-limiting embodiments, processing operations and intermediate variables are performed using fixed-point integers, thereby avoiding the execution of processing operations that depend on floating-point numbers.

[0347] According to some other non-limiting embodiments, the offset value δ for each quantized block group is generated based on parameters that identify the probability distribution of the non-quantized values. i In a non-limiting embodiment, it is assumed that the offset value is calculated at least in part based on a Laplace (i.e., double exponential) probability distribution identified from the metadata. In other non-limiting embodiments, the offset value is decoded directly from the metadata. In a non-limiting embodiment, the offset value δ is the same for all block groups except those centered around zero; the offset value δ is added to the very center edge of the quantized block group for block groups corresponding to positive unquantized values ​​(thus reducing the amount of the dequantized value relative to the average of the block group range), and subtracted to the average of the block group for block groups corresponding to negative unquantized values ​​(thus again reducing the absolute value of the dequantized value relative to the average of the block group range).

[0348] In other non-limiting embodiments described herein, the signal processor is configured to perform a random dequantization process (“statistical jitter” or “statistical dequantization”) to reproduce a probability distribution similar to the probability distribution of the original non-quantized values ​​quantized into the same quantization block group i for dequantized values ​​belonging to the same quantization block group i. The signal processor receives metadata indicating the probability distribution of the original non-quantized values. Based at least in part on parameters of the probability distribution, on the minimum and maximum values ​​of each block group, and on randomly generated values, the signal processor performs a random dequantization process for each quantized value Q. i Generate dequantized values In these embodiments, the same quantization value Q i The multiple occurrences of a given value can be solved into multiple different solution quantization values. In a non-limiting embodiment, the parameters in the metadata indicating the probability distribution of the original unquantized values ​​are assumed to be parameters of a Laplace distribution.

[0349] In other non-limiting embodiments described herein, the signal processor processes the received metadata corresponding to the quantized data and generates a parameter StepWidth indicating the size of the quantized block group. i And information corresponding to the probability distribution of the original unquantized data. Based at least in part on the parameters and the information, the signal processor generates a range of all quantization block groups to be used for dequantizing the received quantized data. In a non-limiting embodiment, the size of the quantization block groups is non-uniform. In other non-limiting embodiments, the size of the quantization block groups is non-uniform and is obtained at least in part based on a lookup table known to the signal processor. In other non-limiting embodiments, dequantization follows a Dead Zone plus uniform step width (DZ+USW) scheme, where DZ and USW are both parameters StepWidth. i The function is denoted by f(StepWidth). In a non-limiting embodiment, the dequantization offset is also a function of the parameter StepWidth and is calculated as f(StepWidth).

[0350] According to other non-limiting embodiments described herein, the signal processor pre-computes a set of parameters based on the received metadata and stores the parameters in a lookup table to accelerate the dequantization process of the quantized data.

[0351] According to some non-limiting embodiments described herein, the information corresponding to the probability distribution of the original unquantized values ​​includes parameters corresponding to the diversity parameter (“b”) of the Laplace distribution. In other non-limiting embodiments, the information corresponding to the probability distribution of the original unquantized values ​​includes parameters corresponding to the probabilities of some quantized block groups (“mini-histograms”).

[0352] Figure 9 The distribution and quantization block groups are shown. It can be seen that the optimal dequantization point is neither in the middle of the quantization bucket nor at the very center limit of the quantization bucket. Under the assumption of a Laplace-like distribution of the residuals, the dequantization offset should be substantially the same for each quantization bucket. The optimal average dequantization offset varies with the stride width. Smaller stride widths require relatively larger offsets. Larger stride widths require smaller offsets. Different transformed coefficient groups are attributed to the quantization matrices encoding with different stride widths. Thus, if the offset is statistically accurate for one direction, it may be incorrect for another. The non-limiting embodiments described herein are based on directional step-width modified offsets. Other non-limiting embodiments described herein extend the decoding quantization stride width relative to the quantization stride width used for encoding to progressively increase the dequantization offset.

[0353] Figure 10The quantization and dequantization processes are illustrated schematically, as described elsewhere in this disclosure.

[0354] The following describes a specific implementation scheme for how the quantization and dequantization processes can be performed. If the dequantization offset is not specified by the encoder in the signaling (the default), the decoder operates conservatively, and

[0355] 1. Assume a possible residual distribution (i.e., with a relatively small optimal decoding offset).

[0356] 2. The decoding offset is applied progressively, relative to the StepWidth used for encoding, to extend the decoding StepWidth. This generates a progressively increasing «effective dequantization offset», which is minimal for quantization bucket 1 and gradually increases for buckets at the tail of the residual distribution.

[0357] In a non-limiting embodiment, the target «tailmost bucket» solution quantization offset is calculated using the following formula, given the transformed coefficients StepWidth i, assuming a Laplace distribution of residuals with β=500:

[0358]

[0359] The StepWidth modifier is then evaluated as follows:

[0360]

[0361]

[0362] And finally:

[0363]

[0364] Formulas with the StepWidth modifier can be derived from this formula:

[0365]

[0366] It should be noted that these formulas are consistent with the example implementation scheme described above.

[0367] In other non-limiting embodiments, if the dequantization offset to be used in dequantization is specified by the encoder in the signaling (e.g., by means of a dual-state switching signaling bit) and is greater than zero, then it is applied by the decoder at the specified LOQ target step width («reference dequantization offset») for dequantization.

[0368] If a quantization matrix is ​​applied, the dequantization offset in each direction is modified according to the formula (which will be decoded with different quantization widths), similar to one of the following non-restrictive examples:

[0369]

[0370]

[0371] Figure 11 and 12 Unrestricted instance values ​​of the Laplace distribution assuming a perfect distribution at each of the «b» (β) parameter levels are shown. These values ​​of β are taken in the range of 300–2,000.

[0372] In another non-restrictive instance, a pattern similar to the default operation is proposed, but the dequantization offset of the target «tailmost_bucket» is calculated as follows:

[0373]

[0374] In this unrestricted instance, the StepWidth modifier is then evaluated as follows:

[0375]

[0376]

[0377] And finally:

[0378]

[0379] Again, it should be noted that in one instance, the formula for the StepWidth modifier can be derived from this formula:

[0380]

[0381] Figure 13 A block diagram of a system used to implement the processes described herein is shown.

[0382] Other implementation plans

[0383] In some embodiments as described herein, in example implementations, it should be noted that the following process may be implemented for lossless compression:

[0384] If stepWidth > 16, then the deadZoneWidthOffset is exported as follows:

[0385] deadZoneWidthOffset [x][y] = (((1 << 16) − ((Aconst * (qm[x +(levelIdxSwap * nTbs)][y] + stepWidthModifier [x][y])) + Bconst) >> 1) * (qm[x + (levelIdxSwap * nTbs)][y] + stepWidthModifier [x][y])) >> 16

[0386] If stepWidth <= 16, then the deadZoneWidthOffset is exported as follows:

[0387] deadZoneWidthOffset [x][y] = stepWidth >> 1

[0388] In other words, when the step width is less than 16, the deadzonewidthoffset does not depend on the parameter matrix.

[0389] Other example implementation schemes

[0390] The appendix below provides examples of non-limiting implementations that embody the principles described elsewhere in this document. These examples can be used to provide context for the described features.

[0391] Other example encoders and decoders

[0392] exist Figure 14 The diagram illustrates the encoding process 2500 used to create the bitstream. First, the input sequence 2502 is fed to the first downsampler 2504, followed by the second downsampler 2506 (i.e., the successive downsamplers referred to as downscalers in the diagram) and processed according to the selected scaling mode. Figure 14 The variation differs from the previous instance in that there are additional downsampling and upsampling stages before the base layer. For example, it is possible to have an additional downsampling stage, represented as the second downscalor 2506, before passing the data to the base encoder 2512, and an additional upsampling stage (represented as...) after receiving the decoded data from the base layer. Figure 14 The first up scaler 2508 is possible. In some instances, a given scaling mode can be used to turn the down scaler and up scaler pair on and off at each level. In one case, the scaling mode can indicate the scaling direction, such as horizontal downsampling / upsampling only as described herein. If the second down scaler 2506 and the first up scaler 2508 are disconnected, spatial scaling is similar to... Figure 1 Space scaling.

[0393] Figure 14 In this example, following the previous one, a base codec is used, which generates a base bitstream 2516 according to its own specification. This encoded base can be included as part of a combined bitstream within the current video decoding framework structure.

[0394] With or without additional upscaling, at the first subtraction component 2520, a reconstructed base image, such as the decoded form of the base coded frame, is subtracted from the first-order down-scaled input sequence to generate sub-layer 1 residuals (as described herein, layer 1 residual data). These residuals form the starting point for the encoding process of the first enhancement layer. Transform component 2521, quantization component 2523, and entropy coding component 2524 (and others) as described herein process the first (layer 1) residual set to generate (layer 1) entropy-coded quantized transform coefficients 2526.

[0395] exist Figure 14 In this example, following the previous instance, the entropy-encoded quantized transform coefficients from sublayer 1 are processed by the in-loop decoder, which performs inverse or decoding operations. These operations simulate the decoding process of the first set of residuals that would otherwise be performed at the decoder. Figure 14 In examples, these include an entropy decoding component 2525, an inverse quantization component 2527, an inverse transform component 2528, and a level 1 filter 2530. These may be similar to the components described previously. The processed or “decoded” first set of residuals is added at summing component 2532 to the data derived from the output of the base encoder (e.g., decoded and optionally upscaled) to generate the reconstructed frame. Figure 14In the process, the reconstructed frame is processed by a second upscaler 2534. The use of the upscaler may again depend on the selected scaling mode. Finally, at the second subtraction component 2536, the residuals of the second sublayer 2 (which may also be referred to as the L2 layer) are calculated by subtraction of the input sequence and the upscaled reconstruction. These form a second (layer 2) residual set, which is further processed by a set of decoding components or tools including a transform component 2541, a timing prediction component 2542, a quantization component 2543, and an entropy coding component 2544. The output is a set of layer 2 coefficient layers 2546. As described in other examples, if the timing mode is activated, additional timing prediction can be applied to the transform coefficients by the timing prediction component 2542 to remove some temporally redundant information and reduce the energy of the layer 2 residual stream (e.g., the number of values ​​and the number of non-zero residual values). The entropy-coded quantized transform coefficients of sublayer 2 and the timing layer 2556, which specifies the use of timing prediction on a block-by-block basis, are included in the enhanced bitstream. The time layer 2556 may include time signaling as described with reference to the previous examples. It may be entropy encoded by the entropy encoding component 2557. The entropy encoding component 2557 may apply at least run-length encoding, as discussed in the reference examples.

[0396] The encoder 2500 can be configured with a set of encoder configuration information 2565. This information can be transmitted to the decoder as a set of headers 2566 for the output bit stream. Figure 14 In the encoder, the combined bit stream may include a header 2566, a time layer 2556, level 2 (L2) coding coefficients 2546, level 1 (L1) coding coefficients 2526, and an encoded base stream 2516.

[0397] Figure 15 This demonstrates a variation of decoder 2600 based on one example. The decoder may include variations of the decoder shown in any of the other figures herein. Figure 15 The decoder can be connected with Figure 14 Use it with an encoder.

[0398] First, in order to create the output sequence of frames, the decoder 2600 analyzes the bitstream. For example... Figure 15 As can be seen, the process can be further divided into three parts.

[0399] To generate the decoded base image (e.g., at layer 0), the extracted base bitstream 2616 is fed to the base decoder 2618. Depending on the selected scaling mode, this reconstructed image can be upscaled by an additional first upscaler 2608 before a summing component 2630, which sums the first (layer 1) residual set. The input from the first upscaler 2608 to the summing component 2630 can be referred to as the preparatory intermediate image.

[0400] After (or in parallel with) the base layer decoding, the enhancement layer bitstream (containing two residual sub-layers) needs to be decoded. First, the coefficients 2626 belonging to sub-layer 1 (L1) are decoded using the inverse form of the decoding components or tools used during the encoding process. Therefore, the layer 1 coefficient layer 2626 is processed sequentially by the entropy decoding component 2671, the inverse quantization component 2672, and the inverse transform component 2673. Furthermore, a sub-layer 1 (L1) filter 2632 may be applied to smooth the boundaries of the transform blocks (i.e., decoding units). The output of the sub-layer 1 (L1) decoding process can be referred to as the enhanced sub-layer 1 output. This enhanced sub-layer 1 output is added to the preparatory intermediate image at the first (lower) summing component 2630 to produce a combined intermediate image. Again, depending on the scaling mode, a second upscaler 2687 may be applied, producing the resulting preparatory output image. The preparatory output image is provided to the second upper summing component 2658. It has the same size as the overall output image.

[0401] As a final step, the encoded coefficients 2646 of the second enhancement sublayer 2 are decoded. Again, this is done using a set of inverse decoding components or tools as described in other examples in this document. Figure 15 These components include an entropy decoding component 2681, an inverse quantization component 2682, and an inverse transform component 2683. If the time mode is activated, the time prediction component 2685 can apply time prediction. Time prediction can be applied at any point within the second enhancement sublayer 2. In one case, it is applied to the quantized transform coefficients. Time prediction can be applied based on signaling received as time layer 2656. Figure 15 In this process, time layer 2656 is decoded by entropy decoding component 2690 (e.g., via run length decoding). The output of time prediction is provided as the output of enhanced sublayer 2 to a second upper summing component 2658. It is then added by said summing component 2658 to the prepared output picture to form a combined output picture 2660 as the final output of the decoding process.

[0402] Furthermore, the decoding process can be controlled according to the decoder configuration 2692 transmitted in the header 2666 of the bit stream.

[0403] As described in the examples above, unlike the contrasting scalable codecs, the new method described in this paper is completely agnostic to the codec used to encode the lower layer. This is because the upper layer can be decoded without any information about the lower layer. Figure 15As shown, the decoder receives multiple streams generated by the encoder. These can be approximately five streams, comprising: a first coded stream (coded basis) generated by feeding a downsampled form of the input video to a base codec (e.g., AVC, HEVC, or any other codec); a second coded stream (layer 1 coefficient layer) generated by processing the residual (layer 1 residual) obtained by reconstructing the difference between the base codec video and the downsampled form of the input video; a third coded stream (layer 2 coefficient layer) generated by processing the residual (layer 2 residual) obtained by reconstructing the oversampled form of the base decoded video and the input video; a fourth coded stream (e.g., in the form of a time layer) generated by time processing to instruct the decoder; and a fifth stream (header) generated for configuring the decoder. The coded basis stream is decoded by implementing a decoding algorithm corresponding to the encoding algorithm implemented by the base codec used in the encoder by the base decoder, and the output of this decoding is the decoded basis. The layer 1 coefficient group is decoded separately and independently to obtain the layer 1 residual data. Furthermore, the level 2 coefficient group is decoded separately and independently to obtain level 2 residual data. The decoded base, level 1 residual data, and level 2 residual data are then combined. Specifically, the decoded base and level 1 residual data are combined to generate an intermediate image. The intermediate image can then be upsampled and further combined with the level 2 residual data.

[0404] Furthermore, the new method employs an encoding and decoding process that processes the image without using any inter-block prediction. In practice, it processes the image by transforming N×N blocks of pixels (e.g., 2×2 or 4×4) and processing these blocks independently of each other. This achieves efficient processing and independence from neighboring blocks, thus allowing for parallelization of image processing.

[0405] In summary, reference Figure 15 The following describes a non-limiting exemplary embodiment. Figure 15 An exemplary decoding module 2600 is depicted. The decoding module 2600 receives multiple input bit streams, including an encoded base 2616, a level 1 coefficient group 2626, a level 2 coefficient group 2646, a time coefficient group 2656, and a header 2666.

[0406] Generally, the decoding module 2600 processes two layers of data. The first layer, the base layer, includes the received data stream 2616 containing the encoded base. The encoded base 2616 is then sent to the base decoding module 2618, which decodes the encoded base 2616 to produce a decoded base image. The base decoding can be performed by the decoder implementing any existing base codec algorithm, such as AVC, HEVC, AV1, VVC, EVC, VC-6, VP9, ​​etc., depending on the encoding format of the encoded base.

[0407] The second layer, the enhancement layer, further consists of two enhancement sub-layers. The decoding module receives a first coefficient group, namely layer 1 coefficient group 2626, which is then passed to the entropy decoding module 2671 to generate a decoded coefficient group. These are then passed to the inverse quantization module 2672, which uses one or more dequantization parameters to generate a dequantized coefficient group. These are then passed to the inverse transform module 2673, which performs an inverse transform on the dequantized coefficient group to generate a residual at enhancement sub-layer 1 (layer 1 residual). The residual can then be filtered by a smoothing filter 2632. The layer 1 residual (i.e., the decoded first enhancement sub-layer) is applied to the processed output of the base image.

[0408] The decoding module receives a second coefficient group, namely level 2 coefficient group 2646, which is then passed to the entropy decoding module 2681 to generate a decoded coefficient group. These are then passed to the inverse quantization module 2682, which uses one or more dequantization parameters to generate a dequantized coefficient group. The dequantization parameters used for enhancing sublayer 2 may differ from those used for enhancing sublayer 1. The dequantized coefficient group is then passed to the inverse transform module 2683, which performs an inverse transform on the dequantized coefficient group to generate the residual at enhancing sublayer 2 (level 2 residual).

[0409] Implementation Plan

[0410] The methods and processes described herein may be embodied as code (e.g., software code) and / or data in both encoders and decoders, for example, implemented in a streaming server or client device or a client device decoding from data storage. Encoders and decoders may be implemented in hardware or software, as is well known in the field of data compression. For example, hardware acceleration using a specially programmed graphics processing unit (GPU) or a specially designed field-programmable gate array (FPGA) may provide some efficiency. For completeness, such code and data may be stored on one or more computer-readable media, which may contain any means or medium capable of storing code and / or data for use by a computer system. When a computer system reads and executes the code and / or data stored on the computer-readable medium, the computer system executes methods and processes embodied as data structures and code stored within the computer-readable storage medium. In some embodiments, one or more steps of the methods and processes described herein may be executed by a processor (e.g., a processor of a computer system or a data storage system).

[0411] In general, any of the functionality described in this text or shown in the figures can be implemented using software, firmware (e.g., fixed logic circuitry), programmable or non-programmable hardware, or combinations of these implementations. In general, as used herein, the terms "component" or "function" denote software, firmware, hardware, or combinations thereof. For example, in the case of a software implementation, the terms "component" or "function" can refer to program code that performs a specified task when executed on one or more processing devices. The illustrated separation of components and functions into distinct units may reflect any actual or conceptual physical grouping and allocation of such software and / or hardware and tasks.

[0412] Attachment

[0413] The following presents non-limiting embodiments that illustrate the principles described elsewhere herein. This example can be used to provide context for the features described.

[0414] Syntax and semantics

[0415] The syntax table specifies a superset of the syntax of all allowed bitstreams. Additional constraints on the syntax can be specified directly or indirectly as needed.

[0416] Note that an actual decoder should implement some means for identifying entry points into the bitstream, and some means for identifying and handling non-conforming bitstreams. Methods for identifying and handling errors and other such

[0417] Process payload - Picture configuration

[0418]

[0419] Data block unit picture configuration semantics

[0420] no_enhancement_bit_flag specifies that there is no enhancement data for all layerIdx < nLayers in the picture.

[0421] quant_matrix_mode specifies which quantization matrix is to be used in the decoding process according to the following table. When quant_matrix_mode is absent, it is inferred to be equal to 0.

[0422] Quantization matrix

[0423]

[0424] `dequant_offset_signalled_flag` specifies whether to signal the offset method and the value of the offset parameter to be applied during dequantization. If it equals 1, the method used for the dequantization offset and the value of the dequantization offset parameter are signaled. When `dequant_offset_signalled_flag` does not exist, it is inferred to be equal to 0.

[0425] The `picture_type_bit_flag` specifies whether encoded data is sent on a frame-based basis (e.g., progressive or interlaced mode) or a field-based basis (e.g., interlaced mode) according to the following table. Error! Reference source not found.

[0426] Image type

[0427]

[0428] The field_type_bit_flag specifies, according to the table below, whether the data sent is for the top or bottom field if picture_type equals 1. (Error! Reference source not found.)

[0429] Field type

[0430]

[0431] `temporal_refresh_bit_flag` specifies whether the time buffer should be refreshed for the image. If it equals 1, the time buffer should be refreshed. For IDR images, `temporal_refresh_bit_flag` will be set to 1.

[0432] `temporal_signalling_present_flag` specifies whether the temporal signaling coefficient group exists in the bit stream. When `temporal_signalling_present_flag` does not exist, it is inferred to be equal to 1 if `temporal_enabled_flag` is equal to 1 and `temporal_refresh_bit_flag` is equal to 0; otherwise, it is inferred to be equal to 0.

[0433] `step_width_level2` specifies the step width value to be used for the coded residual in the decoding enhancement sublayer 2 for the luma plane. The step width value to be used for the coded residual in the decoding enhancement sublayer 2 for the chroma plane will be calculated as `Clip3(1, 32,767,((step_width_level2*chroma_step_width_multiplier)>>6))`.

[0434] `step_width_level1_enabled_flag` specifies whether the step width to be used when decoding the encoded residual in Enhancement Sublayer 1 is the default value or signaled. It should be 0 (the default) or 1 (the signaled value from `step_width_level1`). The default value is 32,767. When `step_width_level1_enabled_flag` is not present, it is inferred to be equal to 0.

[0435] The `dithering_control_flag` specifies whether dithering should be applied. It should be 0 (disable dithering) or 1 (enable dithering). If `dithering_control_flag` is not present, it is assumed to be equal to 0.

[0436] step_width_level1 specifies the step width value to be used when decoding the encoded residual in enhancement sublayer 1.

[0437] The `level1_filtering_enabled_flag` specifies whether the level 1 unblocking filter should be used. It should be 0 (filtering disabled) or 1 (filtering enabled). If `level1_filtering_enabled_flag` does not exist, it is assumed to be equal to 0.

[0438] qm_coefficient_0[layerIdx] specifies the value of the quantization matrix scaling parameter when quant_matrix_mode is equal to 2, 3 or 5.

[0439] qm_coefficient_1[layerIdx] specifies the value of the quantization matrix scaling parameter when quant_matrix_mode is equal to 4 or 5.

[0440] The `dequant_offset_mode_flag` specifies the method used to apply the dequantization offset. If it equals 0, the transmitted `dequant_offset` is used as the parameter, and the default method applies. If it equals 1, the transmitted `dequant_offset` parameter is used, and the constant offset method applies.

[0441] The `dequant_offset` parameter specifies the value of the dequantization offset parameter to be applied. The value of the dequantization offset parameter should be between 0 and 127 (inclusive).

[0442] `dithering_type` specifies which type of dithering will be applied to the final reconstructed image according to the table below. (Error! Reference source not found.)

[0443] Shaking

[0444]

[0445] dithering_strength specifies a value between 0 and 31.

[0446] Decoding process

[0447] A decoding process is specified such that when the decoding process associated with the specified profile is invoked for a bitstream that conforms to the specified profile and level, all decoders conforming to the profile will produce numerically identical cropped decoded output images. Any decoding process that produces the same cropped decoded output image as produced by the process described herein (with the correct output order or output timing, as specified) meets the decoding process requirements.

[0448] General decoding process for L-2 encoded data blocks

[0449] The input for this process is:

[0450] Specify the sample position (xTb0, yTb0) of the top-left sample of the current transform block relative to the top-left sample of the current image, based on the value of the variable transform_type. The specified sub-clause is the size of the current transform block, nTbS (if transform_type equals 0, then nTbS = 2; if transform_type equals 1, then nTbS = 4).

[0451] Such as variables exported from other places: temporal_enabled_flag, temporal_refresh_bit_flag, temporal_signalling_present_flag, and temporal_step_width_modifier.

[0452] Specifies an array recL2ModifiedUpsampledSamples of size (nTbS) x (nTbS) of the upsampled reconstructed samples generated by a procedure specified elsewhere in the current block.

[0453] The array TransformCoeffQ is specified as having (nTbS)x(nTbS) values ​​for the quantized transform coefficients of L-2 entropy decoding.

[0454] If both `temporal_signalling_present_flag` and `temporal_tile_intra_signalling_enabled_flag` are equal to 1, then the variable `TransformTempSig` corresponds to the value in `TempSigSurface` at position (xTb0>>nTbs, yTb0>>nTbs); and if `temporal_tile_intra_signalling_enabled_flag` is also set to 1, then the variable `TileTempSig` corresponds to the value in `TempSigSurface` at position ((xTb0%32)*32, (yTb0%32)*32).

[0455] The step width value is derived from other parts of the variable step_width_level2.

[0456] The variable IdxPlanes specifies which plane the transformation coefficients belong to.

[0457] The output of this process is an array of (nTbS)x(nTbS) of L-2 residuals resL2Residuals with elements resL2Residuals[x][y].

[0458] The sample position (xTbP, yTbP) of the top-left sample of the current transform block relative to the top-left sample of the current image is exported as follows:

[0459] (xTbP,yTbP) = (IdxPlanes == 0) ? (xTb0, yTb0) : (xTb0 >> ShiftWidthC,yTb0 >> ShiftHeightC)

[0460] P can be associated with either the luminance or chrominance plane, depending on which plane the transformation coefficients belong to. ShiftWidthC and ShiftHeightC are specified elsewhere.

[0461] If no_enhancement_bit_flag is set to 0, the following ordered steps apply:

[0462] If the variable temporal_enabled_flag equals 1 and the temporal_refresh_bit_flag equals 0, then the time prediction process specified elsewhere is called with the brightness position (xTbY, yTbY), the transformation size is set to nTbS, the variables TransformTempSig and TileTempSig are used as inputs, and the output is an array tempPredL2Residuals of size (nTbS) x (nTbS).

[0463] If the variables temporal_enabled_flag and temporal_refresh_bit_flag are both equal to 1, then the array tempPredL2Residuals of size (nTbS) x (nTbS) is set to contain only zeros.

[0464] If the variables `temporal_enabled_flag` equal 1, `temporal_refresh_bit_flag` equal 0, `temporal_tile_intra_signalling_enabled_flag` equal 1 (sub-clause 0), and `TransformTempSig` equal 0, then the variable `stepWidth` is modified to `Floor(stepWidth*(1−(Clip3(0, 0.5,(temporal_step_width_modifier / 255))))`. The dequantization process, as specified elsewhere, is called with a transform size set to nTbS, an array `TransformCoeffQ` of size (nTbS) x (nTbS), and the variable `stepWidth` as input, and the output is an array `dequantCoeff` of size (nTbS) x (nTbS).

[0465] The transformation process is called as specified elsewhere, with the brightness position (xTbY, yTbY), the transformation size set to be equal to nTbS, and the array dequantCoeff of size (nTbS)x(nTbS) as input, and the output is the array resL2Residuals of size (nTbS)x(nTbS).

[0466] If the variable temporal_enabled_flag is equal to 1, the arrays of tempPredL2Residuals of size (nTbS)x(nTbS) are added to the array resL2Residuals of size (nTbS)x(nTbS), and the array resL2Residuals is stored in temporalBuffer at the luminance position (xTbY, yTbY).

[0467] If no_enhancement_bit_flag is set to 1, the following ordered steps apply:

[0468] If the variables temporal_enabled_flag equals 1, temporal_refresh_bit_flag equals 0, and temporal_signalling_present_flag equals 1, then the transformation size is set to nTbS with the brightness position (xTbY, yTbY), the variables TransformTempSig and TileTempSig are used as input to call the time prediction process as specified elsewhere, and the output is an array tempPredL2Residuals of size (nTbS)x(nTbS).

[0469] If the variables temporal_enabled_flag, temporal_refresh_bit_flag, and temporal_signalling_present_flag are all equal to 1, then the time prediction process specified elsewhere is called with the brightness position (xTbY, yTbY), the transformation size set to nTbS, the variables TransformTempSig and TileTempSig set to 0 as inputs, and the output is an array tempPredL2Residuals of size (nTbS) x (nTbS).

[0470] If the variables temporal_enabled_flag and temporal_refresh_bit_flag are both equal to 1, then the array tempPredL2Residuals of size (nTbS) x (nTbS) is set to contain only zeros.

[0471] If the variable temporal_enabled_flag is equal to 1, then the array of tempPredL2Residuals of size (nTbS)x(nTbS) is stored in the array resL2Residuals of size (nTbS)x(nTbS), and the array resL2Residuals is stored in temporalBuffer at the brightness position (xTbY, yTbY).

[0472] Otherwise, the array resL2Residuals of size (nTbS)x(nTbS) is set to contain only zeros.

[0473] The image reconstruction process for each plane specified in the sub-clause is incorrect! Reference source not found. It is invoked with the transform block position (xTb0, yTb0), transform block size nTbS, variable IdxPlanes, (nTbS)x(nTbS) array resL2Residuals, and (xTbY)x(yTbY)recL2ModifiedUpsampledSamples as input.

[0474] Decoding process used for dequantization

[0475] Each group of transform coefficients passed to this process belongs to a specific plane and enhancement sublayer. It has been scaled with dead-zone using a uniform quantizer. The quantizer can use a non-centered dequantization offset.

[0476] Scaling process for transform coefficients

[0477] The input for this process is:

[0478] The variable nTbS specifies the size of the current transform block (nTbS=2 if transform_type is zero, and nTbS=4 if transform_type is 1).

[0479] TransformCoeffQ is an array containing entropy-decoded quantized transform coefficients of size (nTbS) x (nTbS).

[0480] The variable stepWidth specifies the step width value parameter.

[0481] The variable levelIdx specifies the index of the enhanced sublayer (levelIdx=1 for enhanced sublayer 1 and levelIdx=2 for enhanced sublayer 2).

[0482] The variables dQuantOffset and dequant_offset are specified as the dequantization offset.

[0483] If quant_matrix_mode is not 0, then the array QmCoeff0 of size 1 x nTbS2 is equal to the array variable qm_coefficient_0. In addition, if quant_matrix_mode is equal to 4, then the array QmCoeff1 of size 1 x nTbS2 is equal to the array qm_coefficient_1.

[0484] If nTbS==2, then there is an array QuantScalerDDBuffer containing the size (3*nTbS)x(nTbS) of the scaling parameter array used in the previous image.

[0485] If nTbS==4, then there is an array QuantScalerDDSBuffer containing the size (3*nTbS)x(nTbS) of the scaling parameter array used in the previous image.

[0486] The output of this process is an array d of (nTbS)x(nTbS) with elements d[x][y] and dequantized transform coefficients of the updated array QuantMatrixBuffer.

[0487] To derive the scaled transformation coefficients d[x][y], where x = 0...nTbS−1, y = 0...nTbS−1, and given the matrix qm[x][y] as specified in subclause 8.6.2, use the following formula:

[0488] d[x][y] = (TransformCoeffQ[x][y] * ((qm[x + (levelIdxSwap * nTbS)][y]+ stepWidthModifier[x][y]) + appliedOffset [x][y]) (1)

[0489] Derivation of dequantization offset and step width modifier

[0490] The variables appliedOffset[x][y] and stepWidthModifier[x][y] are exported as follows:

[0491] if (dequant_offset_signalled_flag == 0) {

[0492] stepWidthModifier [x][y] = ((((Floor(−Cconst * Ln (qm[x +(levelIdxSwap * nTbS)][y]))) + Dconst) *

[0493] (qm[x + (levelIdxSwap * nTbS)][y]2))) / 32768) >> 16

[0494] if (TransformCoeffQ[x][y] < 0)

[0495] appliedOffset [x][y] = (−1 * (−deadZoneWidthOffset [x][y]))

[0496] else if (TransformCoeffQ [x][y] > 0)

[0497] appliedOffset [x][y] = −deadZoneWidthOffset [x][y]

[0498] else

[0499] appliedOffset [x][y] = 0

[0500] } else if (dequant_offset_signalled_flag == 1) && (dequant_offset_mode_flag ==1) {

[0501] stepWidthModifier [x][y] = 0

[0502] if (TransformCoeffQ[x][y] < 0)

[0503] appliedOffset = (−1 * (dQuantOffsetActual [x][y] −deadZoneWidthOffset [x][y]))

[0504] else if (TransformCoeffQ [x][y] > 0)

[0505] appliedOffset [x][y] = dQuantOffsetActual [x][y] −deadZoneWidthOffset [x][y]

[0506] else

[0507] appliedOffset [x][y] = 0}

[0508] } else if (dequant_offset_signalled_flag == 1) && (dequant_offset_mode_flag == 0) {

[0509] stepWidthModifier [x][y] = (Floor((dQuantOffsetActual [x][y]) * (qm[x+ (levelIdxSwap * nTbS)][y]))

[0510] / 32768)

[0511] if (TransformCoeffQ[x][y] < 0)

[0512] appliedOffset = (−1 * (−deadZoneWidthOffset [x][y]))

[0513] else if (TransformCoeffQ [x][y] > 0)

[0514] appliedOffset [x][y] = −deadZoneWidthOffset [x][y]

[0515] else

[0516] appliedOffset[x][y] = 0

[0517] }

[0518] If stepWidth > 16, then the deadZoneWidthOffset is exported as follows:

[0519] deadZoneWidthOffset [x][y] = (((1 << 16) − ((Aconst * (qm[x +(levelIdxSwap * nTbs)][y] + stepWidthModifier [x][y])) + Bconst) >> 1) * (qm[x + (levelIdxSwap * nTbs)][y] + stepWidthModifier [x][y])) >> 16

[0520] If stepWidth <= 16, then the deadZoneWidthOffset is exported as follows:

[0521] deadZoneWidthOffset [x][y] = stepWidth >> 1

[0522] in:

[0523] Aconst = 39

[0524] Bconst = 126484

[0525] Cconst = 5242

[0526] Dconst = 99614

[0527] where dQuantOffsetActual[x][y] is calculated as follows:

[0528] if (dequant_offset == 0)

[0529] dQuantOffsetActual [x][y] = dQuantOffset

[0530] else {

[0531] if (dequant_offset_mode_flag == 1)

[0532] dQuantOffsetActual [x][y] = ((Floor(−Cconst * Ln(qm[x + (levelIdxSwap* nTbs)][y]) +

[0533] (dQuantOffset << 9) + Floor(Cconst * Ln(StepWidth)))) * (qm[x +(levelIdxSwap * nTbs)][y])) >> 16

[0534] else if (dequant_offset_mode_flag == 0)

[0535] dQuantOffsetActual [x][y] = ((Floor(−Cconst * Ln(qm[x + (levelIdxSwap* nTbs)][y]) +

[0536] (dQuantOffset << 11) + Floor(Cconst * Ln(StepWidth)))) * (qm[x +(levelIdxSwap * nTbs)][y]))

[0537] >>16

[0538] }

[0539] where levelIdxSwap is derived as follows:

[0540] if (levelIdx == 2)

[0541] levelIdxSwap = 0

[0542] else

[0543] levelIdxSwap = 1

[0544] Derivation of the quantization matrix

[0545] The quantization matrix qm[x][y] contains the actual quantization step width to be used for decoding each coefficient group.

[0546] if (levelIdx == 2) {

[0547] if (scaling_mode_level2 == 1) {

[0548] for (x = 0; x < nTbS; x++) {

[0549] for (y = 0; y < nTbs; y++)

[0550] qm[x][y] = qm_p[x][y]

[0551] }

[0552] } else {

[0553] for (x = 0; x < nTbS; x++) {

[0554] for (y = 0; y < nTbS; y++)

[0555] qm[x][y] = qm_p[x + nTbS][y]

[0556] }

[0557] }

[0558] } else {

[0559] for (x = 0; x < nTbS; x++) {

[0560] for (y = 0; y < nTbs; y++)

[0561] qm [x][y] = qm_p [x + (2 * nTbS)][y]

[0562] }

[0563] }

[0564] The calculation of qm_p[x][y] is as follows:

[0565] if (nTbs == 2) {

[0566] for (x = 0; x < 6; x++) {

[0567] for (y = 0; y < nTbs; y++)

[0568] qm_p[x][y] = (Clip3 (0, (3 << 16),[(QuantScalerDDBuffer [x][y] *stepWidth) + (1 << 16)]) *

[0569] stepWidth) >> 16

[0570] }

[0571] } else {

[0572] for (y = 0; y < 12; y++) {

[0573] for (x = 0; x < nTbs; x++)

[0574] qm_p[x][y] = (Clip3 (0, (3 << 16),[(QuantScalerDDSBuffer [x][y] *stepWidth) + (1 << 16)]) *

[0575] stepWidth) >> 16

[0576] }

[0577] }

[0578] Furthermore, QuantScalerDDBuffer[x][y] is exported elsewhere, and QuantScalerDDSBuffer[x][y] is exported elsewhere.

[0579] Derivation of scaling parameters for 2×2 transformation

[0580] If the variable nTbS equals 2, the default scaling parameters are as follows:

[0581] default_scaling_dd[x][y] =

[0582] {

[0583] { 0, 2}

[0584] { 0, 0}

[0585] { 32, 3 )

[0586] { 0, 32}

[0587] { 0, 3}

[0588] { 0, 32}

[0589] }

[0590] As a first step, the array QuantScalerDDBuffer[x][y] is initialized as follows:

[0591] If the current image is an IDR image, then QuantScalerDDBuffer[x][y] is initialized to be equal to default_scaling_dd[x][y]. If the current image is not an IDR image, then the QuantScalerDDBuffer[x][y] matrix remains unchanged.

[0592] After initialization, based on the value of quant_matrix_mode, the array QuantScalerDDBuffer[x][y] is processed as follows:

[0593] If quant_matrix_mode equals 0 and the current image is not an IDR image, then QuantScalerDDBuffer[x][y] remains unchanged.

[0594] If quant_matrix_mode equals 1, then QuantScalerDDBuffer[x][y] equals default_scaling_dd[x][y].

[0595] If quant_matrix_mode equals 2, then QuantScalerDDBuffer[x][y] is modified as follows:

[0596] for (MIdx = 0; MIdx < 3; MIdx++)

[0597] for (x = 0; x < 2; x++)

[0598] for (y = 0; y < 2; y++)

[0599] QuantScalerDDBuffer [x + (MIdx * 2)][y] = QmCoeff0[(x * 2) + y]

[0600] If quant_matrix_mode equals 3, then QuantScalerDDBuffer[x][y] is modified as follows:

[0601] for (MIdx = 0; MIdx < 2; MIdx++)

[0602] for (x = 0; x < 2; x++)

[0603] for (y = 0; y < 2; y++)

[0604] QuantScalerDDBuffer [x + (MIdx * 2)][y] = QmCoeff0 [(x * 2) + y]

[0605] If quant_matrix_mode equals 4, then QuantScalerDDBuffer[x][y] is modified as follows:

[0606] for (x = 0; x < 2; x++)

[0607] for (y = 0; y < 2; y++)

[0608] QuantScalerDDBuffer [x + 4][y] = QmCoeff1 [(x * 2) + y]

[0609] If quant_matrix_mode equals 5, then QuantScalerDDBuffer is modified as follows:

[0610] for (MIdx = 0; MIdx < 2; MIdx ++)

[0611] for (x = 0; x < 2; x++)

[0612] for (y = 0; y < 2; y++)

[0613] QuantScalerDDBuffer [x + (MIdx * 2)][y] = QmCoeff0[(x * 2) + y]

[0614] for (x = 4, x < 6; x++)

[0615] for (y = 0; y < 2; y++)

[0616] QuantScalerDDBuffer [x][y] = QmCoeff1[(x * 2) + y]

[0617] Derivation of scaling parameters for 4×4 transformation

[0618] If the variable nTbS equals 4, the default scaling parameters are as follows:

[0619] default_scaling_dds[x][y] =

[0620] {

[0621] { 13, 26, 19, 32}

[0622] { 52, 1, 78, 9}

[0623] { 13, 26, 19, 32}

[0624] { 150, 91, 91, 19}

[0625] { 13, 26, 19, 32}

[0626] { 52, 1, 78, 9}

[0627] { 26, 72, 0, 3}

[0628] { 150, 91, 91, 19}

[0629] { 0, 0, 0, 2}

[0630] { 52, 1, 78, 9}

[0631] { 26, 72, 0, 3}

[0632] { 150, 91, 91, 19}

[0633] }

[0634] As a first step, the array QuantScalerDDSBuffer[][] is initialized as follows:

[0635] If the current image is an IDR image, then QuantScalerDDSBuffer[x][y] is initialized to be equal to default_scaling_dds[x][y]. If the current image is not an IDR image, then the QuantScalerDDSBuffer[x][y] matrix remains unchanged.

[0636] After initialization, based on the value of quant_matrix_mode, the array QuantScalerDDSBuffer[x][y] is processed as follows:

[0637] If quant_matrix_mode equals 0 and the current image is not an IDR image, then QuantScalerDDSBuffer remains unchanged.

[0638] If quant_matrix_mode equals 1, then QuantScalerDDSBuffer equals default_scaling_dds[x][y].

[0639] If quant_matrix_mode equals 2, then QuantScalerDDSBuffer is modified as follows:

[0640] for (MIdx = 0; MIdx < 3; MIdx++)

[0641] for (x = 0; x < 4; x++)

[0642] for (y = 0; y < 4; y++)

[0643] QuantScalerDDSBuffer [x + (MIdx * 4)][y] = QmCoeff0[(x * 4) + y]

[0644] If quant_matrix_mode equals 3, then QuantScalerDDSBuffer is modified as follows:

[0645] for (MIdx = 0; MIdx < 2; MIdx++)

[0646] for (x = 0; x < 4; x++)

[0647] for (y = 0; y < 4; y++)

[0648] QuantScalerDDSBuffer [x + (MIdx * 4)][y] = QmCoeff0[(x * 4) + y]

[0649] If quant_matrix_mode equals 4, then QuantScalerDDSBuffer is modified as follows:

[0650] for (x = 0; x < 4; x++)

[0651] for (y = 0; y < 4; y++)

[0652] QuantScalerDDSBuffer [x + 8][y] = QmCoeff1[(x * 4) + y]

[0653] If quant_matrix_mode equals 5, then QuantScalerDDSBuffer is modified as follows:

[0654] for (MIdx = 0; MIdx < 2; MIdx++)

[0655] for (x = 0; x < 4; x++)

[0656] for (y = 0; y < 4; y++)

[0657] QuantScalerDDSBuffer [x + (MIdx * 4)][y] = QmCoeff0[(x * 4) + y]

[0658] for (x = 8, x < 12; x++)

[0659] for (y = 0; y < 4; y++)

[0660] QuantScalerDDSBuffer [x][y] = qm_coefficient_1[(x * 4) + y]

Claims

1. A method for encoding an input signal into a plurality of coded streams, wherein the coded streams can be combined to reconstruct the input signal, the method comprising: Receive input signals; The input signal is downsampled to create an downsampled signal; The instruction is to use a base encoder to encode the downsampled signal to create a base encoded stream; Instructs the use of a basic decoder to decode the underlying encoded stream to generate the reconstructed signal; The reconstructed signal is compared with the input signal to create a residual set; and... Encoding the residual set to create a hierarchical encoded stream, comprising: The transformation is applied to the residual set to create a set of transformed coefficients; The quantization operation is applied to the set of transformed coefficients to create a set of quantized coefficients; as well as Apply the encoding operation to the quantized coefficients; Wherein, the quantization operation uses a quantization offset, such that dequantizing the quantized coefficients involves applying the dequantization offset to the quantized coefficients; and The quantization offset is determined such that the dequantization offset is derived as a function of the quantization step width.

2. The method of claim 1, wherein the residual set is a first residual set, and the comparison step includes comparing the reconstructed signal with the downsampled signal to create the first residual set, such that the encoded stream is a first-level encoded stream, the method further comprising: Decode the first residual set to generate the decoded first residual set; The reconstructed signal is corrected using the decoded first residual set to generate a corrected reconstructed signal; The corrected reconstructed signal is upsampled to generate an upsampled reconstructed signal; The upsampled reconstructed signal is compared with the input signal to create a second set of residuals; and Encode the second residual set to create a second-level encoded stream, comprising: The transformation is applied to the second residual set to create a second set of coefficients; Apply the quantization operation to the second set of coefficients to create a second set of quantized coefficients; as well as The encoding operation is applied to the second set of quantized coefficients.

3. The method of claim 1 or 2, wherein the quantization operation further comprises applying a block group folding operation to the set of coefficients, the block group folding operation comprising quantizing each coefficient having a value exceeding a predetermined maximum value to have a quantized value corresponding to a first quantized block group among a plurality of quantized block groups having a defined step width, the maximum value being defined by an upper limit of the first quantized block group.

4. The method of claim 3, wherein the block group folding is configured to reduce the bit rate based on network conditions and / or underlying stream processing.

5. The method of claim 1, further comprising sending signaling information to indicate whether a dequantization offset should be applied when the decoder performs dequantization, wherein, The step width used in the quantization operation varies according to the step width parameter, and the signaling information also indicates that the decoder should modify how the step width modifier parameter is derived in dequantization, wherein the step width modifier is used to modify the dequantization step width used in dequantization.

6. The method of claim 1, wherein using the quantization offset comprises applying the quantization offset to a plurality of quantization block groups, the plurality of quantization block groups having a defined step width to adjust one or more corresponding values ​​in the plurality of quantization block groups by the value of the quantization offset.

7. The method of claim 6, wherein the value corresponding to each of the plurality of quantization block groups is adjusted.

8. The method of claim 6, wherein either or both of the value corresponding to the beginning of the first block group of the plurality of block groups and the value corresponding to the end of the last block group of the plurality of block groups remain unadjusted by the value of the quantization offset.

9. The method of claim 1, wherein the value of the quantization offset is configurable.

10. The method of claim 1, wherein the value of the quantization offset is changed based on data indicating the operating conditions under which the encoding is performed.

11. The method of claim 1, further comprising transmitting a signal quantization offset value to a decoder that will receive the encoded stream thereon.

12. The method of claim 1, wherein the quantization operation comprises subtracting a quantization offset value from the residual or coefficient value prior to quantization based on the quantization step width.

13. The method of claim 12, wherein the value of the quantization offset is adjusted based on the sign of the residual or coefficient.

14. The method according to claim 1, wherein, When the value of the quantization offset is set to a first predetermined value, the application of the offset to the quantization block group value is deactivated.

15. The method of claim 1, wherein the value of the quantization offset is adjusted based on the defined width of the dead zone.

16. The method of claim 1, further comprising deriving a corresponding dequantization offset δ for each of the encoded coefficient groups from a single reference dequantization offset δ corresponding to the transmitted signal reference StepWidth of a given data layer. i The offset value δ i The quantization StepWidth depends at least in part on the reference offset δ and the coefficient set i. i .

17. The method of claim 1, wherein the offsets of all quantization steps are derived from the reference step width according to the following relationship: Dequantization_Offset k =f(StepWidth k ), and the following relationships: Actual_Dequantization_Offset i =Reference_Dequantization_Offset+f(StepWidth i )-f(Reference_StepWidth) in, StepWidth k Dequantization_Offset represents the quantization step width associated with a set of coefficients k. k Actual_Dequantization_Offset represents the dequantization offset associated with a set of coefficients k. i Represents the dequantization offset associated with a set of coefficients i, Reference_StepWidth represents the quantization step width of the transmitted reference for a given data layer, Reference_Dequantization_Offset represents the reference dequantization offset associated with the quantization step width of the transmitted reference, and f represents a function.

18. The method of claim 17, wherein f(StepWidth) is a logarithmic relation.

19. The method of claim 1, wherein the step width varies for each of one or more of the coefficients in the set of coefficients.

20. The method of claim 19, wherein the step width variation causes a smaller step width value to be used for one or more of the coefficients that are predetermined to significantly influence the perception of the decoded signal.

21. The method of claim 1, wherein a default value is assigned to the step width according to the base step width parameter, and wherein one or more modified step widths are obtained according to the base step width and the step width modifier parameter.

22. The method of claim 21, wherein the step width for each of one or more of the coefficients is modified using a corresponding step width modifier parameter.

23. The method of claim 1, wherein the corresponding step width value corresponds to each of two or more encoded streams comprising a base encoded stream and one or more enhancement layer encoded streams.

24. The method according to any one of claims 21 to 23, wherein the step width modifier parameter varies according to the enhancement level.

25. The method of claim 24, wherein the step width modifier change causes a smaller step width to be used for the first level coded stream and a larger step width to be used for the base coded stream.

26. The method according to any one of claims 21 to 23, wherein the quantization operation uses a quantization matrix defined by a corresponding set of step width parameter values ​​for each coefficient and enhancement level.

27. The method of claim 26, wherein the quantization matrix is ​​preset at at least one of the encoder and the decoder, or wherein the quantization matrix is ​​at least one of the following: being transmitted between the encoder and the decoder, and being dynamically constructed at at least one of the encoder and the decoder.

28. The method of claim 26, wherein the method further comprises constructing the quantization matrix based on at least one of one or more stored parameters and one or more transmitted parameters.

29. The method of claim 26, wherein the scaled transform coefficients d[x][y] are derived according to the following formula, where x = 0…nTbS⁻¹, y = 0…nTbS⁻¹, and a given quantization matrix qm[x][y] is provided: d[x][y]= (TransformCoeffQ[x][u] * ((qm[x+(levelIdxSwap* nTbS)][y]+stepWidthModifier[x][y])+appliedOffset[x][y]), Where TransformCoeffQ is an array of (nTbS) x (nTbS) sizes of quantized transform coefficients containing entropy decoding. `levelIdx` is a variable that specifies the index of the enhanced sub-layer, and `appliedOffset[x][y]` and `stepWidthModifier[x][y]` are variables. in, `appliedOffset[x][y]` represents an array of functions that serve as the dead zone, and `stepWidthModifier[x][y]` represents an array of modification parameters used to modify the quantization step width.

30. The method of claim 29, wherein the variable stepWidthModifier[x][y] is derived according to the following formula: If dequant_offset_signalled_flag = 0, then stepWidthModifier[x][y] = ((((Floor(-Cconst*Ln(qm[x+(1evelIdxSwap*nTbS)][y])))+Dconst)*(qm[x+(1evelIdxSwap*nTbS)][y]2))) / 32768), where Cconst and Dconst are constants. in, `dequant_offset_signalled_flag` represents a flag indicating whether the offset method and offset parameter values ​​are applied during dequantization, and `1evelIdxSwap` represents the result of determining whether the `levelIdx` variable is 2.

31. The method of claim 1, further comprising transmitting the plurality of encoded streams along with signaling information indicating whether a dequantization offset will be applied during the performance of a dequantization operation.

32. The method of claim 31, wherein the offset is at least one function of the quantization step width.

33. The method of claim 31, wherein the offset is a function of the probability distribution of the values ​​before quantization.

34. The method of claim 31, further comprising applying a modification to the quantization step width to generate a modified quantization step width, and applying the modified quantization step width to quantize one or more coefficients.

35. The method of claim 34, wherein the modification is determined based on the probability distribution of the value prior to quantization.

36. The method of claim 1, wherein the quantization operation includes quantizing the coefficients using a linear quantizer, wherein the linear quantizer uses a variable-size dead zone.

37. The method of claim 36, wherein the size of the dead zone is set to a predetermined multiple of the step width used in the quantization operation.

38. The method according to any one of claims 31 to 37, wherein the size of the dead zone is set as a nonlinear function of the step width used in the quantization operation.

39. The method according to any one of claims 31 to 37, wherein the size of the step width used in the quantization operation is variable, and wherein the size of the dead zone is adjusted according to the variable step width.

40. The method of any one of claims 31 to 37, wherein the size of the dead zone is set by multiplying the step width used in the quantization operation by a multiplier parameter, and wherein the value of the multiplier parameter varies based on data indicating the operating conditions under which the encoding is performed.

41. The method according to any one of claims 31 to 37, wherein the size of the dead zone is calculated as a multiple of the quantization step width according to the formula Dead Zone = DZmultiplier x StepWidth, wherein the dead zone multiplier DZmultiplier is calculated as a linear function of StepWidth according to DZmultiplier = a * StepWidth + b, where a and b are constants.

42. The method of claim 1, wherein the quantization operation is performed according to one or more quantization parameters.

43. The method of claim 42, wherein the one or more quantization parameters are set to control one or more encoded streams.

44. The method of claim 42 or claim 43, wherein the one or more quantization parameters are set to provide the desired bit rate in one or more encoded streams.

45. The method of claim 44, wherein the desired bit rate is a common bit rate for all streams to generate a common encoded stream, or wherein different bit rates are provided for different encoded streams.

46. ​​The method of claim 42 or claim 43, wherein the one or more quantization parameters are set to provide a desired quality level or to maximize the quality level within a set of predefined bit rate constraints.

47. The method of claim 42 or claim 43, further comprising determining the quantization parameters by receiving the state of a buffer that receives the one or more encoded streams and the underlying encoded stream; and by using the state to determine quantization parameters.

48. The method of claim 47, wherein the buffer is used for at least one of the following operations: storing and combining an encoded base stream and an encoded enhancement stream, and is configured to receive input at a variable bit rate while reading output at a constant rate, wherein the rate controller reads the state from the buffer to ensure that the buffer does not overflow or become empty, and that data is always available for reading at its output, and wherein the state of the buffer is used to generate the one or more quantization parameters.

49. The method of claim 48, wherein the one or more quantization parameters are controlled based on the amount of data in the buffer.

50. The method of claim 48 or 49, wherein the value of the quantization parameter is negatively correlated with the amount of data in the buffer.

51. The method of claim 42 or claim 43, wherein the quantization parameter is determined for at least one of the following: each frame, the residual, and each residual group.

52. The method of claim 51, wherein the quantization parameters of the frame are determined using a previous set of quantization parameters based on the target data size of the frame and the current data size of the frame.

53. The method of claim 1, wherein the quantization parameters are based on a prior set of quantization parameters.

54. The method of claim 42 or claim 43, further comprising defining a set of curves to map a normalized size onto the one or more quantization parameters, wherein each curve includes one or more of a multiplier and an offset depending on the nature of the current frame.

55. The method of claim 54, wherein the multiplier is applied to a size normalization function, the size normalization function being a function of the quantization parameters.

56. The method of claim 55, wherein Q is used t-1 The current size of the encoded frame t and Q t-1 Points within the space of the set of curves are defined, and the set of points closest to the curves is selected from the set of curves using the points.

57. The method of claim 56, wherein the set of closest curves, together with the points, is used in an interpolation function to determine a new curve corresponding to the points, and wherein a multiplier and an offset of the determined new curve are determined, the method further comprising using the values ​​of the multiplier and the offset of the determined new curve values, together with a received target size, to determine Q. t The value of .

58. The method of claim 57, wherein the set of curves is stored in an accessible memory and updated based on the set of curves determined for a previous frame.

59. A method for decoding an encoded stream into a reconstructed output signal, the method comprising: Receive a first output signal decoded from a first basic encoded stream according to a first codec; Receive the coded stream at the layer level; Decode the hierarchical encoded stream to obtain the residual set; as well as, The residual set is combined with the first output signal to generate the reconstructed signal. Decoding the hierarchical encoded stream includes: Determine whether a dequantization offset should be applied during dequantization based on signaling information; Decode the set of quantized coefficients from the hierarchical encoded stream; and Dequantize the set of quantized coefficients; The quantization operation uses a quantization offset, such that dequantizing the quantized coefficients involves applying the dequantization offset to the quantized coefficients; and The quantization offset is determined such that the dequantization offset is derived as a function of the quantization step width.

60. The method according to claim 59, wherein: The hierarchical encoded stream is a first-level encoded stream; The set of quantized coefficients is the first set of quantized coefficients; as well as, The residual set is a first residual set, and the method further includes: Receive the second-level encoded stream; Decode the second-level encoded stream to obtain the second residual set; and The second residual set is combined with the upsampling pattern of the reconstructed signal to generate a reconstruction of the original resolution input signal. Decoding the second-level encoded stream includes: Decode the second set of quantized coefficients from the second-level encoded stream; Dequantize the second set of quantized coefficients.

61. The method of claim 59 or 60, wherein the dequantization uses a linear quantizer with a non-centered dequantization offset.

62. The method of claim 61, wherein the value of the dequantization offset is received from the encoder.

63. The method of claim 62, wherein the received offset value is offset and added to the received quantized value before step-width-based dequantization.

64. The method of claim 59 or 60, wherein the signaling information further instructs the decoder to modify how the step width modifier parameter is derived during dequantization, wherein, The step width modifier is used to modify the dequantization step width used in dequantization.

65. The method of claim 59, wherein the dequantization offset is a function of the quantization step width.

66. The method of claim 59, wherein using the dequantization offset comprises applying the dequantization offset to a plurality of quantization block groups, the plurality of quantization block groups having a defined step width to adjust one or more corresponding values ​​in the plurality of quantization block groups by the value of the dequantization offset.

67. The method of claim 66, wherein the value corresponding to each of the plurality of quantization block groups is adjusted.

68. The method of claim 67, wherein either or both of the value corresponding to the beginning of the first block group of the plurality of block groups and the value corresponding to the end of the last block group of the plurality of block groups are maintained without adjustment of the value by the dequantization offset.

69. The method of claim 59, wherein the value of the dequantization offset is configurable.

70. The method of claim 59, wherein the value of the dequantization offset is changed based on data indicating the operating conditions under which the encoding is performed.

71. The method of claim 59, further comprising receiving a dequantized offset value transmitted in the encoded stream.

72. The method of claim 59, wherein the dequantization operation includes adding a quantization offset value to the residual or coefficient according to the quantization step width before quantization, wherein, The value of the dequantization offset is adjusted based on the sign of the residual or coefficient.

73. The method according to claim 59, wherein, When the value of the dequantization offset is set to a first predetermined value, the application of the offset to the quantization block group value is deactivated.

74. The method of claim 59, wherein the value of the dequantization offset is adjusted based on the defined width of the dead zone.

75. The method of claim 59, further comprising deriving a corresponding dequantization offset δ for each of the encoded coefficient groups from a single reference dequantization offset δ corresponding to the transmitted signal reference StepWidth of a given data layer. i The offset value δ i The quantization StepWidth depends at least in part on the reference offset δ and the coefficient set i. i .

76. The method of claim 59, wherein the offset of all quantization steps is derived from the reference step width according to the following relationship: Dequantization_Offsetk=f(StepWidth k ), and the following relationships: Actual_Dequantization_Offset i =Reference_Dequantization_Offset+f(StepWidth i )-f(Reference_StepWidth) in, StepWidth k Dequantization_Offset represents the quantization step width associated with a set of coefficients k. k Actual_Dequantization_Offset represents the dequantization offset associated with a set of coefficients k. i Represents the dequantization offset associated with a set of coefficients i, Reference_StepWidth represents the quantization step width of the transmitted reference for a given data layer, Reference_Dequantization_Offset represents the reference dequantization offset associated with the quantization step width of the transmitted reference, and f represents a function.

77. The method of claim 76, wherein f(StepWidth) is a logarithmic relationship.

78. The method of claim 59, wherein the step width varies for each of one or more of the coefficients in the set of coefficients.

79. The method of claim 78, wherein the step width variation causes a smaller step width value to be used for one or more of the coefficients that are predetermined to significantly influence the perception of the decoded signal.

80. The method of claim 59, wherein a default value is assigned to the step width according to the base step width parameter, and wherein one or more modified step widths are obtained according to the base step width and the step width modifier parameter.

81. The method of claim 80, wherein the step width for each of one or more of the coefficients is modified using a corresponding step width modifier parameter.

82. The method of claim 59, wherein the corresponding step width value corresponds to each of two or more encoded streams comprising a base encoded stream and one or more hierarchical encoded streams.

83. The method according to any one of claims 80 to 82, wherein the step width modifier parameter varies according to the enhancement level.

84. The method of claim 83, wherein the step width modifier change causes a smaller step width to be used for the first level encoded stream and a larger step width to be used for the other encoded stream.

85. The method according to any one of claims 80 to 82, wherein the dequantization uses a quantization matrix defined by a set of corresponding step width parameter values ​​for each coefficient and enhancement level.

86. The method of claim 85, wherein the quantization matrix is ​​preset at at least one of the encoder and the decoder, or wherein the quantization matrix is ​​at least one of: transmitting signals between the encoder and the decoder, and dynamically constructed at at least one of the encoder and the decoder.

87. The method of claim 85, wherein the method further comprises constructing the quantization matrix based on at least one of one or more stored parameters and one or more transmitted parameters.

88. The method of claim 85, wherein the scaled transform coefficients d[x][y] are derived according to the following formula, where x = 0…nTbS⁻¹, y = 0…nTbS⁻¹, and a given quantization matrix qm[x][y] is provided: d[x][u]= (TransformCoeffQ[x][y] * ((qm[x+(levelIdxSwap* nTbS)][u]+stepWidthModifier[x][y])+appliedOffset[x][y]), Where TransformCoeffQ is an array of (nTbS) x (nTbS) sizes of quantized transform coefficients containing entropy decoding. `levelIdx` is a variable that specifies the index of the enhanced sub-layer, and `appliedOffset[x][y]` and `stepWidthModifier[x][y]` are variables. in, `appliedOffset[x][y]` represents an array of functions that serve as the dead zone, and `stepWidthModifier[x][y]` represents an array of modification parameters used to modify the quantization step width.

89. The method of claim 88, wherein the variable stepWidthModifier[x][y] is derived according to the following formula: If dequant_offset_signalled_flag = 0, then stepWidthModifier[x][y] = ((((Floor(-Cconst*Ln(qm[x+(1evelIdxSwap*nTbS)][y])))+Dconst)*(qm[x+(1evelIdxSwap*nTbS)][y]2))) / 32768), where Cconst and Dconst are constants. in, `dequant_offset_signalled_flag` represents a flag indicating whether the offset method and offset parameter values ​​are applied during dequantization, and `1evelIdxSwap` represents the result of determining whether the `levelIdx` variable is 2.

90. The method of claim 59, further comprising receiving the plurality of encoded streams along with signaling information indicating whether a dequantization offset will be applied during dequantization.

91. The method of claim 90, wherein the offset is at least one function of the quantization step width.

92. The method of claim 90, wherein the offset is a function of the probability distribution of the values ​​before quantization.

93. The method of claim 90, further comprising applying a modification to the quantization step width to generate a modified quantization step width, and applying the modified quantization step width to dequantize one or more coefficients.

94. The method of claim 93, wherein the modification is determined based on the probability distribution of the value prior to quantization.

95. The method of claim 59, wherein the dequantization comprises dequantizing the coefficients using a linear quantizer, wherein the linear quantizer uses a variable-size dead zone.

96. The method of claim 95, wherein the size of the dead zone is set to a predetermined multiple of the step width used in the dequantization.

97. The method according to any one of claims 90 to 96, wherein the size of the dead zone is set as a nonlinear function of the step width used in the dequantization.

98. The method according to any one of claims 90 to 96, wherein the size of the step width used in the dequantization is variable, and wherein the size of the dead zone is adjusted according to the variable step width.

99. The method according to any one of claims 90 to 96, wherein the size of the dead zone is set by multiplying the step width used in the dequantization by a multiplier parameter, and wherein the value of the multiplier parameter varies based on data indicating the operating conditions under which the encoding is performed.

100. The method according to any one of claims 90 to 96, wherein the size of the dead zone is calculated as a multiple of the quantization step width according to the formula Dead Zone = DZmultiplier x StepWidth, wherein the dead zone multiplier DZmultiplier is calculated as a linear function of StepWidth according to DZmultiplier = a * StepWidth + b, where a and b are constants.

101. The method of claim 59, wherein the dequantization is performed according to one or more quantization parameters.

102. The method of claim 101, wherein the one or more quantization parameters are set to control one or more encoded streams.

103. The method of claim 101, wherein the one or more quantization parameters are set to provide the desired bit rate in one or more encoded streams.

104. The method of claim 103, wherein the desired bit rate is a common bit rate for all streams to generate a common encoded stream, or wherein different bit rates are provided for different encoded streams.

105. The method according to any one of claims 101 to 103, wherein the one or more quantization parameters are set to provide a desired quality level or maximize the quality level within a set of predefined bit rate constraints.

106. The method according to any one of claims 101 to 103, comprising determining the quantization parameters by receiving a state of a buffer that receives the one or more encoded streams and the underlying encoded stream; and determining the quantization parameters using the state.

107. The method according to any one of claims 101 to 103, wherein the quantization parameter is determined for at least one of the following: each frame, residual, and each residual group.

108. The method of claim 107, wherein the quantization parameters of the frame are determined using a previous set of quantization parameters based on the target data size of the frame and the current data size of the frame.

109. The method of claim 107, wherein the quantization parameter is based on a prior set of quantization parameters.

110. The method of any one of claims 101 to 104, comprising defining a set of curves to map a normalized size onto the one or more quantization parameters, wherein each curve comprises one or more of a multiplier and an offset depending on the nature of the current frame.

111. The method of claim 110, wherein the multiplier is applied to a size normalization function, the size normalization function being a function of the quantization parameters.

112. The method of claim 111, wherein Q is used t-1 The current size of the encoded frame t and Q t-1 Points within the space of the set of curves are defined, and the set of points closest to the curves is selected from the set of curves using the points.

113. The method of claim 112, wherein the set of closest curves, together with the points, is used in an interpolation function to determine a new curve corresponding to the points, and wherein a multiplier and an offset of the determined new curve are determined, the method further comprising using the values ​​of the multiplier and the offset of the determined new curve values, together with a received target size, to determine Q. t The value of .

114. The method of claim 113, wherein the set of curves is stored in an accessible memory and updated based on the set of curves determined for a previous frame.

115. An encoder for encoding input video, the encoder being configured to perform the method according to any one of claims 1 to 58.

116. A decoder for decoding an encoded stream into a reconstructed output video, the decoder being configured to perform the method according to any one of claims 59 to 114.

117. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 114.

Images