Quantization of residuals in video coding

Abstract

The invention relates to a method of encoding an input signal into a plurality of encoded streams, wherein the encoded streams can be combined to reconstruct the input signal. The method comprises receiving the input signal; down-sampling the input signal to create a down-sampled signal; instructing to encode the down-sampled signal using a base encoder to create a base encoded stream; instructing to decode the base encoded stream using a base decoder to generate a reconstructed signal; comparing the reconstructed signal with the input signal to create a set of residuals; and encoding the set of residuals to create an encoded stream, which comprises applying a transform to the set of residuals to create a set of transformed coefficients, applying a quantization operation to the set of transformed coefficients to create a set of quantized coefficients, and applying an encoding operation to the quantized coefficients. Wherein the quantization operation is performed based on temporal information associated with the set of transformed coefficients. The invention also relates to a method of decoding an encoded stream into a reconstructed output signal.

Description

[0001] This application is a divisional application of Chinese patent application No. 202080061959.X, filed on July 6, 2020, entitled "Quantification of Residuals in Video Decoding". Technical Field

[0002] This invention relates to methods for data compression, specifically for compressing and decoding image and video signals. Data compression may include, but is not limited to, acquiring, exporting, encoding, outputting, receiving, decoding, and reconstructing data encoded using a hierarchical (layer-based) decoding format, wherein video signals are encoded in data tiers (e.g., layers or hierarchies) and decoded in subsequent layers of higher quality. Different layers of the signal may also be encoded according to different decoding formats. Background Technology

[0003] Hybrid backward-compatible decoding techniques have previously been proposed in, for example, WO 2014 / 170819 and WO 2018 / 046940 (the contents of which are incorporated herein by reference).

[0004] One proposed method involves parsing a data stream into a first portion of encoded data and a second portion of encoded data; implementing a first decoder to decode the first portion of encoded data into a first reproduction of a signal; implementing a second decoder to decode the second portion of encoded data into reconstructed data, the reconstructed data specifying how to modify the first reproduction of the signal; and applying the reconstructed data to the first reproduction of the signal to produce a second reproduction of the signal.

[0005] A further proposed method involves using a set of residual features to reconstruct a reproduction of a first time-series sample of the signal. A set of spatiotemporal correlation features associated with the first time-series sample is generated. This set of spatiotemporal correlation features indicates the degree of spatial correlation between multiple residual features and the degree of temporal correlation between the first reference data based on the reproduction and the second reference data based on the reproduction of a second time-series sample of the signal. The set of spatiotemporal correlation features is used to generate output data. As mentioned, the residual set is encoded to reduce the overall data size.

[0006] Encoding applications typically employ quantization operations. This compression process, by compressing each of one or more ranges of data values ​​into a single value, allows for a reduction in the number of distinct values ​​in the set of video data, thereby making the data easier to compress. In this way, quantization schemes can be used in some videos to transform signals into quanta, allowing specific variables to take only specific discrete values. Typically, video codecs divide visual data in the form of video frames into discrete blocks, usually of a predetermined size or number of pixels. A transformation is then typically applied to these blocks to express the visual data based on the sum of frequency components. The transformed data can then be pre-multiplied by a quantization bit code and then divided element-wise by the quantization matrix, with each pre-multiplied element divided by a matrix element to obtain an output element that is then rounded. Dividing the different transformed elements by a divisor (i.e., different elements of the quantization matrix) is often used to allow those frequency elements that have a greater impact on the viewer's visual appearance of the video to be efficiently allocated more data or resolution compared to less perceptible components.

[0007] The goal is to optimize for further reducing the overall data size, while balancing the objective of not compromising the user's overall impression once the signal has been reconstructed with optimizing processing speed and complexity.

[0008] Most data compression methods involve quantization levels that are typically applied in a domain of transform coefficients that are reversible relative to the display settings coordinates (e.g., luminance and chrominance values, RGB values, etc.).

[0009] In the following description, for simplicity, we will refer to video signals in all embodiments. However, it should be understood that the same embodiments, with necessary modifications, can also be applied to other types of data.

[0010] Quantization—which can be performed in several different ways—is a so-called "lossy" operation because it is most commonly used to reduce the information entropy of a signal, resulting in a compressed signal that, when decoded, is similar to but different from the original signal. Thus, it is useful to apply coarser quantization to regions of the signal that are considered to have lower priority (e.g., where fidelity relative to the original signal is less important) and finer quantization to regions of the signal that are considered to have higher priority (e.g., where fidelity relative to the original signal is more important). Quantization is typically controlled by the quantization step size, where a smaller quantization step size corresponds to finer quantization and a larger quantization step size corresponds to coarser quantization.

[0011] Traditional DCT-based coding formats address this problem by employing adaptive quantization: different decoding units (e.g., blocks) within the same image (or frame) can be decoded according to different QP (quantization parameter) values, effectively driving the concatenation of different quantization steps for each transform coefficient of blocks with different QPs. These methods produce substantial improvements in compression, but they require consuming bits to transmit so-called "QP mappings".

[0012] Unlike DCT-based coding formats, hierarchical decoding formats are characterized by much smaller decoding units, typically containing as few as 2×2 or 4×4 pixels (in contrast to the up to 128×128 pixels of decoding units used in DCT-based codecs). Therefore, adaptive quantization of the base signaling information per block would be costly. Furthermore, hierarchical decoding methods are generally targeted at software applications, and decoding additional data layers adds extra processing to both the encoding and decoding processes, which would diminish the compression benefits of this approach in several ways.

[0013] The method described in this paper allows for efficient driving of adaptive quantization levels for hierarchical coding formats without requiring a dedicated communication layer for adaptive quantization mapping. Summary of the Invention

[0014] According to one aspect, a method for encoding an input signal into a plurality of coded streams, wherein the coded streams can be combined to reconstruct the input signal, the method comprising: receiving the input signal; downsampling the input signal to create a downsampled signal; instructing a base encoder to encode the downsampled signal to create a base coded stream; instructing a base decoder to decode the base coded stream to generate a reconstructed signal; comparing the reconstructed signal with the input signal to create a set of residuals; and encoding the set of residuals to create the coded stream, comprising: applying a transform to the set of residuals to create a set of transform coefficients; applying a quantization operation to the set of transform coefficients to create a set of quantized coefficients; and applying an encoding operation to the quantized coefficients.

[0015] According to another aspect, a method for decoding an encoded stream into a reconstructed output signal can be provided, the method comprising: receiving a first output signal decoded from a first base encoded stream according to a first codec; receiving a hierarchical encoded stream; decoding the hierarchical encoded stream to obtain a set of residuals; and combining the set of residuals with the first output signal to generate a reconstructed signal, wherein decoding the hierarchical encoded stream comprises: decoding a set of quantized coefficients from the hierarchical encoded stream; and dequantizing the set of quantized coefficients, wherein the dequantization of the set of quantized coefficients is based on time information associated with the hierarchical encoded stream. The combination may include an upsampling pattern combination with the first output signal. The hierarchical coded stream can be a first-level coded stream; the set of quantized coefficients can be a first set of quantized coefficients; and the residual set can be a first residual set, and the method may further include: receiving a second-level coded stream; decoding the second-level coded stream to obtain a second residual set; and combining the second residual set with an upsampling pattern of the reconstructed signal to generate a reconstruction of the original resolution input signal, wherein decoding the second-level coded stream includes: decoding a second set of quantized coefficients from the second-level coded stream; and dequantizing the second set of quantized coefficients.

[0016] The method advantageously allows for improved efficiency in the encoding and decoding processes by means of varying the degree and / or manner of compression applied to coefficients during quantization based on changes in any of several factors, according to the video data to be decoded. Thus, the manner in which a typically lossy quantization procedure is performed during the encoding of the video stream can be adapted to apply an appropriate balance between encoding or compression efficiency and visually perceptible compression of the input video, depending on the nature and content of the input video—a relationship that can vary considerably across different video frames and streams. This adaptable form of quantization can be used in conjunction with the dequantization process at the receiving decoder, for example, by signaling to the decoder, via a transmission of parameters having values ​​representing or indicating the information, how quantization has been performed or the degree to which quantization has been changed from the default mode. Specifically, the applied offset can advantageously be used to modify the entropy of the residual data during encoding to improve compression efficiency.

[0017] In some embodiments, using a quantization offset includes applying the quantization offset to a plurality of quantization block groups having a defined step width to adjust each of one or more corresponding values ​​in the plurality of quantization block groups by means of or based on the value of the quantization offset.

[0018] In these embodiments, the value corresponding to each of the plurality of quantization block groups may be adjusted. Alternatively, any one or both of the values ​​corresponding to the beginning of the first block group and the end of the last block group may be left unadjusted by a quantization offset; that is, they remain unadjusted by a quantization offset. The first block group can be understood as corresponding to the numerical minimum value or the minimum value of the range. Similarly, the last block group can be understood as representing the maximum value of the range or the numerical maximum value. These adjustments and non-adjustments may be combined with quantization operations involving dead zones and block group folding, as described later in this disclosure.

[0019] Typically, the quantization offset value is adjustable or configurable. In some embodiments, the quantization offset value may vary based on data indicating the operating conditions under which encoding is performed.

[0020] In some embodiments, the method further includes signaling a quantization offset value to a decoder that will receive the encoded stream. This signaling may be performed, for example, in an embodiment where the quantization offset value changes dynamically during encoding.

[0021] Quantization operations typically involve subtracting the quantization offset from the residual or coefficient value before quantization based on the quantization step width.

[0022] In some embodiments, the value of the quantization offset is adjusted based on the sign of the residual or coefficient. This can be implemented to allow symmetric operations with respect to zero values.

[0023] The method can be executed such that when the quantization offset value is set to a first predetermined value, the application of the offset to the block group value is deactivated. For example, this can be done by setting the quantization or dequantization offset value to zero.

[0024] In some embodiments, the quantization offset value is adjusted based on the defined width of the dead zone. In these embodiments, the dead zone can be used to perform quantization operations, as detailed later in this disclosure. The quantization offset can be configured or adjusted based on the defined width of the dead zone.

[0025] It is also recognized that temporal information can be advantageously used to further improve the quantization process. It is more likely to utilize residual data about the "temporally stable" region for multiple frames in a video sequence, and therefore its entropy cost can be "amortized" over multiple frames. Furthermore, it is more likely to be understood by the viewer (who will have more time to assess its fidelity relative to the more "brief" regions of the video sequence).

[0026] For example, a "temporally stable" region in a video includes those portions of a frame that are static and / or quasi-static relative to one or more preceding and / or one or more subsequent frames. By way of a non-limiting example, the background scene in a news report video can be relatively stable in time across multiple frames. "Temporally stable" can also include portions of a frame that are easily predicted using motion compensation between frames. For example, portions of a frame that are translated between frames without significantly altering their shape / form can also be "temporally stable." "Temporally stable" regions are typically temporally predictable from one or more preceding and / or subsequent frames. On the other hand, a "transient" region in a video includes those portions of a frame that are not "temporally stable" and change (e.g., are dynamic) between frames. These changes can be substantial or not substantial. "Transient" regions are typically not temporally predictable from one or more preceding and / or subsequent frames.

[0027] Temporal signaling can be effectively used to drive additional decoding efficiency and improve visual quality: Although a single main quantization step is signaled for each enhancement level or residual data “echelon,” the decoder typically produces different (e.g., larger) quantization steps for regions of the image that are not temporally predicted. In this way, no bits or processing are spent on additional signaling, fewer bits are spent on transient residuals, and therefore relatively more bits are spent on residuals that are utilized / reused (and understood by the viewer) for multiple frames.

[0028] In some embodiments, a mapping of time-related information between one or more previous frames and the current frame is utilized to increase the quantization step size of coefficients corresponding to non-temporally predicted (e.g., by means of a non-limiting instance, non-static) regions of the image, or similarly, to decrease the quantization step size of coefficients corresponding to temporally predicted (e.g., by means of a non-limiting instance, static or quasi-static) regions of the image. In one non-limiting embodiment, the temporal data layer transmits regions to be predicted based on residual data echelons of previously corresponding regions, in a manner that, for example, these regions are generated by means of at least partially based on the contents of a temporal buffer. In this way, non-temporally predicted regions use different quantization step sizes relative to temporally predicted regions, according to a known relationship (e.g., a formula) between both the encoder and decoder.

[0029] In some embodiments, the time-predicted region is a static, quasi-static, or motion-compensated region, and the residual data of the time-predicted region is obtained by combining the corresponding values ​​of the corresponding regions in one or more previous frames (e.g., stored in a time buffer) with values ​​obtained by dequantizing the corresponding quantization coefficients from the encoded data stream, wherein the dequantization process is based on the main step width of the residual data echelon; conversely, the residual data of the untime-predicted region is obtained by dequantizing the corresponding quantization coefficients from the encoded data stream, wherein the dequantization process is based on the step width obtained from the main step width of the residual data echelon modified according to a formula known to both the encoder and the decoder.

[0030] In some embodiments, the time-predicted region is a static, quasi-static, or motion-compensated region, and the residual data of the time-predicted region is obtained by combining the corresponding values ​​of the corresponding regions in one or more previous frames (e.g., stored in a time buffer) with values ​​obtained by dequantizing the corresponding quantization coefficients from the encoded data stream, wherein the dequantization process is based on the main step width of the residual data echelon modified according to a formula known to both the encoder and the decoder; conversely, the residual data of the untime-predicted region is obtained by dequantizing the corresponding quantization coefficients from the encoded data stream, wherein the dequantization process is based on the step width obtained from the main step width of the residual data echelon.

[0031] In other embodiments, the time-predicted region is a static region, a quasi-static region, or a motion-compensated (e.g., if a motion-compensated algorithm is used) region. Accordingly, different quantization step widths can be used for any of the above regions (static, quasi-static, motion-compensated, and non-time-predicted regions) according to formulas that may be known to both the encoder and decoder. In a non-limiting instance, the static, quasi-static, or motion-compensated (if present) regions may have the same quantization step width as (e.g., lower than) that used for the non-time-predicted region. In another non-limiting instance, each of these four regions (static, quasi-static, motion-compensated (if present), and non-time-predicted) may have a different step width. For example, the static step width may be lower than the quasi-static step width, and both may be lower than the step width of the motion-compensated region, while they are all lower than the step width of the non-time-predicted region. However, it should be understood that other combinations are possible, and in fact, the scheme allows for flexibility in determining which formulas will be best applied to a particular sequence and / or type of region.

[0032] In some embodiments, the encoder may optionally signal parameters to be used—instead of default parameters—to the decoder to derive the quantization stride of the non-time-predicted region and / or the quantization stride of the time-predicted region. In some non-limiting embodiments, the quantization stride of the non-time-predicted region is derived from the quantization stride of the time-predicted region using a linear relationship. In other non-limiting embodiments, the quantization stride of the non-time-predicted region is derived from the quantization stride of the time-predicted region using a non-linear relationship. In other non-limiting embodiments, the quantization stride of the non-time-predicted region is derived from the quantization stride of the time-predicted region using a lookup table.

[0033] According to a second non-limiting embodiment, a mapping of time-related information between the residual data echelon of the current frame and the corresponding residual data echelon of subsequent frames is utilized to increase the quantization step size of coefficients corresponding to non-temporally predicted (e.g., by means of a non-limiting instance, non-static) regions of the image, or similarly, to decrease the quantization step size of coefficients corresponding to temporally predicted (e.g., by means of a non-limiting instance, static or quasi-static) regions of the image. In one non-limiting embodiment, the temporal data layer signals the residual data echelon of subsequent frames, at least in part, based on regions generated from the content of previous frames (e.g., stored in a temporal buffer) that include the current frame. The region of the residual data echelon of the current frame (“transient residual”) that corresponds to the region of the subsequent frame that will generate residuals based at least in part on the corresponding region in the current and / or previous frames (e.g., as stored in the time buffer) uses a different quantization step width than the region of the residual data echelon of the current frame (“temporally stable residual”) that corresponds to the region of the subsequent frame that will be predicted based on the corresponding region in the current and / or previous frames (e.g., as stored in the time buffer), the formula being known to both the encoder and the decoder, for obtaining the quantization step width of the transient residual (or temporally stable residual) from the main step width of the residual data echelon.

[0034] In some embodiments, the encoder generates signaling information relative to which regions of the residual data echelon should be decoded, at least in part, based on one or more residual data from previous or subsequent frames (e.g., stored in a time buffer). In a non-limiting embodiment, the encoder determines, according to a formula, whether to signal regions of the residual data echelon as data to be predicted at least in part based on one or more residual data from previous or subsequent frames (e.g., stored in a time buffer), the formula assuming that the residual data of regions generated at least in part based on one or more residual data from previous or subsequent frames (e.g., stored in a time buffer) is quantized in a manner different from the residual data generated independently of the one or more residual data from previous or subsequent frames (e.g., stored in a time buffer). In one non-limiting embodiment, the encoder employs a rate-distortion optimization (RDO) formula that estimates the distortion of two alternatives (i.e., time-predicted versus independently generated) and the required rate for a given main quantization step width. The rate of the time-predicted alternative is estimated using the given main quantization step width, while the rate of the independently generated alternative is calculated from the main quantization step width using a larger quantization step width (“residual intra-quantization step width”) according to a formula known to both the encoder and decoder. In one non-limiting embodiment, the encoder generates parameters that modulate the formula, producing a residual intra-quantization step width based on a given total quantization step width; these modulation parameters are signaled to the decoder as decoding metadata when they differ from the decoder's known default parameters, allowing the decoder to generate the correct intra-patch quantization step width. In one non-limiting embodiment, the encoder generates modulation parameters at least partially based on the average percentage of the time-predicted image. In one non-limiting embodiment, the encoder generates modulation parameters at least partially based on the estimated number of frames to be used from one or more portions of the time buffer. In one non-limiting embodiment, the encoder generates modulation parameters at least partially based on a saliency map, thereby estimating the relative importance of different regions of the image.

[0035] In some embodiments, the formula for generating the residual in-quantization step width from the main quantization step width is linear, as shown in the following equation:

[0036] Intra-residuals_quantization_step-width = master_quantization_step- width * (1 + step-width_modifier / c)

[0037] The value of c depends on the number of bits used in the step-width_modifier parameter, where c is a constant. For example, if 8 bits are used in the step-width_modifier, c will be equal to the maximum percentage increase between the main quantization step width and the residual in-quantization step width divided by 255. In some non-limiting embodiments, when the step-width_modifier is not signaled to the decoder, the decoder uses a default value instead.

[0038] Alternatively, the rate of alternatives can be independently generated by estimating the rate of the given main quantization step width, while the rate of the time-predicted alternatives is calculated from the main quantization step width using a smaller quantization step width (“inter-residual quantization step width”) according to a formula known to both the encoder and decoder. In a non-limiting embodiment, the encoder generates parameters that modulate the formula, producing an inter-residual quantization step width based on a given total quantization step width; these modulation parameters are signaled to the decoder as decoding metadata when they differ from the decoder’s known default parameters, so as to allow the decoder to generate the correct intra-patch quantization step width. In a non-limiting embodiment, the encoder generates modulation parameters at least in part based on the average percentage of the time-predicted image. In a non-limiting embodiment, the encoder generates modulation parameters at least in part based on the estimated number of frames to be used from one or more portions of the time buffer. In a non-limiting embodiment, the encoder generates modulation parameters at least in part based on a saliency map, thereby estimating the relative importance of different regions of the image.

[0039] In some non-limiting embodiments, the formula for generating the inter-residual quantization step width from the main quantization step width is linear, as shown in the following equation:

[0040] Inter-residuals_quantization_step-width = master_quantization_step- width / (1 + step-width_modifier / c)

[0041] The value depends on the number of bits used for the step-width_modifier parameter, where c is a constant.

[0042] In other non-limiting embodiments, the formulas used to generate the intra-residual quantization step width or the inter-residual quantization step width are non-linear formulas.

[0043] More generally, in some embodiments, quantization is performed based on temporal information associated with the positively quantized data or set of coefficients. The temporal information can be information specific to or about the set of coefficients. For example, temporal information may have been derived for each video frame, for instance, using a time prediction module. In this disclosure, the relationship between time and coefficients can be understood as such. In some instances, temporal information describing the temporal predictability of frames corresponding to a given set of coefficients can be obtained. This information may indicate which regions of a particular frame—time-predicted regions—can be predicted at least partially from one or more other frames (previous and / or subsequent), and which regions of a particular frame—non-time-predicted regions—cannot be predicted at least partially from one or more other frames (previous and / or subsequent). This information can then be used by a quantizer performing the quantization operation to determine different quantization parameters for the time-predicted and non-time-predicted regions.

[0044] In some embodiments where time information is used by the user, the method may further include deriving one or more quantization parameters based on the time information. Alternatively or additionally, the method may include determining a first quantization parameter of a first subset of a set of coefficients associated with the first time information. In these cases, a second quantization parameter of a second subset of a set of coefficients associated with second time information may be determined.

[0045] These implementations may further include determining the temporal information associated with the coefficient sets by deriving the correlation between the coefficient sets co-located at different samples. This may be temporal information between one or more previous frames and the current frame, and is used to increase the quantization step width of the coefficients.

[0046] In some of these embodiments, the time information includes or contains an indication of whether a set or subset of coefficients is one of the following: static, quasi-static, and dynamic. Preferably, in some embodiments, a given set / subset is static if the difference between the corresponding co-located set or subset of coefficients in previous and / or subsequent samples, i.e., the difference between a given set or subset in a layer and its corresponding co-located set / subset, has an estimated information entropy that is substantially zero; alternatively, "substantially zero" means a threshold value less than zero such that the difference takes a zero value when the difference is below the threshold.

[0047] If the difference between the corresponding co-located set or subset of coefficients in previous and / or subsequent samples has an estimated information entropy lower than the estimated information entropy of a given set or subset of coefficients in the layer, then the given set or subset of coefficients can be quasi-static. Correspondingly, if the difference between the corresponding co-located set or subset of coefficients in previous and / or subsequent samples has an estimated information entropy substantially the same as or higher than the estimated information entropy of a given set or subset of coefficients in the layer, then the given set or subset of coefficients can be dynamic.

[0048] In some embodiments, the block folding process can also be used to enhance decoding efficiency. Specifically, in these cases, the quantization operation further includes applying a block folding operation to a set of coefficients, wherein each coefficient having a value exceeding a predetermined maximum value is quantized to have a quantized value corresponding to a first quantized block group among a plurality of quantized block groups having a defined step width, the maximum value being defined by an upper limit of the first quantized block group. This can be done to place all residual or coefficient values ​​residing above a selected quantized block group into the selected block group. The first block group can be considered to correspond to an upper value, or a block group corresponding to the highest (absolute) quantized value, relative to what can be understood as the endpoint of the range of values ​​involved. Block folding can be implemented at either the upper or lower endpoints. A similar process can be performed for negative values ​​within the range. Block folding can be configured to adjust or reduce the bit rate based on at least one of network conditions and underlying stream processing. Thus, the block folding process can be configurable itself, for example, in various embodiments, any one or two of conditions and underlying stream processing or parameters derived therefrom are used to configure block folding, such as parameters defining block folding. Block group folding can be considered as a limit, where values ​​above a threshold are set at an upper limit.

[0049] In some embodiments, the method may involve varying the step width used in the quantization operation according to a step width parameter. Specifically, the step width may be varied for each of one or more coefficients in a set of coefficients, for example, for different coefficients within a 2×2 or 4×4 block of coefficients. For instance, the step width may be varied such that a smaller step width value is used for one or more coefficients that are pre-determined to have a greater impact on the perception of the decoded signal. The degree of impact is typically determined experimentally, thereby obtaining information indicating which coefficients have a greater impact on the viewer's perception of the decoded signal.

[0050] The step width is typically assigned a default value based on the base step width parameter. One or more modified step widths can be obtained from the base step width and the step width modifier parameter. For example, this can be done by obtaining the modified step width according to the formula modified_stepwidth = base_stepwidth * modifier, where the modifier can be set based on a specific coefficient within the block or cell.

[0051] In these embodiments, the corresponding step width modifier parameter can be used to modify the step width for each of one or more of the coefficients.

[0052] In some embodiments, a corresponding step width value may be used for, or associated with, each of two or more coded streams or enhancement layers, which include a base coded stream and one or more enhancement layer coded streams.

[0053] In some embodiments, the stride modifier parameter is varied according to the enhancement level, which depends on the enhancement level employed. The stride modifier can be varied such that a smaller stride is used for the first-level coded stream and a larger stride is used for the base coded stream.

[0054] In some preferred embodiments, the quantization operation uses a quantization matrix defined by a set of step width modifier parameter values ​​for different coefficients and different enhancement levels. Therefore, the method may involve a corresponding step width modifier parameter value for each coefficient and each enhancement level. The method can be performed by an encoder and a corresponding decoding process by a decoder, and the quantization matrix can be obtained by various means in different embodiments. Specifically, the quantization matrix can be preset at at least one of the encoder and decoder, or the quantization matrix can be transmitted between the encoder and decoder, and alternatively, the quantization matrix can be dynamically constructed at at least one of the encoder and decoder.

[0055] The method may further include constructing the quantization matrix as a function of at least one of one or more stored and one or more transmitted parameters.

[0056] In some embodiments, the scaled transform coefficients d[x][y] can be derived according to the following formula, where x=0...nTbS−1, y=0...nTbS-1, and a given quantization matrix qm[x][y]:

[0057] d[x][y] = (TransformCoeffQ[x][y] * ((qm[x + (levelIdxSwap * nTbS)][y] + stepWidthModifier[x][y]) + appliedOffset [x][y]),

[0058] Where TransformCoeffQ is an array of (nTbS) x (nTbS) sizes of quantized transform coefficients containing entropy decoding.

[0059] `levelIdx` is a variable that specifies the index of the enhanced sublayer, and `appliedOffset[x][y]` and `stepWidthModifier[x][y]` are variables. `appliedOffset[x][y]` can correspond to the dead zone, as described elsewhere in this document.

[0060] For example, levelIdx can be set to 1 for enhancement sublayer 1 and 2 for enhancement sublayer 2.

[0061] Typically, the derived variable stepWidthModifier[x][y] is defined as follows:

[0062] If dequant_offset_signalled_flag=0, then stepWidthModifier[x][y]=((((Floor(−Cconst*Ln(qm[x+(levelIdxSwap*nTbS)][y])))+Dconst)*(qm[x+(levelIdxSwap*nTbS)][y]2))) / 32768), where Cconst and Dconst are constants and can have values ​​of 5242 and 99614 respectively in a single instance.

[0063] In some embodiments, the quantization operation may include quantizing coefficients using a linear quantizer, wherein the linear quantizer preferably uses a variable-size dead zone. In these embodiments, the size of the dead zone may be set to a predetermined multiple of the step width used in the quantization operation, for example, set to a linear function of the step width value. Alternatively, a nonlinear function of the step width value may be used.

[0064] In some preferred embodiments, the step size used in the quantization operation is variable, and the size of the dead zone is more preferably adjusted according to the variable step size.

[0065] The size of the dead zone can be set by multiplying the step width used in the quantization operation by a multiplier parameter, wherein the value of the multiplier parameter varies based on data indicating the operating conditions under which encoding is performed (e.g., the available bit rate). Therefore, in some embodiments, the multiplier can also be adaptive.

[0066] Quantization is typically performed based on one or more quantization parameters. These parameters are typically set to control and provide a desired bit rate in one or more encoded streams. That is, they can be set to control and / or provide a desired bit rate in one or more encoded streams. The desired bit rate is a common bit rate for all streams to generate a common encoded stream, or it may provide different bit rates for different encoded streams.

[0067] In some embodiments, the one or more quantization parameters are set to provide the desired quality level or maximize the quality level within a set of predefined bit rate constraints.

[0068] The method may include determining quantization parameters by receiving the state of a buffer that receives one or more encoded streams and a base encoded stream, and by using the state to determine quantization parameters.

[0069] The buffer is preferably used to store and / or combine the encoded base stream and the encoded enhancement stream, and is configured to receive input at a variable bit rate while reading output at a constant rate. The rate controller can read the buffer's state to ensure that the buffer does not overflow or become empty, and that data is always available for reading at its output. The buffer's state can also be used to generate the one or more quantization parameters. The one or more quantization parameters can be controlled based on the amount of data within the buffer.

[0070] Typically, the value of the quantization parameter is inversely related to the amount of data in the buffer.

[0071] In some embodiments, quantization parameters are determined for each frame, residual, and / or residual group, that is, at least one of the following: each frame, residual, and residual group. Typically, the quantization parameters of a frame can be determined using a previous set of quantization parameters based on the target data size of the frame and the current data size of the frame. In any of these embodiments, the quantization parameters may be based on a previous set of quantization parameters.

[0072] The method may include defining a set of curves to map a normalized size to one or more quantization parameters, wherein each curve includes one or more of a multiplier and an offset depending on the nature of the current frame. The set of curves may be defined to map a normalized size to quantization parameters. Each curve may have one or more of a multiplier and an offset, the offset of which may depend on the nature of the current frame (e.g., it may depend on the complexity of the information to be encoded intra-frame). The multiplier and offset may define the shape of the curve. The multiplier may be applied to a size normalization function, which is a function of the quantization parameter Q. In one case, the current size (i.e., in Q...) t-1 (size of encoded frame t) and Q t-1 A point can be used to define a space within the set of curves. This point can be used to select a set of curves closest to the given curves. These curves can be curves above and below the point, or the highest or lowest curve of the point. The set of closest curves, along with the point, can be used in an interpolation function to determine a new curve associated with the point. Once this new curve is determined, a multiplier and offset can be determined for the new curve. These values ​​can then be used, along with the received target size, to determine the value of Qt (e.g., the curve can define a function of size and Q). Therefore, typically, a multiplier is applied to a size normalization function, which is a function of the quantization parameter Q. t-1 The current size of the encoded frame t and Q t-1 Points within a space that can be used to define a set of curves, wherein said points are used to select a set of curves that are closest to the given curves. The closest curves can be the curves above and below the point, or the highest or lowest curve of the point.

[0073] In these cases, the set of closest curves is typically used together with the points in the interpolation function to determine a new curve associated with those points, and the multiplier and offset of the determined new curve can be determined, further including using the values ​​of the multiplier and offset of the determined new curve values ​​together with the received target size to determine Q. t The value of .

[0074] The set of curves can be stored in accessible memory and updated based on the set of curves determined for a previous frame. In some cases, adaptive quantization can be applied in different ways for different coefficient positions within a decoding unit or block, such as for different elements in an array of 4 or 16 coefficients (e.g., for a 2×2 or 4×4 transform).

[0075] In some embodiments, quantization is performed using a quantization matrix derived from the obtained values ​​of quantization matrix mode parameters. Preferably, in these cases, the quantization matrix mode parameters specify the quantization matrix to be used in the encoding process.

[0076] Typically, correspondingly different quantization matrices are used for each of two or more layers in the encoded stream. More preferably, different quantization matrices are used for each encoded stream, wherein a default quantization configuration is predetermined, and variations relative to the default configuration are transmitted between the encoder and decoder.

[0077] In these embodiments, the method may include causing different quantization matrices to be used for the corresponding encoded streams by means of at least one of the following: default configuration, and causing a common quantization matrix to be used for the corresponding encoded streams by means of signaling overriding the default configuration.

[0078] In some embodiments, the quantization matrix is ​​used only in one of a plurality of enhancement layers. The quantization matrix is ​​typically indexed according to the position of the coefficients within a block in which the coefficients are arranged.

[0079] In some embodiments, the base quantization matrix is ​​defined by a set of values ​​and modified according to a scaling factor, which is a function of the stride width of one or more enhancement layers. For example, the scaling factor may be computed as a clamping function of the stride width parameter. In some embodiments, each entry in the quantization matrix is ​​scaled using an exponential function of the scaling factor.

[0080] According to a second aspect, a method is provided for decoding an encoded stream into a reconstructed output signal, the method comprising: receiving a first output signal decoded from a first underlying encoded stream according to a first codec; and combining a set of residuals with the first output video to generate a decoded video, wherein the received encoded stream includes a group of transform coefficients scaled using a linear quantizer, and the decoding includes performing a dequantization operation, which can be understood as applying dequantization to the quantized data, wherein the linear quantizer uses a non-centered dequantization offset, wherein the value of the dequantization offset is received from an encoder. The combination may include a combination with an upsampling pattern of the first output signal. The linear quantizer may use different sizes of dead zones relative to the quantization step size and the non-centered dequantization offset.

[0081] Typically, the received offset value is added to the received quantized value before the step-width-based dequantization.

[0082] According to a third aspect, a method is provided for decoding an encoded stream into a reconstructed output signal, the method comprising: receiving a first output signal decoded from a first base encoded stream according to a first codec; and combining a set of residuals with the first output signal to generate a decoded signal, wherein the received encoded stream includes a group of transform coefficients scaled using a linear quantizer, and the decoding includes an application, i.e., performing a dequantization operation, wherein the dequantization operation is performed using a dead zone, wherein the size of the dead zone is obtained according to the step width used in the dequantization operation and quantization parameters received from an encoder. The combination may include a combination with an upsampled form of the first output signal.

[0083] The method according to any of the second and third aspects may further include any of the following implementation details. The method may include performing a dequantization operation based on time information associated with the encoded stream. The method may further include deriving one or more quantization parameters from the encoded stream based on time information associated with a dataset.

[0084] Typically, the method further includes determining a first quantization parameter for a first subset of data associated with the first time information.

[0085] The method may further include determining a second quantization parameter for a second subset of data associated with the second time information. The method may further include determining the time information associated with the dataset. The derived quantization parameter may further include determining from the encoded stream whether a default value should be used.

[0086] Typically, deriving quantization parameters further includes determining from the encoded stream whether the transmitted signal value should be used instead of the default value.

[0087] Dequantization can be performed based on one or more quantization parameters, wherein the one or more quantization parameters are generated based on a group of encoded data / transform coefficients or time information associated with the encoded stream.

[0088] Furthermore, the method may include decoding a data layer by combining one or more sets of decoded data with corresponding data contained in a time buffer, wherein the decoded data combined with the corresponding data in the time buffer may be dequantized with different quantization parameters relative to decoded data not combined with the corresponding data in the time buffer. This can be understood as the decoded data combined with the corresponding data in the time buffer being dequantized with quantization parameters different from those used for decoded data not combined with the corresponding data in the time buffer.

[0089] In some embodiments, a decoder receives a single quantization parameter for the entire data layer, and the decoder generates different quantization parameters, at least in part, based on said single quantization parameter, for datasets that will be combined with corresponding data in the time buffer and datasets that will not be combined with corresponding data in the time buffer. The formula used by the decoder to derive the quantization parameters for datasets that will not be combined with corresponding data in the time buffer may be at least in part based on default parameters, wherein the default parameters are overridden by configuration parameters if specified in the encoded stream.

[0090] According to a fourth aspect, an encoder for encoding input video is provided, the encoder being configured to perform the method according to a first aspect.

[0091] According to a fifth aspect, a decoder is provided for decoding an encoded stream into a reconstructed output video, the decoder being configured to perform the method according to any one of the second and third aspects.

[0092] According to a sixth aspect, a non-transitory computer-readable storage medium is provided that stores instructions, which, when executed by a processor, cause the processor to perform the method according to any one of the first to third aspects.

[0093] More generally, a method is provided for encoding an input video into multiple encoded streams, wherein the encoded streams can be combined to reconstruct an input signal. The method includes: receiving the input video; downsampling the input video to create a downsampled video; instructing the encoding of the downsampled video using a base encoder to create a base encoded stream; instructing the decoding of the base encoded stream using a base decoder to generate a reconstructed video; comparing the reconstructed video with the downsampled video to create a first set of residuals; and encoding the first set of residuals to create a first-level encoded stream, comprising: applying a transform to the residual set to create a set of coefficients; applying a quantization operation to the set of coefficients to create a set of quantized coefficients; and applying an encoding operation to the quantized coefficients. This method can be performed using any of the implementation details described above. Attached Figure Description

[0094] Figure 1 A high-level diagram illustrating the coding process;

[0095] Figure 2 A high-level diagram illustrating the decoding process;

[0096] Figure 3 A high-level diagram illustrating the encoding process and specific encoding steps;

[0097] Figure 4 A high-level diagram illustrating the decoding process and specific decoding steps;

[0098] Figure 5 A high-level diagram illustrating the coding process;

[0099] Figure 6 A high-level diagram illustrating another decoding process;

[0100] Figure 7 A flowchart illustrating the concepts described in this article;

[0101] Figures 8A-8D Demonstrates how quantization can be performed during the coding process based on a specific instance;

[0102] Figure 9 This is an exemplary schema of a frame sequence in a video sequence based on another instance process;

[0103] Figure 10 Demonstrates an example encoder and decoder system;

[0104] Figure 11 This is a schematic diagram illustrating the encoding process based on the examples in this article;

[0105] Figure 12 This is a schematic diagram illustrating the decoding process based on the examples in this article;

[0106] Figure 13A and 13B Each is a schematic diagram illustrating the encoding process based on the examples in this article;

[0107] Figure 14A and 14B Each is a schematic diagram illustrating the decoding process based on the examples in this article;

[0108] Figure 15A and 15B This is a schematic diagram illustrating various features of the time prediction process based on the examples in this paper;

[0109] Figure 16 It is a schematic diagram of an encoder; and,

[0110] Figure 17 This is a schematic diagram of a decoder. Detailed Implementation

[0111] This invention relates to methods. More specifically, this invention relates to methods for encoding and decoding signals. Data processing may include, but is not limited to, acquiring, exporting, outputting, receiving, and reconstructing data.

[0112] The decoding technique discussed in this paper is a flexible, adaptable, efficient, and computationally inexpensive decoding format that combines a video decoding format, a base codec (e.g., AVC, HEVC, or any other current or future codec), and decoded data with enhancement layers encoded using a different technique than the base codec. The technique uses a downsampled source signal, which is encoded using the base codec to form a base stream. Enhancement streams are formed using, for example, a set of encoded residuals from the base stream, either by increasing resolution or by improving frame rate correction or enhancement. Multiple enhancement layers can exist in the hierarchical structure. In a particular arrangement, the base stream can be decoded by a hardware decoder, while the enhancement stream can be adapted for processing using a software implementation.

[0113] Any optimizations used in the new decoding technique, tailored to the specific requirements or constraints of the enhancement stream and with low complexity, are crucial. Such requirements or constraints include: the potential reduction in computational power resulting from the need for software decoding of the enhancement stream; the need for the combination of the decoded set of residuals with the decoded frames; the possible structure of the residual data, i.e., a relatively high proportion of zero values ​​and a large range of highly variable data values; subtle differences in the input quantization blocks of the coefficients; and the structure of the enhancement stream for a set of discrete residual frames separated into various components. It should be noted that the constraints placed on the enhancement stream imply simple and fast entropy decoding operations as the basis so that the enhancement stream can effectively correct or enhance individual frames of the underlying decoded video. It should also be noted that in some scenarios, the underlying stream is decoded substantially simultaneously before combination, which puts pressure on resources.

[0114] In one scenario, the method described herein can be applied to so-called data planes that reflect different color components of a video signal. For example, the method described herein can be applied to different planes that reflect YUV or RGB data for different color channels. Different color channels can be processed in parallel. Therefore, a reference to the residual set as described herein can include multiple residual sets, where each color component has a different residual set forming part of a combined enhancement stream. Components of each first-order stream can be compared in any logical order; for example, each plane at the same level can be grouped together and sent together, or residual sets from different levels within each plane can be sent together.

[0115] This current document preferably meets the requirements of the following ISO / IEC documents: "Call for Proposals for Low Complexity Video Coding Enhancements" ISO / IEC JTC1 / SC29 / WG11 N17944, Macau, China, October 2018, and "Requirements for Complexity Video Coding Enhancements" ISO / IEC JTC1 / SC29 / WG11 N18098, Macau, China, October 2018 (which are incorporated herein by reference).

[0116] The general structure of the proposed coding scheme, to which the currently described technology can be applied, uses a downsampled source signal encoded by a base codec, adds first-level correction data to the decoded output of the base codec to generate a corrected picture, and then adds another level of enhancement data to the upsampled form of the corrected picture. Thus, the stream is considered as a base stream and an enhancement stream. This structure creates multiple degrees of freedom, allowing for great flexibility and adaptability to many situations, making the decoding format suitable for many use cases, including over-the-top (OTT) transmission, live streaming, live ultra-high-definition (UHD) broadcasting, etc. Although the decoded output of the base codec is not intended for viewing, it is a fully decoded video at a lower resolution, thus making the output compatible with existing decoders and usable as a lower-resolution output where appropriate. In some cases, the base codec can be used to create a base stream. The base codec may include a standalone codec controlled in a modular or "black box" manner. The methods described herein can be implemented using computer program code, which is executed by a processor and makes function calls on a hardware and / or software-implemented codec.

[0117] Generally, as used herein, the term "residual" refers to the difference between the values ​​of a reference array or reference frame and the actual array or frame of data. The array can be a one-dimensional or two-dimensional array representing a decoding unit. For example, a decoding unit can be a 2×2 or 4×4 set of residual values ​​corresponding to a region of similar size to an input video frame. It should be noted that this generalized example is unknown to the nature of the encoding operation performed and the input signal. References to "residual data" as used herein refer to data derived from the residual set, such as the residual set itself or the output of a set of data processing operations performed on the residual set. Throughout this specification, generally, a residual set contains multiple residuals or residual elements, each corresponding to a signal element, i.e., an element of the signal or original data. The signal can be an image or video. In these instances, the residual set corresponds to an image or frame of video, where each residual is associated with a pixel of the signal, which is a signal element. The examples disclosed herein describe how these residuals can be modified (i.e., processed) to affect the encoding pipeline or the final decoded image while reducing the overall data size. Residuals or sets can be processed on a per-residual-feature (or residual) basis, or on a group basis, such as per-patch or per-decoding-unit basis, where a patch or decoding unit is a neighboring subset of the residual set. In one case, a patch may comprise a group of smaller decoding units. It should be noted that processing can be performed on each video frame or on only a specified number of frames in the sequence.

[0118] Generally, each enhancement stream or two enhancement streams can be encapsulated into one or more enhancement bitstreams using a set of Network Abstraction Units (NALUs). NALUs are intended to encapsulate enhancement bitstreams so that enhancements are applied to the correct underlying reconstructed frames. A NALU may, for example, contain a reference index to the NALU containing the underlying decoder reconstructed frame bitstream to which the enhancements must be applied. In this way, enhancements can be synchronized to the underlying stream, and frames from each bitstream are combined to produce the decoded output video (i.e., the residual of each frame from the enhancement layer combined with frames from the underlying decoded stream). A group of images can represent multiple NALUs.

[0119] Returning to the initial process described above, where the base stream is provided along with enhancements at two levels (or sub-levels) within the enhancement stream, an instance of the generalized coding process is depicted in... Figure 1In the block diagram, the input full-resolution video 100 is processed to generate various coded streams 101, 102, and 103. A first coded stream (coded base stream) is generated by feeding a downsampled version of the input video to a base codec (e.g., AVC, HEVC, or any other codec). The coded base stream may be referred to as the base layer or base level. A second coded stream (coded level 1 stream) is generated by processing the residual obtained by the difference between the downsampled version of the reconstructed base codec video and the input video. A third coded stream (coded level 2 stream) is generated by processing the residual obtained by the difference between the upsampled version of the reconstructed base decoded video and the input video. In some cases, Figure 1 The components can provide a general low-complexity encoder. In some cases, enhanced streams can be generated through an encoding process that forms part of the low-complexity encoder, and the low-complexity encoder can be configured to control independent base encoders and decoders (e.g., encapsulated as a base codec). In other cases, the base encoder and decoder can be supplied as part of the low-complexity encoder. In one case, Figure 1 A low-complexity encoder can be viewed as a form of envelope for a base codec, where the functionality of the base codec is hidden from the entity implementing the low-complexity encoder.

[0120] The downsampling operation illustrated by downsampling component 105 can be applied to input video to produce downsampled video to be encoded by the base encoder 113 of the base codec. Downsampling can be performed in both the vertical and horizontal directions, or alternatively only in the horizontal direction. The base encoder 113 and the base decoder 114 can be implemented by the base codec (e.g., as different functions of a common codec). The base codec and / or one or more of the base encoder 113 and base decoder 114 may include appropriately configured electronic circuitry (e.g., hardware encoder / decoder) and / or computer program code executed by a processor.

[0121] Each enhanced stream coding process may not necessarily include an upsampling step. For example, in Figure 1 In this context, the first enhancement flow is conceptually a correction flow, while the second enhancement flow is upsampled to provide enhancement levels.

[0122] For more details, see the process of generating the enhanced stream. To generate the encoded Level 1 stream, the encoded base stream is decoded by the base decoder 114 (i.e., a decoding operation is applied to the encoded base stream to generate the decoded base stream). Decoding can be performed by the decoding function or mode of the base codec. Then, at the Level 1 comparator 110, the difference between the decoded base stream and the undersampled input video is created (i.e., a subtraction operation is applied to the undersampled input video and the decoded base stream to generate a first set of residuals). The output of comparator 110 may be referred to as the first set of residuals, such as a surface or frame of residual data, where a residual value is determined for each pixel at the resolution of the outputs of the base encoder 113, the base decoder 114, and the undersampled block 105.

[0123] The difference is then encoded by the first encoder 115 (i.e., the level 1 encoder) to generate an encoded level 1 stream 102 (i.e., the encoding operation is applied to the first residual set to generate the first enhanced stream).

[0124] As described above, the enhancement stream may include a first enhancement level 102 and a second enhancement level 103. The first enhancement level 102 may be considered as a corrected stream, for example, a stream that provides a correction level to the underlying coded / decoded video signal at a resolution lower than that of the input video 100. The second enhancement level 103 may be considered as another enhancement level that converts the corrected stream into the original input video 100, for example, by applying an enhancement level or correction to the signal reconstructed from the corrected stream.

[0125] exist Figure 1 In this example, a second enhancement layer 103 is created by encoding another set of residuals. This other set of residuals is generated by a layer 2 comparator 119. The layer 2 comparator 119 determines the difference between the upsampled form of the decoded layer 1 stream, such as the output of the upsampling component 117, and the input video 100. The input to the upsampling component 117 is generated by applying a first decoder (i.e., a layer 1 decoder) to the output of the first encoder 115. This generates a decoded set of layer 1 residuals. These residuals are then combined with the output of the base decoder 114 at the summing component 120. This effectively applies the layer 1 residuals to the output of the base decoder 114. This allows losses during layer 1 encoding and decoding to be corrected by layer 2 residuals. The output of the summing component 120 can be considered as an analog signal representing the output of the encoded base stream 101 and the encoded layer 1 stream 102 at the decoder after applying layer 1 processing.

[0126] As mentioned, the upsampled stream is compared with the input video, which creates another set of residuals (i.e., the difference operation is applied to the recreated upsampled stream to generate another set of residuals). The other set of residuals is then encoded by the second encoder 121 (i.e., the level 2 encoder) into an encoded level 2 enhanced stream (i.e., the encoding operation is then applied to the other set of residuals to generate another encoded enhanced stream).

[0127] Therefore, as Figure 1 As shown and described above, the output of the encoding process is a base stream 101 and one or more enhancement streams 102, 103, which preferably include a first enhancement level and another enhancement level. The three streams 101, 102, and 103 can be combined, with or without additional information such as control headers, to generate a combined stream representing the video coding framework structure of the input video 100. It should be noted that... Figure 1 The components shown operate on blocks or decoding units of data, such as 2×2 or 4×4 portions of a frame at a specific resolution level. These components operate without any inter-block dependencies, thus allowing them to be applied in parallel to multiple blocks or decoding units within a frame. This differs from contrasting video coding schemes, where dependencies (e.g., spatial or temporal) exist between blocks. These dependencies limit the level of parallelism and require significantly higher complexity.

[0128] exist Figure 2 The block diagram depicts the corresponding generalized decoding process. It can be said that... Figure 2 The display corresponds to Figure 1 The low-complexity decoder is a low-complexity encoder. The low-complexity decoder receives three streams 101, 102, and 103 generated by the low-complexity encoder, along with a header 204 containing additional decoding information. The encoded base stream 101 is decoded by a base decoder 210 corresponding to the base codec used in the low-complexity encoder. The encoded level 1 stream 102 is received by a first decoder 211 (i.e., a level 1 decoder), which decodes streams generated by the low-complexity encoder. Figure 1 The first residual set encoded by the first encoder 115 is decoded. At the first summing component 212, the output of the base decoder 210 is combined with the decoded residual obtained from the first decoder 211. The combined video, which may be referred to as the layer 1 reconstructed video signal, is upsampled by the upsampling component 213. The encoded layer 2 stream 103 is received by the second decoder 214 (i.e., the layer 2 decoder). The second decoder 214 decodes the data as shown by the first encoder 115. Figure 1 The second encoder 121 decodes the second residual set encoded by the second encoder 121. Although header 204 is in Figure 2The video is shown as being used by the second decoder 214, but it can also be used by the first decoder 211 and the base decoder 210. The output of the second decoder 214 is a second set of decoded residuals. These can be at a higher resolution than the first set of residuals and the input to the upsampling component 213. At the second summing component 215, the second set of residuals from the second decoder 214 is combined with the output of the upsampling component 213 (i.e., the upsampled reconstructed level 1 signal) to reconstruct the decoded video 250.

[0129] According to a low-complexity encoder, Figure 2 The low-complexity decoder can operate in parallel on different blocks or decoding units of a given frame of the video signal. Furthermore, decoding performed by two or more of the base decoder 210, the first decoder 211, and the second decoder 214 can be executed in parallel. This is possible because there is no inter-block dependency.

[0130] During decoding, the decoder can parse header 204 (which may contain global configuration information, image or frame configuration information, and data block configuration information) and configure the low-complexity decoder based on those headers. To recreate the input video, the low-complexity decoder can decode each of the base stream, the first enhancement stream, and another or a second enhancement stream. Frames of the streams can be synchronized and then combined to derive decoded video 250. Depending on the configuration of the low-complexity encoder and decoder, decoded video 250 can be a lossy or lossless reconstruction of the original input video 100. In many cases, decoded video 250 can be a lossy reconstruction of the original input video 100, wherein the loss has a reduced or minimal impact on the perception of decoded video 250.

[0131] exist Figure 1 and 2 In each of these operations, the Level 2 and Level 1 encoding operations may include transform, quantization, and entropy encoding steps (e.g., in the order described). Similarly, at the decoding stage, the residual may be passed through an entropy decoder, a dequantizer, and an inverse transform module (e.g., in the order described). Any suitable encoding and corresponding decoding operations can be used. However, preferably, the Level 2 and Level 1 encoding steps may be performed in software (e.g., by one or more central or graphics processing units in the encoding device).

[0132] The transforms described herein can use directional decomposition transforms, such as Hadamard-based transforms. Both can include small kernels or matrices applied to the flattened decoding units of the residuals (i.e., 2×2 or 4×4 residual blocks). Further details regarding the transforms can be found, for example, in patent applications PCT / EP2013 / 059847 or PCT / GB2017 / 052632, which are incorporated herein by reference. The encoder can select between different transforms to be used, such as between kernel sizes to be applied.

[0133] The transformation can transform the residual information onto four surfaces. For example, the transformation can produce the following components: average, vertical, horizontal, and diagonal. As mentioned earlier in this disclosure, these components output by the transformation can be used as coefficients to be quantized according to the described method in these embodiments.

[0134] In summary, the methods and apparatus described in this paper are based on a general approach built upon existing encoding and / or decoding algorithms (e.g., MPEG standards such as AVC / H.264, HEVC / H.265, etc.; and non-standard algorithms such as VP9, ​​AV1, etc.) as a baseline for corresponding enhancement layers used for different encoding and / or decoding methods. The concept behind the general approach is to encode / decode video frames hierarchically, in contrast to the block-based approach used in MPEG family algorithms. Hierarchical frame encoding involves generating residuals for the entire frame, followed by generating residuals for extracted frames, and so on.

[0135] The video compression residual data for a full-size video frame can be referred to as LoQ-2 (e.g., 1920×1080 for HD video frames, or even higher for UHD frames), while the video compression residual data for a decimated frame can be referred to as LoQ-x, where x represents the number of decimations corresponding to the hierarchical decimation. Figure 1 and 2 In the described instance, the variable x may have values ​​1 and 2 representing the first and second enhancement streams. Therefore, there exist two hierarchical levels that will generate compressed residuals. Other naming schemes for the levels can also be applied without any functional changes (e.g., the level 1 and level 2 enhancement streams described herein can instead be referred to as level 1 and level 2 streams – indicating a countdown from the highest resolution).

[0136] Figure 3The block diagram depicts a more detailed encoding process. The encoding process is divided into two halves, as indicated by the dashed lines. Below the dashed lines is the base level of the encoder 300, which can be usefully implemented in hardware or software. Above the dashed lines is the enhancement level, which can be usefully implemented in software. The encoder 300 may include enhancement level-only processes, or a combination of base level and enhancement level processes, as needed. The encoder 300 can be usefully implemented in software, particularly at the enhancement level. This arrangement allows for upgrades, for example, using firmware (e.g., software) updates to provide the base level, where the firmware is configured to provide the enhancement level. In newer devices, both the base level and the enhancement level may be provided in hardware and / or a combination of hardware and software.

[0137] The encoder topology in a typical horizontal configuration is as follows. Encoder 300 includes input I for receiving input signal 30. Input signal 30 may include an input video signal, wherein the encoder is applied frame by frame. Input I is connected to downsampler 305D and processing block 300-2. Downsampler 305D may correspond to... Figure 1 The downsampling component 105, and the processing block 300-2 can correspond to Figure 1 The second encoder 121. The downsampler 305D outputs at the base level of the encoder 300 to the base codec 320. The base codec 320 can be implemented... Figure 1 The basic encoder 113 and basic decoder 114. The downsampler 305D also outputs to the processing block 300-1. The processing block 300-1 can correspond to Figure 1 The first encoder 115. Processing block 300-1 passes the output to upsampler 305U, which in turn outputs to processing block 300-2. Upsampler 305U can correspond to Figure 1 The upsampling component 117. Each of processing blocks 300-2 and 300-1 includes one or more of the following modules: transform block 310, quantization block 320, entropy encoding block 330, and residual processing block 350. Residual block 350 may occur before transform block 310 and / or control residual processing in processing block 300. The order of processing can be illustrated as shown in the figure.

[0138] Input signal 30, such as full (or highest) resolution video in this example, is processed by encoder 300 to generate various encoded streams. A basic encoded stream is generated by feeding a downsampled form of the input video 30 to a base codec 320 (e.g., AVC, HEVC, or any other codec) at the base level using downsampler 305D. The basic encoded stream may include the output of the base encoder of the base codec 320. A first encoded stream (encoded level 1 stream) is created by reconstructing the encoded basic stream to create a basic reconstruction, and then taking the difference between the basic reconstruction and the downsampled form of the input video 30. Reconstructing the encoded basic stream may include receiving the decoded basic stream from the base codec (i.e., the input to processing block 300-1 includes, for example, the decoded basic stream). Figure 1 The base decoded stream is shown. Next, the difference signal is processed at block 300-1 to create an encoded level 1 stream. Block 300-1 includes transform block 310-1, quantization block 320-1, and entropy coding block 330-1. A second encoded stream (encoded level 2 stream) is created by upsampling the corrected form of the base reconstruction using upsampler 305U and taking the difference between the corrected form of the base reconstruction and the input signal 30. This difference signal is then processed at block 300-2 to create an encoded level 2 stream. Block 300-2 includes transform block 310-2, quantization block 320-2, entropy coding block 330-2, and residual processing block 350-2. Following processing block 300-1, the blocks can be executed in the order shown in the figure (e.g., residual processing followed by transform followed by quantization followed by entropy coding).

[0139] Quantization schemes can be used to create quantities from residual signals, allowing specific variables to take only specific discrete values. In one case, quantization involves dividing by a predetermined step width. This can be applied at two levels (1 and 2). For example, quantization at block 320 could involve dividing the transformed residual value by the step width (e.g., where an integer quotient is used to generate the quantized value and the remainder is ignored). The step width can be predetermined, for example, chosen based on the desired quantization level. In one case, dividing by the step width can be converted to multiplying by the inverse step width, which can be implemented more efficiently in hardware. In this case, dequantization at, for example, block 320 could involve multiplying by the step width. Entropy coding as described herein can include run-length encoding (RLE), followed by processing the encoded output using a Huffman encoder. In some cases, when entropy coding is required, only one of these schemes can be used.

[0140] The encoded basic stream can be called the basic layer stream.

[0141] As previously described, the residual is calculated by comparing the original image signal with the reconstructed image signal. For example, in one case, the residual of the L-2 enhancement stream is determined by subtracting the upsampled output from the original image signal (e.g., the input video indicated in the figure). The upsampled input can be referred to as the reconstruction of the signal after analog decoding. In another case, the residual of the L-1 enhancement stream is determined by subtracting the image stream output by the base decoder from the downsampled original image signal (e.g., the downsampled output).

[0142] exist Figure 4 The block diagram depicts the corresponding Figure 3 The decoder 400 performs the decoding process of the encoder. The decoding process is divided into two halves, as shown by the dashed lines. Below the dashed lines is the base layer of the decoder 400, which can be usefully implemented in hardware. Above the dashed lines is the enhancement layer, which can be usefully implemented in software. The decoder 400 may include only the enhancement layer process, or a combination of the base layer process and the enhancement layer process, as needed. The decoder 400 can be usefully implemented in software, especially at the enhancement layer, and may suitably outperform conventional decoding techniques, especially conventional hardware techniques. Conventional techniques refer to earlier technologies that were previously developed and marketed, are inconvenient to replace and / or expensive to replace, and are still usable for the purpose of decoding signals. In other cases, the base layer may include any existing and / or future video coding tools or technologies.

[0143] The decoder topology at a typical level is as follows. Decoder 400 includes inputs (not shown) for receiving one or more input signals, including an encoded base stream, an encoded level 1 stream, and an encoded level 2 stream, along with an optional header containing additional decoding information. Decoder 400 includes a base decoder 420 at the base level and processing blocks 400-1 and 400-2 at the enhancement level. An upsampler 405U is also disposed between processing blocks 400-1 and 400-2 to provide processing block 400-2 with an upsampled version of the signal output from processing block 400-1. The base decoder 420 may correspond to... Figure 2 The basic decoder 210, processing block 400-1 can correspond to Figure 2 The first decoder 211, processing block 400-2 can correspond to Figure 2 The second decoder 214, and the upsampler 405U can correspond to Figure 2 The upper sampler 213.

[0144] Decoder 400 receives the one or more input signals and directs three streams generated by encoder 300. The encoded base stream is directed to and decoded by base decoder 420, which corresponds to the base codec 420 used in encoder 300 and is used to reverse the encoding process at the base level. The encoded level 1 stream is processed by block 400-1 of decoder 400 to recreate the first set of residuals created by encoder 300. Block 400-1 corresponds to processing block 300-1 in encoder 300 and is used at the base level to reverse or substantially reverse the processing of block 300-1. The output of base decoder 420 is combined with the first set of residuals obtained from the encoded level 1 stream. The combined signal is sampled by upsampler 405U. The encoded level 2 stream is processed by block 400-2 to recreate additional residuals created by encoder 300. Block 400-2 corresponds to processing block 300-2 of encoder 300 and is used at the base level to reverse or substantially reverse the processing of block 300-2. The upsampled signal from upsampler 405U is combined with additional residuals obtained from the encoded layer 2 stream to create a layer 2 reconstruction of the input signal 30. The output of processing block 400-2 can be considered similar to Figure 2 250 decoded videos.

[0145] As described above, the enhancement stream may include two streams: a coding level 1 stream (first enhancement level) and a coding level 2 stream (second enhancement level). The coding level 1 stream provides a set of correction data, which can be combined with the decoded form of the base stream to generate a corrected image.

[0146] Figure 5 More detailed display Figure 1 The encoder 300. The encoded base stream is created directly by the base encoder 320E and can be quantized and entropy encoded as needed. In some cases, these subsequent processes can be performed as part of the encoding performed by the base encoder 320E. To generate the encoded level 1 stream, the encoded base stream is decoded at the encoder 300 (i.e., the decoding operation is applied to the encoded base stream at the base decoder block 320D). The base decoder block 320D is presented as part of the base level of the encoder 300 and is presented as separate from the corresponding base encoder block 320E. For example, the base decoder 320D can be a decoding component that supplements the encoding components in the form of the base encoder 320E with the base codec. In other instances, the base decoder block 320D can actually be part of an enhancement level, and more precisely, part of processing block 300-1.

[0147] Return to Figure 5The difference between the decoded base stream output from the base decoder block 320D and the downsampled input video is created (i.e., subtraction operation 310-S is applied to the downsampled input video and the decoded base stream to generate a first set of residuals). Here, the term residual is used in the same manner as known in this art; that is, a residual represents the error or difference between a reference signal or frame and a desired signal or frame. Here, the reference signal or frame is the decoded base stream, and the desired signal or frame is the downsampled input video. Therefore, the residuals used in the first enhancement layer can be considered as correction signals because they are able to 'correct' the decoded base stream in the future to a closer approximation of the downsampled input video used in the base coding operation. This is useful because it can correct for quirks or other characteristics of the base codec. These characteristics include, in particular, the motion compensation algorithm applied by the base codec, the quantization and entropy coding applied by the base codec, and the block adjustment applied by the base codec.

[0148] exist Figure 5 More detailed display Figure 3 The components of block 300-1. Specifically, the first set of residuals is transformed, quantized, and entropy-encoded to produce an encoded level 1 stream. Figure 5 In this process, transform operation 310-1 is applied to the first residual set; quantization operation 320-1 is applied to the transformed residual set to generate a quantized residual set; and entropy coding operation 330-1 is applied to the quantized residual set to generate a coded level 1 stream at the first enhancement level. However, it should be noted that in other instances, only quantization step 320-1 or only transform step 310-1 may be performed. Entropy coding may not be used, or it may optionally be used as a supplement to one or both of transform step 110-1 and quantization step 320-1. The entropy coding operation can be any suitable type of entropy coding, such as a Huffman coding operation or a run-length encoding (RLE) operation, or a combination of both Huffman coding and RLE operations.

[0149] For example, the choice of entropy decoding schemes, such as iso-entropy decoding schemes, can have a beneficial effect on decoding performance in combination with the described quantization. This can be understood from the tendency to apply a higher degree of quantization to residual data as described in this disclosure to produce a high proportion of zero values. Run-length encoding, as mentioned above, is particularly suitable for encoding data with this distribution, and thus these methods can synergistically improve the efficiency of the overall process. Similarly, for embodiments where quantization is applied with a larger step width and the distribution of the quantized data results in a relatively large number of integer values, the efficiency of the encoding process will generally benefit from the use of prefix / Huffman coding, which is particularly suitable for these distributions. This is especially true in the case where the higher integer values ​​have lower frequency residuals. These forms of distributions can be efficiently encoded using Huffman coding, which works by allocating fewer bits to high-frequency symbols. In this way, quantization and entropy coding operations are complementary.

[0150] As described above, the enhancement stream may include a coded level 1 stream (first enhancement level) and a coded level 2 stream (second enhancement level). The first enhancement level can be viewed as implementing the corrected video at the base level, that is, for example, correcting encoder and / or decoder artifacts. The second enhancement level can be viewed as another enhancement level that can be used to convert the corrected video into the original input video or an approximation thereof (e.g., adding detail or sharpness). For example, the second enhancement level may add fine details lost during undersampling and / or help correct errors introduced by one or more of the transform operation 310-1 and quantization operation 320-1.

[0151] See Figure 3 and Figure 5 In order to generate the encoded level 2 stream, another enhancement level information is created at block 300-2 by generating and encoding another set of residuals. The other set of residuals is the difference between the upsampled form of the corrected form of the decoded base stream (reference signal or frame) (via upsampler 305U) and the input signal 30 (desired signal or frame).

[0152] To achieve the reconstruction of the corrected form of the decoded base stream generated at decoder 400, at least some processing steps of block 300-1 are reversed to simulate the process of decoder 200, taking into account at least some losses and oddities in the transform and quantization processes. For this purpose, block 300-1 includes an inverse quantization block 320-1i and an inverse transform block 310-1i. The quantized first residual set is inverse quantized at inverse quantization block 320-1i in encoder 100 and inverse transformed at inverse transform block 310-1i to regenerate the decoder-side form of the first residual set.

[0153] This improved decoder-side format combination of the decoded base stream from decoder 320D and the first residual set (i.e., performing a summation operation 310-C on the decoder-side format of the decoded base stream and the first residual set). The summation operation 310-C generates a reconstruction, as is likely to be performed on the subsampled format of the input video generated at the decoder—i.e., the reconstructed base codec video). Figure 3 and Figure 5 As shown, the reconstructed base codec video is then upsampled using upsampler 305U.

[0154] The upsampled signal (i.e., the reference signal or frame) is then compared with the input signal 30 (i.e., the desired signal or frame) to create a second residual set (i.e., the difference operation 300-S is applied to the recreated stream after upsampling to generate another residual set). The second residual set is then processed at block 300-2 to become an encoded level 2 stream (i.e., the encoding operation is then applied to another or a second residual set to generate another or a second encoded enhanced stream).

[0155] Specifically, the second residual set is transformed (i.e., transformation operation 310-2 is performed on another residual set to generate another transformed residual set). The transformed residuals are then quantized and entropy encoded in the manner described above with respect to the first residual set (i.e., quantization operation 320-2 is applied to the transformed residual set to generate another quantized residual set; and entropy encoding operation 320-2 is applied to the other quantized residual set to generate an encoded level 2 stream containing another level of enhanced information). However, only quantization step 20-1, or only transformation and quantization steps, can be performed. Entropy encoding can optionally be used as a supplement. Preferably, the entropy encoding operation can be a Huffman coding operation or a run-length encoding (RLE) operation, or both.

[0156] Therefore, as Figure 3 and 5 As shown and described above, the output of the encoding process is the base stream at the base layer, and one or more enhancement streams at enhancement layers, preferably including a first enhancement layer and another enhancement layer. As discussed with reference to previous examples, Figure 5 The operations can be applied in parallel to the decoding units or blocks of the color components of a frame because there is no inter-block dependency. Encoding of each color component within the set of color components can also be performed in parallel (e.g., such that the copying is based on (number of frames) * (number of color components) * (number of decoding units per frame)). Figure 5 (The operation). It should also be noted that different color components may have different numbers of decoding units per frame, for example, the luminance (e.g., Y) component may be processed at a high resolution of the colorimetric (e.g., U or V) component set when the change in illuminance is greater than the change in color that can be detected by human vision.

[0157] At decoder 400, the encoded base stream and one or more enhancement streams are received. Figure 6 More detailed display Figure 4 The decoder.

[0158] At base decoder 420, the encoded base stream is decoded to produce a base reconstruction of the input signal 30 received at encoder 300. This base reconstruction can be used in practice to provide a visually reproducible representation of signal 30 at a lower quality level. However, the primary purpose of this base reconstruction is to provide a basis for a higher quality reproduction of the input signal 30. For this purpose, the decoded base stream is provided to processing block 400-1. Processing block 400-1 also receives the encoded level 1 stream and reverses any encoding, quantization, and transform applied by encoder 300. Block 400-1 includes an entropy decoding process 430-1, an inverse quantization process 420-1, and an inverse transform process 410-1. Optionally, only one or more of these steps may be performed depending on the operations performed at the corresponding block 300-1 at encoder. By performing these corresponding steps, the decoded level 1 stream, including a first set of residuals, becomes available at decoder 400. The first residual set is combined with the decoded base stream from the base decoder 420 (i.e., a summation operation 410-C is performed on the decoded base stream and the decoded first residual set to generate a reconstructed, subsampled form of the input video—i.e., the reconstructed base codec video). For example... Figure 4 and Figure 6 As shown, the reconstructed base codec video is then upsampled by the upsampler 405U.

[0159] Furthermore, and optionally in parallel, in Figure 2 At block 400-2, the encoded level 2 stream is processed to produce another set of decoded residuals. Similar to processing block 300-2, processing block 400-2 includes an entropy decoding process 430-2, an inverse quantization process 420-2, and an inverse transform process 410-2. Of course, these operations will correspond to the operations performed at block 300-2 in encoder 300, and one or more of these steps may be omitted as needed. Block 400-2 produces a decoded level 2 stream including another set of residuals, and these residuals are summed at operation 400-C with the output from upsampler 405U to create a level 2 reconstruction of the input signal 30. Level 2 reconstruction can be viewed as, for example... Figure 2 The decoded video output from the 250 level. In some instances, it is also possible to obtain and view the reconstructed video passed to the upsampler 405U - this will have a first enhancement level but may be at a lower resolution than the level 2 reconstruction.

[0160] Therefore, as shown and described above, the output of the decoding process is (optionally) a base reconstruction, as well as a reconstruction of the original signal at a higher level. This example is particularly well-suited for creating encoded and decoded video at different frame resolutions. For instance, the input signal 30 could be an HD video signal comprising frames at a resolution of 1920×1080. In some cases, both the base reconstruction and the Layer 2 reconstruction can be used by the display device. For example, in the case of network traffic, the Layer 2 stream may be interrupted more often than the Layer 1 stream and the base stream (because it may contain up to 4× data, where undersampling reduces the dimension by 2 in each direction). In this case, when traffic occurs, the display device can resume displaying the base reconstruction, while the Layer 2 stream is interrupted (e.g., when the Layer 2 reconstruction is unavailable), and then resume displaying the Layer 2 reconstruction when network conditions improve. A similar approach can be applied when the decoding device is under resource constraints; for example, a set-top box performing a system update may have an operating base decoder 220 to output the base reconstruction, but may not have the processing capacity to compute the Layer 2 reconstruction.

[0161] The encoding arrangement also allows the video distributor to distribute video to a set of heterogeneous devices; those devices with only the basic decoder 220 examine the basic reconstruction, while those with enhancement layers examine the higher-quality layer 2 reconstruction. In the comparative case, two complete video streams at separate resolutions are needed to serve the two sets of devices. Because the layer 2 and layer 1 enhancement streams encode residual data, the layer 2 and layer 1 enhancement streams can be encoded more efficiently, for example, the distribution of residual data is typically mostly around 0 quality (i.e., no difference) and usually takes a small range of values ​​around 0. This is especially true after quantization. In contrast, the complete video streams at different resolutions will have different distributions of non-zero mean or median, which require higher bit rates to transmit to the decoder.

[0162] In some instances, residuals can be viewed as errors or differences at a specific quality level or resolution. In the described instance, there are two quality levels or resolutions and therefore two sets of residuals (L-1 and L-2). Each set of residuals described herein models a different form of error or difference. For example, L-1 residuals typically correct for characteristics of the base encoder, such as artifacts introduced by the base encoder as part of the encoding process. In contrast, L-2 residuals typically correct for the combined effects introduced by changes in quality levels and the differences introduced by L-1 correction (e.g., artifacts generated by the L-1 encoding pipeline at a wider spatial scale, such as a region of 4 or 16 pixels). This means that the following is not self-evident: an operation performed on one set of residuals will necessarily provide the same effect to the other set of residuals; for example, each set of residuals may have different statistical patterns and correlation sets.

[0163] In the example described in this paper, the residuals are encoded by an encoding pipeline. This may include transform, quantization, and entropy encoding operations. It may also include residual grading, weighting, and filtering. These pipelines are shown in... Figure 1 And in 3A and 3B. The residual is then transmitted to the decoder, for example as L-1 and L-2 enhancement streams, which can be combined with the base stream as a hybrid stream (or transmitted separately). In one case, a bit rate is set for the hybrid data stream including the base stream and the two enhancement streams, and then different adaptive bit rates are applied to individual streams based on the data being processed to meet the set bit rate (e.g., high-quality video perceived with low artifact levels can be constructed by adaptively assigning bit rates to different individual streams (even at the frame-by-frame level) so that constrained data can be used by the individual stream that is most perceptibly influential, which can change as the image data changes).

[0164] The residual set described in this paper can be considered sparse data, for example, in many cases there is no difference for a given pixel or region, and the resulting residual value is zero. When looking at the distribution of the residuals, many probability masses are assigned to small residual values ​​located close to zero, such as for some video values ​​of -2, -1, 0, 1, 2, etc., which occur most frequently. In some cases, the distribution of residual values ​​is symmetrical or approximately symmetrical about 0. In some test video cases, the distribution of residual values ​​is found to have a shape similar to a logarithmic or exponential distribution about 0 (e.g., symmetrical or approximately symmetrical). The exact distribution of the residual values ​​can depend on the content of the input video stream.

[0165] The residual can be viewed itself as a two-dimensional image, such as a difference image of the difference. In this way, the sparsity of the data can be seen to involve features visible in the residual image, such as “points,” small “lines,” “edges,” and “corners.” These features have been found to be generally not perfectly correlated (e.g., spatially and / or temporally). These features have properties that differ from those of the image data from which they originate (e.g., the pixel characteristics of the original video signal).

[0166] Because the characteristics of residuals differ from those of the image data from which they originate, it is generally impossible to apply standard coding methods, such as those found in the traditional Moving Picture Experts Group (MPEG) coding and decoding standards. For example, many contrast schemes use large transforms (e.g., transforms of large pixel regions in a normal video frame). Due to the characteristics of residuals, such as those described above, using these large transforms for residual images would be extremely inefficient. For example, encoding small points in a residual image using large blocks of regions designed for normal images would be very difficult.

[0167] Some of the examples described in this paper address these issues by alternatively using smaller and simpler transform kernels (e.g., 2×2 or 4×4 kernels – directed decomposition and directed decomposition squared, as presented in this paper). The transforms described in this paper can be applied using Hadamard matrices (e.g., 4×4 matrices for flattening 2×2 decoded blocks, or 16×16 matrices for flattening 4×4 decoded blocks). This shifts in a different direction from the contrasting video coding methods. Applying these new methods to residual blocks yields compression efficiencies. For example, some transforms generate uncorrelated coefficients (e.g., in space) that can be efficiently compressed. While correlations between coefficients can be utilized, for example, for lines in the residual image, these correlations can lead to coding complexity, making implementation difficult on conventional and low-resource devices, and these correlations often generate other complex artifacts that require correction. Preprocessing residuals by setting certain residual values ​​to 0 (i.e., not forwarding these residual values ​​for processing) provides a controllable and flexible way to manage bit rates and stream bandwidth, as well as resource usage.

[0168] For the sake of completeness, Figure 7 The broad principles of the concepts described in this article are illustrated in the form of flowcharts. Method 1000 includes: receiving an input video (step 1001); downsampling the input video to create a downsampled video (step 1002); instructing the encoding of the downsampled video using a base encoder to create a base encoded stream (step 1003); instructing the decoding of the base encoded stream using a base decoder to generate a reconstructed video (step 1004); comparing the reconstructed video with the downsampled video to create a first residual set (step 1005); and encoding the first residual set to create a first-level encoded stream, including: applying a transform to the first residual set to create a first set of coefficients (step 1006); applying a quantization operation to the first set of coefficients to create a first set of quantized coefficients (step 1007); and applying an encoding operation to the first set of quantized coefficients (step 1008), wherein applying the quantization operation includes: adapting the quantization based on the first set of coefficients to be quantized, including changing the step width for different coefficients in the first set of coefficients, wherein a first set of parameters derived from the adaptation is transmitted to the decoder to achieve dequantization of the first set of quantized coefficients.

[0169] Figure 8A Provides examples of how to perform quantization of residuals and / or coefficients (transformed residuals) based on block groups with defined step widths. Figure 8A The x-axis represents the residual or coefficient value. In this example, the number of block groups is limited by a step width of 5. The step width can be understood as the quantization step size, as shown in the diagram. The size of the step width can be selected, for example, based on parameter values. In some cases, the size of the step width can be set dynamically, for example, based on the rate control example described above. Figure 8AIn this example, the step width generates blocks corresponding to residual values ​​in the ranges of 0-4, 5-9, 10-14, and 15-19 (i.e., 0 to 4 includes both 0 and 4). The block width can be configured to include or exclude endpoints as needed. In this example, quantization is performed by replacing all values ​​falling within the block with integer values ​​(e.g., residual values ​​between 0 and 4 (including endpoints) have a quantization value of 1). Figure 8A In quantization, the following operations can be performed: divide by the step size (e.g., 5), take the base of the result (i.e., for positive values, the nearest integer less than a certain decimal), and then add one (e.g., 3 / 5 = 0.6, floor(0.6) = 0, 0 + 1 = 1; or 16 / 5 = 3.2, floor(3.2) = 3, 3 + 1 = 4). Negative values ​​can be handled in a similar way, for example, by applying an absolute value and then converting it to a negative value after calculation (e.g., abs(-9) = 9, 9 / 5 = 1.8, floor(1.8) = 1, 1 + 1 = 2, 2 * -1 = -2). Figure 8A This example demonstrates a linear quantization scenario where all block groups share a common step width. It should be noted that various different implementations based on this approach can be developed; for example, the first block group may have a quantization value of 0 instead of 1, or may include values ​​from 1 to 5 (including the endpoints). Figure 8A This is merely an illustration of quantization based on a block group with a given step width.

[0170] Figure 8B This demonstrates how the so-called "dead zone" (DZ) can be implemented. This can be understood as the area near the zero output value of the quantizer; that is, a band containing zero signals and whose size can be the same as or different from the step width. Therefore, for this band of input close to zero, the signal can be effectively attenuated, so that low-level signals, which typically correspond to noise in visual data, are not unnecessarily allocated to the data. Figure 8B In this context, residuals or coefficients with values ​​within a predefined range are set to 0. Figure 8B In this context, the predefined range is the range around the value 0. Figure 8B In this configuration, values ​​less than 6 and greater than -6 are set to 0. The dead zone can be set to a fixed range (e.g., -6 to 6) or based on the step size. In one case, the dead zone can be set to multiple predefined step sizes, such as a linear function of the step size values. Figure 8B In this example, the dead zone is set to 2.4 * step width. Therefore, with a step width of 5, the dead zone extends from -6 to +6. In other cases, the dead zone can be set as a non-linear function of the step width value.

[0171] In one case, the dead time is set based on a dynamic step width, which can be adaptive, for example. In this case, the dead time can change with the step width. For example, if the step width is updated to 3 instead of 5, the dead time of 2.4 * step width can change from the range -6 to +6 to the range -3.6 to 3.6; or if the step width is updated to 10, the dead time can change to extend from -12 to 12. In one case, the step width multiplier can be between 2 and 4. In another case, the multiplier can also be adaptive, for example based on operating conditions such as the available bit rate.

[0172] Having a dead time helps reduce the amount of data to be transmitted over the network, for example, by helping to reduce the bit rate. When using a dead time, residual or coefficient values ​​that fall within the dead time are effectively ignored. This method also helps remove low-level residual noise. Having an adaptive rather than constant dead time means that smaller residual or coefficient values ​​are not over-filtered when the step size decreases (e.g., if more bandwidth is available), and the bit rate decreases appropriately if the step size increases. The dead time only needs to be specified at the encoder; the decoder simply receives a quantized value of 0 for any residual or coefficient that falls within the dead time.

[0173] Figure 8C This demonstrates how a method called block group folding can be applied. Figure 8C In some instances, block group folding is used in conjunction with dead zones, but in others, block group folding can be used without dead zones and / or in conjunction with other quantization methods. Figure 8C In this context, block folding is used to place all residual or coefficient values ​​residing above a selected quantization block into the selected block. For example, this can be viewed as a form of clipping, as mentioned above.

[0174] Figure 8C In the next step, a step width of 5 is applied again. A dead zone with a range of 2.4 * step width is also applied, such that values ​​between -6 and 6 are set to 0. This can also be seen as following into the larger first quantization block group (with a value of 0). Then, two quantization block groups with a width of 5 are defined for positive and negative values. For example, the block group with quantization value 1 is defined between 6 and 11 (e.g., with a step width of 5), and the block group with quantization value 2 is defined between 11 and 16. In this example, to perform block group folding, all residuals or coefficients with values ​​that would normally fall above the second block group (e.g., with a value greater than 16) are "folded" into the second block group, for example, limited to have a quantization value of 2. This can be done by setting all values ​​greater than a threshold to the maximum block group value (e.g., 2). A similar process occurs for negative values. Figure 8C The large arrow indicates this.

[0175] Block group folding can be an optional processing option at the encoder. It does not need to be specified during dequantization at the decoder (e.g., a value of 2 that has been "folded" or "limited" is simply dequantized as if it were in a second block group). Block group folding can be performed to reduce the number of bits sent to the decoder via the network. Block group folding can be configured based on network conditions and / or underlying streaming processing to reduce the bit rate.

[0176] Figure 8D This demonstrates how quantization offsets can be used in specific situations. Quantization offsets can be used to shift the position of quantized block groups. Figure 8D The diagram shows a line indicating the possible real-world counts of the residual or coefficient values ​​along the x-axis. In this example, many values ​​are close to zero, with the count of higher values ​​decreasing as we move away from 0. If the counts are normalized, the line can also indicate the probability distribution of the residual or coefficient values.

[0177] Figure 8D The left-hand side bars and the dashed lines on the right-hand side illustrate the histogram modeling the quantization. For clarity, the counts of the first to third blocks after the dead zone are shown (for both positive and negative values, the latter are striped to represent bars). For example, the bars show the counts of quantized values ​​1, 2, 3 and -1, -2, -3. Due to quantization, the distribution modeled by the histogram differs from the actual distribution shown by the lines. For example, the error 'e' is shown, illustrating the degree to which the bars differ from the lines.

[0178] To modify the nature of the error e, a quantization offset qO can be applied. The quantization offset can be understood as a parameter whose value determines whether and to what extent the quantization interval or block group will be shifted from a predetermined or default position or set of values. For positive values, a positive quantization offset is used to shift each block group to the right, and a negative quantization offset is used to shift each block group to the left. The quantization offset can be applied in combination with a dead zone. In one case, the dead zone can be applied based on a first set of thresholds, for example, all values ​​less than (n * step width) / 2 and greater than (n * step width * -1) / 2 are set to 0.

[0179] In some instances, the quantization offset can be signaled to the decoder for use during dequantization.

[0180] In one case, at the encoder, the quantization offset can be subtracted from the residual or coefficient value before step-width-based quantization. Therefore, in the decoder, the transmitted offset can be added to the received quantized value for step-width-based dequantization. In some cases, the offset can be adjusted based on the sign of the residual or coefficient to allow symmetric operation about 0 values. In one case, the use of the offset can be disabled by setting the quantization or dequantization offset value to 0. In one case, the applied quantization offset can be adjusted based on a defined dead-time width. In one case, the dead-time width can be calculated at the decoder, for example, based on the step width and quantization parameters received from the encoder.

[0181] In one scenario, the step width used for quantization can vary for different coefficients within a 2×2 or 4×4 coefficient block. For example, a smaller step width can be assigned to coefficients that are experimentally determined to have a greater impact on the perception of the decoded signal. For instance, in a 2×2 or 4×4 directional decomposition (DD-squared or "DDS") as described above, smaller step widths can be assigned to the A, H, V, and D coefficients, with larger step widths assigned to later coefficients. In this case, the base_stepwidth parameter can be used to define the default step width, and a modifier can then be applied to this parameter to calculate modified_stepwidth for use in quantization (and dequantization), for example, modified_stepwidth = base_stepwidth * modifier (where the "modifier" can be set based on specific coefficients within a block or cell and can be derived from signaling such as the variable "qm" described below).

[0182] In some cases, the modifier may additionally or alternatively depend on the enhancement level. For example, for a level 1 enhancement stream, the step size can be smaller because it can affect multiple reconstructed pixels at a higher quality level.

[0183] In some cases, modifiers can be defined based on both the coefficients within a block and the enhancement level. In one case, the quantization matrix can be defined with a set of modifiers for different coefficients and different enhancement levels. This quantization matrix can be preset (e.g., at the encoder and / or decoder), signaled between the encoder and decoder, and / or dynamically constructed at the encoder and / or decoder. For example, in the latter case, the quantization matrix can be constructed at the encoder and / or decoder based on other stored and / or signaled parameters, such as parameters received via a configuration interface.

[0184] In one scenario, different quantization modes can be defined, or different schemes of quantization matrices to be applied to a given set of coefficients can be defined. In one mode, a common quantization matrix can be used for two enhancement levels; in another mode, separate matrices can be used for different levels; and in yet another mode, a quantization matrix can be used for only one enhancement level, such as only for level 2. The quantization matrix can be indexed by the position of the coefficients within the block (e.g., 0 or 1 along the x-direction and 0 or 1 along the y-direction for a 2×2 block, or 0 to 3 for a 4×4 block).

[0185] In one case, the set of values ​​can define the underlying quantization matrix. This underlying quantization matrix can be modified by a scaling factor, which is a function of the stride width of one or more enhancement layers. In another case, the scaling factor can be a clamping function of the stride width variable. At the decoder, the stride width variable can be received from the encoder for one or more of the layer 2 and layer 1 streams.

[0186] With the aid of other examples of processes involving the principles described above, advantageous patterns for configuring and adapting quantization to further improve encoding and decoding procedures can be understood by further examining the following text. Data block units can be applied to enhance payload semantics, involving several parameters that can be used to signal and configure the properties of the quantization and dequantization steps.

[0187] In one instance, the parameter dequant_offset_signalled Specifies whether the offset parameter value will be applied during signal dequantization. This allows you to determine whether an offset is sent during signal transmission. In this example, if an offset is sent, it is used. If it is not sent, the default offset can be used, or no offset may be used.

[0188] In an example of a method for encoding an input video into multiple encoded streams, wherein the encoded streams can be combined to reconstruct the input video, the method may involve receiving the input video and downsampling the input video to create an downsampled video.

[0189] The method typically further includes instructing the encoding of the downsampled video using a base encoder to create a base encoded stream; instructing the decoding of the base encoded stream using a base decoder to generate a reconstructed video; comparing the reconstructed video with the downsampled video to create a first set of residuals; and encoding the first set of residuals to create a first-level encoded stream. This preferably includes: applying a transform to the residual set to create a set of coefficients; applying a quantization operation to the set of coefficients to create a set of quantized coefficients; and applying an encoding operation to the quantized coefficients, wherein the quantization operation is performed using a quantization matrix derived from obtained values ​​of quantization matrix mode parameters.

[0190] As previously described in this disclosure, the quantization matrix mode parameter can be advantageously used to specify the quantization matrix to be used in the encoding process. In some instances, when the quantization matrix mode parameter value is equal to a predetermined value, such as when it is equal to zero, the method may involve using a default quantization matrix for each of two quality levels. These levels are typically or generally correspond to level 1 and level 2 enhancement streams. When the quantization matrix mode parameter value is equal to 1, a first quantization matrix can be used for each of the two quality levels, and the first quantization matrix can be signaled, for example, from the encoder to the decoder or to a device to which the encoded stream will be transmitted. When the quantization matrix mode parameter value is equal to 2, a second quantization matrix can be used for quality level 2, and the second quantization matrix can be signaled. In this case, a quantization matrix is ​​not used for quality level 1, or a default value can be used for this level. When the quantization matrix mode parameter value is equal to 3, a third quantization matrix is ​​preferably used for quality level 1 or the first level encoded stream, and the third quantization matrix is ​​signaled. In this case, a quantization matrix is ​​not used for quality level 2, or a default value can be used for this level. When the quantization matrix mode parameter value is equal to 4, a fourth quantization matrix can be used for the first-level encoded stream, and a fifth quantization matrix can be used for the second-level encoded stream (for example, two matrices can be used), each of which can be equal to or unequal to each other, and equal to or not equal to any of the aforementioned first to third matrices. In this fifth mode, the fourth and fifth quantization matrices can be transmitted to the decoder or other devices to which the encoded stream will be transmitted.

[0191] In the procedures described in this disclosure, as mentioned above, each group of transform coefficients passed to this process typically belongs to a specific plane and layer. Typically, they have been scaled using a linear quantizer, which in some instances uses a non-centered dequantization offset. A scaling procedure can be applied to the transform coefficients as follows. This procedure can acquire the block's location information, along with a set of parameters that can indicate the block's properties (e.g., its size), the nature of the quantization operation (e.g., stride and offset values), and the enhancement layer to which it is applied. For example, the location can be indicated by a pair of coordinate values ​​or, for example, parameters (xTbP, yTbP) specifying the top-left sample of the current luma or chroma transform block relative to the top-left luma or chroma sample of the current image. This can be associated with a specific portion of the data representing the image and, for example, with the luma plane or chroma plane, depending on the plane to which the transform coefficients belong.

[0192] The aforementioned parameter specifies the size of the current transform block (in some instances, it may be called...). nTbS This parameter can have a value that depends on the type of transform, and more specifically, on the value of the parameter that defines the transform used for decoding. This type of parameter may be referred to in some instances as... transform_typeFurthermore, in some applications, it can have values ​​of 0, 1, or 2-3, corresponding to 2×2 oriented decomposition transformation, 4×4 oriented decomposition transformation, or a value or parameter specified as zero (because those elements are not used in the bitstream). In some instances, this parameter has a value of 0. transform_type The parameter can correspond to a layer number equal to 4, and if transform_type If the value is 1, then the corresponding number of layers can be 16. Size nTbS Parameters can be found transform_type It has a value of 2 when it is equal to zero, and in transform_type When the value is equal to 1, it can have a value of 4.

[0193] The other input to the process is typically in the form of an array of quantized coefficients obtained from entropy decoding. This can be called... TransCoeffQ And it has a size related to the size parameter mentioned above, specifically, it has a size ( nTbS )x( nTbS The array can include decoding units or blocks as described herein. This array may be referred to as... TransCoeffQ The step width value can be referred to as... stepWidth The parameter specifies the enhancement level index. The index of the enhancement level can be specified by a parameter, and in some instances, it can be called... idxLevel If a dequantization offset is used, this can be, for example, called... dQuantOffset The parameters are specified. These parameters typically specify the values ​​of the dequantization offset parameters to be applied. This process usually produces an array of dequantized transform coefficients. This may, for example, be a feature array. d The dimensions of [x][y] nTbS )x( nTbS It takes the form of an array.

[0194] Can be called d The values ​​in the output array of [x][y] are typically based on, for example, the step size as described above and / or, as applied to the quantization matrix (which may be called...). qm The enhanced hierarchy of indexes and entropy decoding quantized coefficients (which can be referred to as the corresponding elements) TransCoeffQ [ xTbP ][ yTbP Any one of the derivations in ]). Furthermore, the values ​​of the output array features can be derived additionally by applying the offset to the result of the aforementioned operations. For example, it can be called appliedOffset The parameters can be added with the values ​​to produce output element values.

[0195] Therefore, this can involve the following relationship to calculate each element d of the array:

[0196] d [x][y] = ( TransCoeffQ [ xTbP ][ yTbP ]*( stepWidth * qm [ yTbP +( idxLevel * 4)][ xTbP ]))+ appliedOffset

[0197] In the above calculation, which serves as an example of how dequantization can be performed at the decoder, idxLevel can be 0 or 1, representing level 2 and level 1, respectively. In the above calculation, and as described in other examples herein, the value from the quantization matrix is ​​selected based on specific coefficients and a specific enhancement level.

[0198] Relative to the offset that can be applied in some instances, as shown in the relationships above, it can be called appliedOffset This parameter is usually exported as follows.

[0199] In some cases, residual filtering can be applied conditionally based on whether or not it will be applied. appliedOffset This filtering can be applied to inverse transforms (e.g., Figure 6 The offset is applied after 410-1) and can be a deblocking filter. In these cases, the offset can be applied only when residual filtering is applied. For example, if it is specified whether a deblocking filter should be applied, it can be called... deblocking_signalled The parameter has a specific value, such as equal to 1, then it can be called based on the specified value. dQuantOffset The offset is calculated using the aforementioned parameters of the dequantized offset.

[0200] appliedOffset Also from dQuantOffset A single value is derived and is suitable for application to both positive and negative coefficient values. For example, appliedOffset It can be configured to be negative or positive, typically having the same absolute value as the dequantization offset parameter and correspondingly positive or negative signs. This can be... TransCoeffQ Apply if the value has a value less than zero; otherwise, if TransCoeffQ If the value is greater than zero, a value equal to the dequantized offset parameter can be assigned to the applied offset. If TransCoeffQ If the value is zero, then a zero value can also be assigned to the applied offset.

[0201] Therefore, in some instances, the offsets of the output array elements can be derived by the following algorithm (where... TransCoeffQ It can be the usual size ( nTbS )x( nTbS (an array of quantized coefficients containing entropy decoding).

[0202] If deblocking_signalled equals 1

[0203] If TransCoeffQ[ xTbP ][ yTbP ] < 0

[0204] appliedOffset = (dQuantOffset * -1)

[0205] else If TransCoeffQ [ xTbP ] [ yTbP ] > 0

[0206] appliedOffset = dQuantOffset

[0207] else

[0208] appliedOffset = 0

[0209] else

[0210] appliedOffset = 0

[0211] For example, a parameter or set of parameters can be signaled as described above and used to specify how the quantization matrix will be applied to the set of coefficients. This parameter can be... quant_matrix_mode The parameter specifies which quantization matrices, according to Table 1 below, will be used in the decoding process. The parameter can be one byte. quant_matrix_mode Parameters can be used to configure how the quantization matrix is ​​exported (for example, as described later below).

[0212] Thus, there are technical benefits that can be achieved by signaling and sending default values ​​in different ways. This can be particularly important because it allows for repetition on a set of frames or a subset of data. For example, the method could involve a signaling matrix, and then using a default value for a frame of that set. For example, these patterns could be set on a per-frame or per-group basis.

[0213] Table 1 - Quantization Matrix

[0214] quant_matrix_mode Value of type 0 Both quality levels use the default matrix 1 A matrix of modifiers is transmitted and should be used on two quality levels. 2 A matrix of modifiers is transmitted and should be used at quality level 2. 3 A matrix of modifiers is transmitted and should be used at quality level 1. 4 The two matrices of the modifier are transmitted – the first for quality level 2, the second for quality level 1. 5-7 Reserved_zero

[0215] Quantized matrix data semantics can be applied based on the following specific instances. The derivation can be referred to as... qm The quantization matrix of [y][x].

[0216] matrix qm It can have size k*M take N In other words, a matrix can be limited to correspond to M take N The quantization coefficients contained in the matrix, and qm It may contain, for example, k The corresponding quantization coefficients for each of the enhancement levelsM take N This data is in matrix form.

[0217] In the current instance, two enhancement levels are involved, namely levels 1 and 2 as described in this disclosure, and k It equals 2. If the maximum transformation described for these procedures is 4×4, and therefore... M and N Each can be equal to 4.

[0218] In some instances, this is achieved by using a transformation corresponding to the maximum available transformation (which is a 4×4 transformation as mentioned above, thus including 16 coefficients). M and N Value limit qm It can be obtained from qm Read and in the corresponding M take N A subset of coefficients in the matrix is ​​applied to perform the application of a quantization matrix that employs a smaller transformation.

[0219] In a specific instance, if the value of the quantization matrix mode parameter is equal to zero, that is, if quant_matrix_mode If the value is 0, the following default quantization matrix will be used in the quantization operation:

[0220] qm[ y][ x ] =

[0221] { 0.500 0.500 0.500 0.617}

[0222] { 0.862 0.610 1.064 0.781}

[0223] { 0.500 0.500 0.500 0.617}

[0224] { 3.125 1.851 1.851 1.316}

[0225] { 0.500 0.500 0.500 0.617}

[0226] { 0.862 0.610 1.064 0.781}

[0227] { 0.862 1.064 0.610 0.781}

[0228] { 3.125 1.851 1.851 1.316},

[0229] Where y = 0..3 are the coefficients to be used for enhancement level 2, and y = 4..7 are the coefficients to be used for enhancement level 1; and

[0230] If the quantization matrix mode parameter is equal to 1, then a matrix of modifiers is transmitted and used at both enhancement levels. In some applications, the matrix can be constructed using an iterative process. Therefore, it can be obtained row by row. qm The [x][y] values ​​are used to represent the values ​​in each row, and each column of the matrix is ​​filled as the rows are processed. The step size modifier parameter can be used in obtaining these matrix values, as described below. In the current example, the number of rows to be filled can be 8, and the number of columns can be 4, as described above in conjunction with the matrix size in this specific example. Specifically, the iterative process for this example can be written as:

[0231] for (y = 0; y < 8; y++)

[0232] for (x = 0; x < 4; x++)

[0233] qm[y][x] = step_width_modifier_2[x + y*4],

[0234] In this example, it can be referred to as step_width_modifier_2 The step width modifier parameter specifies the values ​​of the 16 level 2 enhancement coefficients to be applied at different levels of the transform coefficients; and

[0235] If the value of the quantization matrix mode parameter is equal to 2, then a matrix of the modifier is transmitted and used at enhancement level 2. Similarly, more precisely, an iterative procedure corresponding to the iterative procedure described above can be used:

[0236] for (y = 0; y < 4; y++)

[0237] for (x = 0; x < 4; x < 4)

[0238] qm[ y][ x ] = step_width_modifier_2[x + y*4]; and

[0239] If the value of the quantization matrix mode parameter is equal to 3, then a matrix of the modifier is transmitted and used on enhancement level 1:

[0240] for (y = 0; y < 4; y++)

[0241] for (x = 0; x < 4; x < 4)

[0242] qm[y + 4][ x ] = step_width_modifier_2[x + y*4]; and

[0243] If the quantization matrix mode parameter is equal to 4, then the two matrices of the modifier are transmitted, the first for enhancing level 2 and the second for enhancing level 1:

[0244] for (y = 0; y < 4; y++)

[0245] for (x = 0; x < 4; x < 4)

[0246] qm[ y][ x ] = step_width_modifier_2[x + y*4]

[0247] for (y = 0; y < 4; y++)

[0248] for (x = 0; x < 4; x < 4)

[0249] qm[ y][ x ] = step_width_modifier_1[x + y*4],

[0250] in step_width_modifier_1 This parameter specifies the values ​​of the 16 level 1 enhancement coefficients to be applied at different levels of the transformation coefficients.

[0251] As described above, in the matrix d The values ​​of [x][y] can be calculated, for example, as the product of the corresponding transformation coefficients at relevant elements in the matrix and the sum of the corresponding quantized matrix element values ​​at the columns, where the columns are determined by a parameter specifying the size of the current transform block and... levelIdxSwap The product of parameters is identified, and in the corresponding row, the step width modifier parameter value corresponds to the element, wherein the offset described above is also typically applied additively to the product.

[0252] The dequantization process described above can be performed according to the following ordered steps. The dequantization process according to the aforementioned example can be invoked, where the brightness position ( xTbY , yTbY ), set to equal nTbS Transform size (i.e., the size of the current transform block), size ( nTbS )x( nTbS The array described above TransCoeffQ The step width parameter is taken as input. The output can therefore be the dimension ( nTbS )x( nTbS An array of dequantized coefficients (e.g., the dequantized transformed residuals), which in this instance may be called dequantCoeff .

[0253] Adaptive quantization based on time information

[0254] As described in this article, quantization can be adjusted based on time information with careful consideration. For example, smaller quantization steps can be used for static features. This allows for finer quantization of differences to ensure that sufficient additional information is provided. If the information becomes more static, it becomes clearer what the values ​​should be.

[0255] Typically, quantization is performed without any temporal information. There is usually a stride for a frame, but the proposed technique allows for distinctions within and between pieces. According to the technique proposed in this paper, temporal information and quantization are entangled, whereas in conventional decoding techniques they remain separate. The proposed technique can be considered as a change in stride before quantization is performed, and the decisions made regarding the stride change are related to temporal information.

[0256] Figure 9 This diagram illustrates an exemplary sequence of frames in a video sequence, demonstrating the use of time information in some example methods. Specifically, the diagram shows five frames in a video sequence, where two frames (frame N-2 and frame N-1) precede the current frame N, and two frames (frame N+1 and frame N+2) follow the current frame N. Figure 9 In this example, a region (dark rectangle) in the current frame is shown that can be predicted at least partially from a corresponding region in a previous frame and / or from a corresponding region in a subsequent frame. In this example, the region remains at the same coordinates (x0 and y0) within the first four frames and moves to different coordinates (x1 and y1) in the last frame. In this example, the region can be considered substantially static in the first four frames and motion-compensated in the last frame due to relative movement of the region with respect to frames N+1 and N+2. The remainder of the frame (shaded area) cannot be temporally predicted between frames.

[0257] Figure 10 An example of an encoder and decoder system according to the present invention is illustrated. Specifically, the encoder receives a sequence of several consecutive frames and performs analysis to derive timing information for each frame using a timing prediction module. This timing information includes determining which regions of a particular frame can be predicted at least partially from one or more other frames (previous and / or subsequent) – time-predicted regions – and which regions cannot be predicted at least partially from one or more other frames (previous and / or subsequent) – non-time-predicted regions. This information is then used by a quantizer to determine different quantization parameters for the time-predicted and non-time-predicted regions. The quantization parameters may be linked by a value (step modifier) ​​that allows other quantization parameters to be derived from one of the quantization parameters. In one embodiment, the step modifier may be a default value, in which case it may not be signaled to avoid unnecessary signaling. In another embodiment, the step modifier is signaled to the decoder.

[0258] Specifically, the signaling may include a bit indicating whether the default modifier should be used at the decoder or whether the default modifier should be modified by a custom modifier. In the latter case, another field (e.g., an 8-bit field) may be signaled along with the actual value of the custom modifier.

[0259] At the decoder, there is a module that reads the bitstream and determines timing information for each decoding unit (e.g., transform block), and another module that reads the bitstream and determines the presence of a SW modifier. Specifically, the timing information module determines whether a particular decoding unit (i.e., an inter-frame decoding unit that will undergo timing prediction) will be reconstructed at least partially based on timing information from one or more previous and / or subsequent frames, or whether it will be reconstructed independently (i.e., an intra-frame decoding unit that will not undergo timing prediction). A stride calculation module calculates the stride to be applied to the inter-frame and intra-frame decoding units. Specifically, this module may first check whether the bitstream signals that different stride modifiers should be used. If the bitstream signals that different stride modifiers should be used, the module then reads different fields within the bitstream to derive the stride modifier to be used. If the bitstream does not signal that different stride modifiers should be used, a default stride modifier is used. Once the correct stride modifier has been derived, the module derives the stride for intra-frame decoding units (intra-frame stride) and the stride for inter-frame decoding units (inter-frame stride), where the stride modifier is used to derive the intra-frame stride from the inter-frame stride and / or vice versa (depending on the relationship between the two strides). The inter-frame stride is lower than the intra-frame stride. The dequantizer module then uses this information to dequantize the inter-frame decoding units using the inter-frame stride and the intra-frame decoding units using the intra-frame stride.

[0260] Specifically, a non-limiting embodiment of the hierarchical enhancement encoder generates signaling information at least in part based on what region of the echelon of residual data coefficients to be decoded, containing the corresponding residual data coefficients from previous frames.

[0261] The enhancement encoder generates echelons (e.g., layers) of residual data at a subsequent higher quality level (e.g., resolution). Each echelon is combined with a predicted reproduction of the signal at the same quality level to generate a final reproduction of the signal at said quality level. When the echelon of residual data has an available time buffer, one or more regions (e.g., mosaicks) of the residual data echelon are decoded at least partially based on the time buffer by combining the corresponding values ​​contained in the time buffer with values ​​derived by decoding the encoded data stream for the signaled regions of the image that are time-predicted. Accordingly, the encoder generates signaling information to identify the portions of the echelon that must be decoded at least partially based on the time buffer, and the encoded residual data for the entire echelon (i.e., for both the time-predicted and untime-predicted portions).

[0262] By way of non-limiting examples, the described embodiments utilize the following relationship:

[0263] Intra-residuals_quantization_step-width=master_quantization_step- width*(1+step-width_modifier / 255) The step-width_modifier is an 8-bit value that is optionally communicated between GOP configuration parameters, with a default value of 128.

[0264] For each decoding unit of the residual data (e.g., a transform block, but this can be done on units of other sizes or on multiple decoding units or multiple blocks (e.g., mosaicks), the encoder must choose between two options:

[0265] ●Time Prediction - In this case, the residual contained in the corresponding position of the time buffer is subtracted from the residual before transformation and quantization. The quantization process is performed using the main step width of the frame.

[0266] ●Non-time prediction - In this case, it involves transforming and quantizing the residuals. The quantization process is performed using the residual inner step width calculated from the main step width of the frame, according to the formula shown above.

[0267] For both options, the residual is reconstructed, and the distortion D is calculated as the difference between the original residual and the reconstructed residual (i.e., after dequantization and inverse transformation).

[0268] Based on the options, we then calculate the estimated cost C:

[0269] C = D + lambda * R

[0270] Where R is a parameter that estimates the rate requirement for each option based on the coded bits, and lambda is a Lagrangian parameter that is calculated as a function of the step width (e.g., in a non-limiting embodiment, lambda = 1.84 * step width).

[0271] The theoretical basis for the above process is that if the residual in the current frame is similar to the corresponding residual in a previous frame, then the estimation cost C will be smaller for the temporal prediction option. At the same time, it is reasonable to expect that if a region of the image is currently static (e.g., the encoder chooses the temporal prediction option for the current frame), it could also be static in the near future, and therefore using finer quantization for the temporal prediction option compared to the non-temporal prediction option would be efficient.

[0272] In a non-limiting embodiment, the encoder identifies cases where it would be more efficient to pass the step-width_modifier parameter to the decoder instead of using the default value.

[0273] Time information in layered decoding schemes

[0274] The following text is in Figure 11-16The implementation of the time buffer in the hierarchical decoding scheme is described in the context of the above. Only variations of the schematic diagrams and processes described above are described below. It will be understood that the following is provided to illustrate the context of the adaptive quantization described herein for a hierarchical decoding scheme utilizing a time buffer.

[0275] Figure 11 A first example encoder 100 is shown. The components shown can also be implemented as steps in the corresponding encoding process. Level 1 encoding operation 1114 operates with an optional Level 1 time buffer 1130, which can be used to apply timing processing as further described below. Level 2 encoding operation 1126 also operates with an optional Level 2 time buffer 1132, which can be used to apply timing processing as further described below. Level 1 time buffer 1130 and Level 2 time buffer 1132 can operate under the control of a timing selection component 1134. The timing selection component 1134 can receive one or more of the input video 102 and the output of the downsampled video 1104 to select a timing mode. This is explained in more detail in later examples. Figure 12 The first instance decoder 1200 is shown.

[0276] Figure 13A and 13B Different variations of the second example encoders 1300 and 1380 are shown. The second example encoders 1300 and 1380 may include... Figure 1 The first instance of the encoder 1100 implementation scheme. Figure 13A and 13B The examples illustrate the encoding steps of the stream in more detail to provide examples of how the steps can be performed. Figure 13A The first variant is shown, which has time predictions provided only in the second level of the enhancement process (i.e., relative to level 2 encoding). Figure 13B A second variant of time prediction is shown, which is performed in two enhancement levels (i.e., levels 1 and 2).

[0277] Figure 13A A variation of the second instance encoder 1300 is shown, in which timing prediction is performed as part of the level 2 encoding process. Timing prediction is performed using a timing selection component 1334 and a level 2 timing buffer 1332. The timing selection component 1334, as described in more detail below, determines the timing processing mode and accordingly controls the use of the level 2 timing buffer 1332. For example, if no timing processing will be performed, the timing selection component 1334 may instruct the content of the level 2 timing buffer 1332 to be set to 0. Figure 13B A variant of the second instance encoder 1380 is shown, in which time prediction is performed as part of both the level 1 and level 2 encoding processes. Figure 13BIn addition to the level 2 time buffer 1332, a level 1 time buffer 1330 is also provided. Although not shown, other variations in which time processing is performed at level 1 instead of level 2 are also possible.

[0278] When selecting time forecasting Figure 3 The second instance encoders 1300 and 1380 of A or 3B can further modify the coefficients (i.e., the transformed residuals output by the transform component) by subtracting a corresponding set of coefficients derived from an appropriate time buffer. The corresponding set of coefficients may include a set of coefficients for the same spatial region (e.g., the same decoding unit located within the frame) derived from a previous frame (e.g., coefficients for the same region of the previous frame). These coefficients can be derived or otherwise obtained from the time buffer. The coefficients obtained from the time buffer may be referred to herein as temporal coefficients. Subtraction can be applied by a subtraction component, such as a third subtraction component (for corresponding levels 2 and 1). In summary, when temporal prediction is applied, the encoded coefficients correspond to the difference between the frame and another frame in the stream. The other frame can be an earlier or later frame in the stream (or a block within a frame). Therefore, instead of encoding the residual between the upsampled recreated stream and the input video, the encoding process can encode the difference between the transformed frame in the stream and the transformed residual of the frame. Thus, entropy can be reduced. Timing predictions can be selectively applied to groups of decoding units (referred to herein as “pieces”) based on control information, and the application of timing predictions at the decoder can be achieved by sending additional control information along with the encoded stream (e.g., within the header or as another surface).

[0279] like Figure 13A and 13B As shown in the figure, when the time prediction is active, each transformation coefficient can be:

[0280] ∆ = Fcurrent − Fbuffer

[0281] The time buffer can store data associated with previous frames. Time prediction can be performed against a single color plane or against multiple color planes. Generally, subtraction can be applied to video "frames" as a pro-element subtraction, where the elements of a frame represent transformed coefficients, with the transformation applied relative to a specific n-by-n decoding unit size (e.g., 2×2 or 4×4). The difference generated by the time prediction (e.g., the aforementioned difference) can be stored in a buffer for use in subsequent frames. Therefore, in practice, the residual generated by the time prediction is the coefficient residual relative to the buffer.

[0282] although Figure 13A and 13B This demonstrates that time prediction can be performed after the transformation operation, but it can also be performed after the quantization operation. This avoids the need to apply a Level 2 inverse quantization component and / or a Level 1 inverse quantization component 360. Therefore, as... Figure 13A and 13B As shown and described above, after the encoding process is performed, the output of the second instance encoder is an encoded base stream and one or more enhancement streams, which preferably include an encoded layer 1 stream for a first enhancement layer and an encoded layer 2 stream for another or a second enhancement layer.

[0283] Figure 14A and 14B The corresponding variations of the second instance decoders 1400 and 1480 are shown. As can be clearly seen, the decoding steps and components are described in more detail to provide an example of how decoding can be performed. (As...) Figure 13A and 13B , Figure 14A This illustrates a variant where time prediction is used only for the second level (i.e., level 2), and Figure 14B This illustrates a variation where time prediction is used for two levels (i.e., levels 1 and 2). As previously mentioned, consider another variation (e.g., level 1, but not level 2) where signaling information can be used to control the configuration. Figure 14A In this process, time prediction is applied during level 2 decoding. Figure 14A In this example, timing prediction is controlled by timing prediction component 1466. In this variant, control information for timing prediction is extracted from the encoded level 2 stream, as indicated by the arrows from the stream to timing prediction component 1466. For example... Figure 14B In other embodiments shown, control information for time prediction may be sent separately, for example, in the header, from the encoded layer 2 stream. The time prediction component 1466 controls the use of the layer 2 time buffer 1432, for example, it may determine the time pattern and control time refresh, as described with reference to a later example. The contents of the time buffer 1432 may be updated based on data from previous frames of residuals. When the time buffer 1432 is applied, its contents are added 1468 to a second set of residuals. Figure 14A In the middle, the contents of time buffer 1432 are added to layer 2 decoding component 1446 (in Figure 14A In this process, the output of the time buffer (which performs entropy decoding, inverse quantization, and inverse transform) is used. In other instances, the contents of the time buffer can represent any set of intermediate decoded data, and therefore, the addition of 468 can be appropriately shifted to apply the contents of the time buffer at the appropriate stage (e.g., if the time buffer is applied during the dequantization coefficient stage, the addition can be placed before the inverse transform). The second set of time-corrected residuals is then combined with the upsampled output to generate the decoded video. The decoded video is at a Level 2 spatial resolution, which can be higher than a Level 1 spatial resolution. The second set of residuals applies corrections to the (inspectable) upsampled reconstructed video, where the corrections are added back in detail and improve the sharpness of lines and features.

[0284] Figure 14BThis demonstrates a variation of the second instance decoder. In this case, the timing prediction component receives timing prediction control data from the header. The timing prediction component controls both level 1 and level 2 timing predictions, but in other instances, separate control components may be provided for the two levels as needed. Figure 14B This demonstrates how the reconstructed second residual set, added to the output of the layer 2 decoding component, can be fed back to be stored in the layer 2 time buffer 432 for the next frame (for clarity, from...). Figure 14A (The feedback is omitted). A Level 1 time buffer 1430, operating in a similar manner to the Level 2 time buffer 1432 described above, is also shown, and the feedback loop for the buffer is shown in this figure. The contents of the Level 1 time buffer 1430 are added to the Level 1 residual processing pipeline via summation. Again, the location of this summation can vary along the Level 1 residual processing pipeline depending on where time prediction is applied (e.g., if time prediction is applied in the transformed coefficient space, it can be located before the Level 1 inverse transform component).

[0285] Figure 14B This demonstrates two ways in which timing control information can be relayed to the decoder. The first method is via header 436, as described above. The second method, which can be used as an alternative or additional relay path, is via data encoded within the residual itself. Figure 14B This demonstrates a case where the data can be encoded as HH transform coefficients and therefore extracted after entropy decoding. This data can be extracted from the Level 2 residual processing pipeline and passed to the time prediction component 1466.

[0286] In some instances, at least two time patterns may exist.

[0287] The first time mode does not use a time buffer or uses a time buffer with all zero values. The first time mode can be considered an intra-frame mode because it only uses information from the current frame. In the first time mode, after any applied grading and transformation, coefficients can be quantized without modification based on information from one or more previous frames.

[0288] A time buffer can be used, for example, a second time mode with a time buffer that may have non-zero values. The second time mode can be considered an inter-frame mode because it uses information from outside the current frame, such as from multiple frames. In the second time mode, after any applied grading and transformation, the dequantization coefficients of the previous frame -C can be subtracted from the coefficients to be quantized. x,y,n,inter = C x,y,n -dqC x,y,n-1 .

[0289] In one scenario, the first time mode can be applied by subtracting from the set of zero-time coefficients. In another scenario, subtraction can be performed selectively based on time signaling data.

[0290] Figure 15A This demonstrates an example of time processing that can be performed at the encoder. Figure 15A This demonstrates the timing processing subunit of the example encoder. The timing processing subunit receives the set of residuals, indicated by R. The timing processing subunit outputs the set of quantized coefficients, indicated by qC. In this example, the timing processing subunit also outputs timing signaling data, indicated by TS, for transmission to the decoder. Figure 15B The corresponding instance decoder is shown, where the decoder receives the temporal_refresh bit for each frame and the temporal_mode bit for each decoding unit. Figure 15B In the instance decoder, a time processing subunit is provided at the decoder.

[0291] Other implementation plans

[0292] In some embodiments as described herein, in example implementations, it should be noted that the following process may be implemented for lossless compression:

[0293] If stepWidth > 16, then the deadZoneWidthOffset is exported as follows:

[0294] deadZoneWidthOffset [x][y] = (((1 << 16) − ((Aconst * (qm[x + (levelIdxSwap * nTbs)][y] + stepWidthModifier [x][y])) + Bconst) >> 1) * (qm [x + (levelIdxSwap * nTbs)][y] + stepWidthModifier [x][y])) >> 16

[0295] If stepWidth <= 16, then the deadZoneWidthOffset is exported as follows:

[0296] deadZoneWidthOffset [x][y] = stepWidth >> 1

[0297] In other words, when the step width is less than 16, the deadzonewidthoffset does not depend on the parameter matrix.

[0298] Adaptive Quantization Implementation Plan

[0299] In a particular implementation, if the first variable flag (e.g., temporal_enabled_flag The first variable flag is equal to 1, the second variable flag (e.g., temporal_refresh_bit_flag) is equal to 0, and the third variable flag (e.g., ... temporal_ tile_intra_signalling_enabled_flag ) equals 1, and the variables based on the group of coefficients of the entropy-decoded time signal (e.g., TransformTempSig If ) equals 0, then the variable stepWidth Modified to

[0300] Floor(stepWidth * (1 − (Clip3(0, 0.5, (temporal_step_width_modifier / 255)))))

[0301] in temporal_step_width_modifier Specifies the value of the variable step size modifier used to calculate the transformation using time prediction. If temporal_step_width_modifier_signalled_flag If it equals 0, then temporal_step_width_modifier It can be set to 48.

[0302] Other example implementation schemes

[0303] The appendix below provides examples of non-limiting implementations that embody the principles described elsewhere in this document. These examples can be used to provide context for the described features.

[0304] Other example encoders and decoders

[0305] Figures 25 and 26 respectively show Figure 1 , 3 The encoder architectures of A and 3B and Figure 2 , 5 Variations of the decoder architectures of A and 5B.

[0306] exist Figure 16 The diagram illustrates the encoding process 2500 used to create the bitstream. First, the input sequence 2502 is fed to the first downsampler 2504, followed by the second downsampler 2506 (i.e., the successive downsamplers referred to as downscalers in the diagram) and processed according to the selected scaling mode. Figure 16 The variation differs from the previous instance in that there are additional downsampling and upsampling stages before the base layer. For example, it is possible to have an additional downsampling stage, represented as the second downscalor 2506, before passing the data to the base encoder 2512, and an additional upsampling stage (represented as...) after receiving the decoded data from the base layer. Figure 16 The first up scaler 2508 is possible. In some instances, a given scaling mode can be used to turn the down scaler and up scaler pair on and off at each level. In one case, the scaling mode can indicate the scaling direction, such as horizontal downsampling / upsampling only as described herein. If the second down scaler 2506 and the first up scaler 2508 are disconnected, spatial scaling is similar to... Figure 1 , 3 Space scaling for A and 3B.

[0307] Figure 16 In this example, following the previous one, a base codec is used, which generates a base bitstream 2516 according to its own specification. This encoded base can be included as part of a combined bitstream within the current video decoding framework structure.

[0308] With or without additional upscaling, at the first subtraction component 2520, a reconstructed base image, such as the decoded form of the base coded frame, is subtracted from the first-order down-scaled input sequence to generate sub-layer 1 residuals (as described herein, layer 1 residual data). These residuals form the starting point for the encoding process of the first enhancement layer. Transform component 2521, quantization component 2523, and entropy coding component 2524 (and others) as described herein process the first (layer 1) residual set to generate (layer 1) entropy-coded quantized transform coefficients 2526.

[0309] exist Figure 16 In this example, following the previous instance, the entropy-encoded quantized transform coefficients from sublayer 1 are processed by the in-loop decoder, which performs inverse or decoding operations. These operations simulate the decoding process of the first set of residuals that would otherwise be performed at the decoder. Figure 16 In examples, these include an entropy decoding component 2525, an inverse quantization component 2527, an inverse transform component 2528, and a level 1 filter 2530. These may be similar to the components described previously. The processed or “decoded” first set of residuals is added at summing component 2532 to the data derived from the output of the base encoder (e.g., decoded and optionally upscaled) to generate the reconstructed frame. Figure 16 In the process, the reconstructed frame is processed by a second upscaler 2534. The use of the upscaler may again depend on the selected scaling mode. Finally, at the second subtraction component 2536, the residuals of the second sublayer 2 (which may also be referred to as the L2 layer) are calculated by subtraction of the input sequence and the upscaled reconstruction. These form a second (layer 2) residual set, which is further processed by a set of decoding components or tools including a transform component 2541, a timing prediction component 2542, a quantization component 2543, and an entropy coding component 2544. The output is a set of layer 2 coefficient layers 2546. As described in other examples, if the timing mode is activated, additional timing prediction can be applied to the transform coefficients by the timing prediction component 2542 to remove some temporally redundant information and reduce the energy of the layer 2 residual stream (e.g., the number of values ​​and the number of non-zero residual values). The entropy-coded quantized transform coefficients of sublayer 2 and the timing layer 2556, which specifies the use of timing prediction on a block-by-block basis, are included in the enhanced bitstream. The time layer 2556 may include time signaling as described with reference to the previous examples. It may be entropy encoded by the entropy encoding component 2557. The entropy encoding component 2557 may apply at least run-length encoding, as discussed in the reference examples.

[0310] Encoder 2500 can be configured with a set of encoder configuration information 2565, for example, as shown in the reference. Figure 14A to 1 As described in the 4C example. This information can be transmitted to the decoder as a set of header 2566 of the output bitstream. Figure 16In the encoder, the combined bit stream may include a header 2566, a time layer 2556, level 2 (L2) coding coefficients 2546, level 1 (L1) coding coefficients 2526, and an encoded base stream 2516.

[0311] Figure 17 This demonstrates a variation of decoder 2600 based on one example. The decoder may include variations of the decoder shown in any of the other figures herein. Figure 17 The decoder can be connected with Figure 16 Use it with an encoder.

[0312] First, in order to create the output sequence of frames, the decoder 2600 analyzes the bitstream. For example... Figure 17 As can be seen, the process can be further divided into three parts.

[0313] To generate the decoded base image (e.g., at layer 0), the extracted base bitstream 2616 is fed to the base decoder 2618. Depending on the selected scaling mode, this reconstructed image can be upscaled by an additional first upscaler 2608 before a summing component 2630, which sums the first (layer 1) residual set. The input from the first upscaler 2608 to the summing component 2630 can be referred to as the preparatory intermediate image.

[0314] After (or in parallel with) the base layer decoding, the enhancement layer bitstream (containing two residual sub-layers) needs to be decoded. First, the coefficients 2626 belonging to sub-layer 1 (L1) are decoded using the inverse form of the decoding components or tools used during the encoding process. Therefore, the layer 1 coefficient layer 2626 is processed sequentially by the entropy decoding component 2671, the inverse quantization component 2672, and the inverse transform component 2673. Furthermore, a sub-layer 1 (L1) filter 2632 may be applied to smooth the boundaries of the transform blocks (i.e., decoding units). The output of the sub-layer 1 (L1) decoding process can be referred to as the enhanced sub-layer 1 output. This enhanced sub-layer 1 output is added to the preparatory intermediate image at the first (lower) summing component 2630 to produce a combined intermediate image. Again, depending on the scaling mode, a second upscaler 2687 may be applied, producing the resulting preparatory output image. The preparatory output image is provided to the second upper summing component 2658. It has the same size as the overall output image.

[0315] As a final step, the encoded coefficients 2646 of the second enhancement sublayer 2 are decoded. Again, this is done using a set of inverse decoding components or tools as described in other examples in this document. Figure 17These components include an entropy decoding component 2681, an inverse quantization component 2682, and an inverse transform component 2683. If the time mode is activated, the time prediction component 2685 can apply time prediction. Time prediction can be applied at any point within the second enhancement sublayer 2. In one case, it is applied to the quantized transform coefficients. Time prediction can be applied based on signaling received as time layer 2656. Figure 17 In this process, time layer 2656 is decoded by entropy decoding component 2690 (e.g., via run length decoding). The output of time prediction is provided as the output of enhanced sublayer 2 to a second upper summing component 2658. It is then added by said summing component 2658 to the prepared output picture to form a combined output picture 2660 as the final output of the decoding process.

[0316] Furthermore, the decoding process can be controlled according to the decoder configuration 2692 transmitted in the header 2666 of the bit stream.

[0317] As described in the examples above, unlike the contrasting scalable codecs, the new method described in this paper is completely agnostic to the codec used to encode the lower layer. This is because the upper layer can be decoded without any information about the lower layer. Figure 17 As shown, the decoder receives multiple streams generated by the encoder. These can be approximately five streams, comprising: a first coded stream (coded basis) generated by feeding a downsampled form of the input video to a base codec (e.g., AVC, HEVC, or any other codec); a second coded stream (layer 1 coefficient layer) generated by processing the residual (layer 1 residual) obtained by reconstructing the difference between the base codec video and the downsampled form of the input video; a third coded stream (layer 2 coefficient layer) generated by processing the residual (layer 2 residual) obtained by reconstructing the oversampled form of the base decoded video and the input video; a fourth coded stream (e.g., in the form of a time layer) generated by time processing to instruct the decoder; and a fifth stream (header) generated for configuring the decoder. The coded basis stream is decoded by implementing a decoding algorithm corresponding to the encoding algorithm implemented by the base codec used in the encoder by the base decoder, and the output of this decoding is the decoded basis. The layer 1 coefficient group is decoded separately and independently to obtain the layer 1 residual data. Furthermore, the level 2 coefficient group is decoded separately and independently to obtain level 2 residual data. The decoded base, level 1 residual data, and level 2 residual data are then combined. Specifically, the decoded base and level 1 residual data are combined to generate an intermediate image. The intermediate image can then be upsampled and further combined with the level 2 residual data.

[0318] Furthermore, the new method employs an encoding and decoding process that processes the image without using any inter-block prediction. In practice, it processes the image by transforming N×N blocks of pixels (e.g., 2×2 or 4×4) and processing these blocks independently of each other. This achieves efficient processing and independence from neighboring blocks, thus allowing for parallelization of image processing.

[0319] In summary, reference Figure 17 The following describes a non-limiting exemplary embodiment. Figure 17 An exemplary decoding module 2600 is depicted. The decoding module 2600 receives multiple input bit streams, including an encoded base 2616, a level 1 coefficient group 2626, a level 2 coefficient group 2646, a time coefficient group 2656, and a header 2666.

[0320] Generally, the decoding module 2600 processes two layers of data. The first layer, the base layer, includes the received data stream 2616 containing the encoded base. The encoded base 2616 is then sent to the base decoding module 2618, which decodes the encoded base 2616 to produce a decoded base image. The base decoding can be performed by the decoder implementing any existing base codec algorithm, such as AVC, HEVC, AV1, VVC, EVC, VC-6, VP9, ​​etc., depending on the encoding format of the encoded base.

[0321] The second layer, the enhancement layer, further consists of two enhancement sub-layers. The decoding module receives a first coefficient group, namely layer 1 coefficient group 2626, which is then passed to the entropy decoding module 2671 to generate a decoded coefficient group. These are then passed to the inverse quantization module 2672, which uses one or more dequantization parameters to generate a dequantized coefficient group. These are then passed to the inverse transform module 2673, which performs an inverse transform on the dequantized coefficient group to generate a residual at enhancement sub-layer 1 (layer 1 residual). The residual can then be filtered by a smoothing filter 2632. The layer 1 residual (i.e., the decoded first enhancement sub-layer) is applied to the processed output of the base image.

[0322] The decoding module receives a second coefficient group, namely level 2 coefficient group 2646, which is then passed to the entropy decoding module 2681 to generate a decoded coefficient group. These are then passed to the inverse quantization module 2682, which uses one or more dequantization parameters to generate a dequantized coefficient group. The dequantization parameters used for enhancing sublayer 2 may differ from those used for enhancing sublayer 1. The dequantized coefficient group is then passed to the inverse transform module 2683, which performs an inverse transform on the dequantized coefficient group to generate the residual at enhancing sublayer 2 (level 2 residual).

[0323] Implementation Plan

[0324] The methods and processes described herein may be embodied as code (e.g., software code) and / or data in both encoders and decoders, for example, implemented in a streaming server or client device or a client device decoding from data storage. Encoders and decoders may be implemented in hardware or software, as is well known in the field of data compression. For example, hardware acceleration using a specially programmed graphics processing unit (GPU) or a specially designed field-programmable gate array (FPGA) may provide some efficiency. For completeness, such code and data may be stored on one or more computer-readable media, which may contain any means or medium capable of storing code and / or data for use by a computer system. When a computer system reads and executes the code and / or data stored on the computer-readable medium, the computer system executes methods and processes embodied as data structures and code stored within the computer-readable storage medium. In some embodiments, one or more steps of the methods and processes described herein may be executed by a processor (e.g., a processor of a computer system or a data storage system).

[0325] Generally, any of the functionalities described in this text or illustrated in the diagrams may be implemented using software, firmware (e.g., a fixed logic circuit system), programmable or non-programmable hardware, or a combination of these embodiments. Generally, as used herein, the terms "component" or "function" refer to software, firmware, hardware, or a combination of these. For example, in the case of a software embodiment, the terms "component" or "function" may refer to program code that performs a specified task when executed on one or more processing devices. The illustrated separation of components and functions into distinct units may reflect any actual or conceptual physical grouping and allocation of such software and / or hardware and tasks.

[0326] appendix

[0327] The following are examples of non-limiting implementations that embody the principles described elsewhere in this document. These examples can be used to provide context for the described features.

[0328] Syntax and Semantics

[0329] The syntax table specifies a superset of the syntax for all allowed bitstreams. Additional constraints on the syntax can be specified directly or indirectly as needed.

[0330] Note that the actual decoder should implement a component for identifying entry points in the bitstream, and another component for identifying and processing non-matching bitstreams. Methods for identifying and processing errors and other similar issues should also be implemented.

[0331] Process payload - image configuration

[0332]

[0333] Data block unit image configuration semantics

[0334] The no_enhancement_bit_flag specifies that there is no enhancement data for all layerIdx < nLayers in the picture.

[0335] The quant_matrix_mode specifies which quantization matrix, according to the table below, is to be used during the decoding process. When quant_matrix_mode is not present, it is inferred to be equal to 0.

[0336] Quantization matrix

[0337] quant_matrix_mode Value of type 0 Each enhancement sublayer uses the matrix used for the previous frame, unless the current image is an IDR image, in which case both enhancement sublayers use the default matrix. 1 The two enhancement sub-layers use the default matrix 2 A matrix of modifiers is transmitted and should be used on two residual planes. 3 A matrix of modifiers is transmitted and should be used on the residual plane of enhanced sublayer 2. 4 A matrix of modifiers is transmitted and should be used on the residual plane of enhanced sublayer 1. 5 The two matrices of the modifier are transmitted – the first is used to enhance the residual plane of sublayer 2, and the second is used to enhance the residual plane of sublayer 1. 6-7 reserve

[0338] The dequant_offset_signalled_flag specifies whether the dequantization offset method and the value of the offset parameter to be applied during dequantization are signalled. If equal to 1, the method for dequantization offset and the value of the dequantization offset parameter are signalled. When dequant_offset_signalled_flag is not present, it is inferred to be equal to 0.

[0339] The picture_type_bit_flag specifies whether the encoded data is sent on a frame basis (e.g., progressive mode or interlaced mode) or on a field basis (e.g., interlaced mode) according to the table below.

[0340] Picture type

[0341] picture_type_bit_flag Value of type 0 frame 1 Fields

[0342] The field_type_bit_flag specifies, according to the table below, whether the data sent, if picture_type is equal to 1, is for the top or bottom field.

[0343] Field type

[0344] field_type_bit_flag Value of type 0 top 1 bottom

[0345] The temporal_refresh_bit_flag specifies whether the temporal buffer should be refreshed for the picture. If equal to 1, the temporal buffer should be refreshed. For IDR pictures, the temporal_refresh_bit_flag will be set to 1.

[0346] `temporal_signalling_present_flag` specifies whether the temporal signaling coefficient group exists in the bit stream. When `temporal_signalling_present_flag` does not exist, it is inferred to be equal to 1 if `temporal_enabled_flag` is equal to 1 and `temporal_refresh_bit_flag` is equal to 0; otherwise, it is inferred to be equal to 0.

[0347] `step_width_level2` specifies the step width value to be used for the coded residual in the decoding enhancement sublayer 2 for the luma plane. The step width value to be used for the coded residual in the decoding enhancement sublayer 2 for the chroma plane will be calculated as `Clip3(1, 32,767,((step_width_level2*chroma_step_width_multiplier)>>6))`.

[0348] `step_width_level1_enabled_flag` specifies whether the step width to be used when decoding the encoded residual in Enhancement Sublayer 1 is the default value or signaled. It should be 0 (the default) or 1 (the signaled value from `step_width_level1`). The default value is 32,767. When `step_width_level1_enabled_flag` is not present, it is inferred to be equal to 0.

[0349] The `dithering_control_flag` specifies whether dithering should be applied. It should be 0 (disable dithering) or 1 (enable dithering). If `dithering_control_flag` is not present, it is assumed to be equal to 0.

[0350] step_width_level1 specifies the step width value to be used when decoding the encoded residual in enhancement sublayer 1.

[0351] The `level1_filtering_enabled_flag` specifies whether the level 1 unblocking filter should be used. It should be 0 (filtering disabled) or 1 (filtering enabled). If `level1_filtering_enabled_flag` does not exist, it is assumed to be equal to 0.

[0352] qm_coefficient_0[layerIdx] specifies the value of the quantization matrix scaling parameter when quant_matrix_mode is equal to 2, 3 or 5.

[0353] qm_coefficient_1[layerIdx] specifies the value of the quantization matrix scaling parameter when quant_matrix_mode is equal to 4 or 5.

[0354] The `dequant_offset_mode_flag` specifies the method used to apply the dequantization offset. If it equals 0, the transmitted `dequant_offset` is used as the parameter, and the default method applies. If it equals 1, the transmitted `dequant_offset` parameter is used, and the constant offset method applies.

[0355] The `dequant_offset` parameter specifies the value of the dequantization offset parameter to be applied. The value of the dequantization offset parameter should be between 0 and 127 (inclusive).

[0356] `dithering_type` specifies which type of dithering will be applied to the final reconstructed image according to the table below. (Error! Reference source not found.)

[0357] Shaking

[0358] dithering_type Value of type 0 none 1 unified 2-3 reserve

[0359] dithering_strength specifies a value between 0 and 31.

[0360] Decoding process

[0361] A decoding process is specified such that when the decoding process associated with the specified profile is invoked for a bitstream that conforms to the specified profile and level, all decoders conforming to the profile will produce numerically identical cropped decoded output images. Any decoding process that produces the same cropped decoded output image as produced by the process described herein (with the correct output order or output timing, as specified) meets the decoding process requirements.

[0362] General decoding process for L-2 encoded data blocks

[0363] The input for this process is:

[0364] Specify the sample position (xTb0, yTb0) of the top-left sample of the current transform block relative to the top-left sample of the current image, based on the value of the variable transform_type. The specified sub-clause is the size of the current transform block, nTbS (if transform_type equals 0, then nTbS = 2; if transform_type equals 1, then nTbS = 4).

[0365] Such as variables exported from other places: temporal_enabled_flag, temporal_refresh_bit_flag, temporal_signalling_present_flag, and temporal_step_width_modifier.

[0366] Specifies an array recL2ModifiedUpsampledSamples of size (nTbS) x (nTbS) of the upsampled reconstructed samples generated by a procedure specified elsewhere in the current block.

[0367] The array TransformCoeffQ is specified as having (nTbS)x(nTbS) values ​​for the quantized transform coefficients of L-2 entropy decoding.

[0368] If both `temporal_signalling_present_flag` and `temporal_tile_intra_signalling_enabled_flag` are equal to 1, then the variable `TransformTempSig` corresponds to the value in `TempSigSurface` at position (xTb0>>nTbs, yTb0>>nTbs); and if `temporal_tile_intra_signalling_enabled_flag` is also set to 1, then the variable `TileTempSig` corresponds to the value in `TempSigSurface` at position ((xTb0%32)*32, (yTb0%32)*32).

[0369] The step width value is derived from other parts of the variable step_width_level2.

[0370] The variable IdxPlanes specifies which plane the transformation coefficients belong to.

[0371] The output of this process is an array of (nTbS)x(nTbS) of L-2 residuals resL2Residuals with elements resL2Residuals[x][y].

[0372] The sample position (xTbP, yTbP) of the top-left sample of the current transform block relative to the top-left sample of the current image is exported as follows:

[0373] (xTbP,yTbP) = (IdxPlanes == 0) ? (xTb0, yTb0) : (xTb0 >> ShiftWidthC,yTb0 >> ShiftHeightC)

[0374] P can be associated with either the luminance or chrominance plane, depending on which plane the transformation coefficients belong to. ShiftWidthC and ShiftHeightC are specified elsewhere.

[0375] If no_enhancement_bit_flag is set to 0, the following ordered steps apply:

[0376] If the variable temporal_enabled_flag equals 1 and the temporal_refresh_bit_flag equals 0, then the time prediction process specified elsewhere is called with the brightness position (xTbY, yTbY), the transformation size is set to nTbS, the variables TransformTempSig and TileTempSig are used as inputs, and the output is an array tempPredL2Residuals of size (nTbS) x (nTbS).

[0377] If the variables temporal_enabled_flag and temporal_refresh_bit_flag are both equal to 1, then the array tempPredL2Residuals of size (nTbS) x (nTbS) is set to contain only zeros.

[0378] If the variables `temporal_enabled_flag` equal 1, `temporal_refresh_bit_flag` equal 0, `temporal_tile_intra_signalling_enabled_flag` equal 1 (sub-clause 0), and `TransformTempSig` equal 0, then the variable `stepWidth` is modified to `Floor(stepWidth*(1−(Clip3(0, 0.5,(temporal_step_width_modifier / 255))))`. The dequantization process, as specified elsewhere, is called with a transform size set to nTbS, an array `TransformCoeffQ` of size (nTbS) x (nTbS), and the variable `stepWidth` as input, and the output is an array `dequantCoeff` of size (nTbS) x (nTbS).

[0379] The transformation process is called as specified elsewhere, with the brightness position (xTbY, yTbY), the transformation size set to be equal to nTbS, and the array dequantCoeff of size (nTbS)x(nTbS) as input, and the output is the array resL2Residuals of size (nTbS)x(nTbS).

[0380] If the variable temporal_enabled_flag is equal to 1, the arrays of tempPredL2Residuals of size (nTbS)x(nTbS) are added to the array resL2Residuals of size (nTbS)x(nTbS), and the array resL2Residuals is stored in temporalBuffer at the luminance position (xTbY, yTbY).

[0381] If no_enhancement_bit_flag is set to 1, the following ordered steps apply:

[0382] If the variables temporal_enabled_flag equals 1, temporal_refresh_bit_flag equals 0, and temporal_signalling_present_flag equals 1, then the transformation size is set to nTbS with the brightness position (xTbY, yTbY), the variables TransformTempSig and TileTempSig are used as input to call the time prediction process as specified elsewhere, and the output is an array tempPredL2Residuals of size (nTbS)x(nTbS).

[0383] If the variables temporal_enabled_flag, temporal_refresh_bit_flag, and temporal_signalling_present_flag are all equal to 1, then the time prediction process specified elsewhere is called with the brightness position (xTbY, yTbY), the transformation size set to nTbS, the variables TransformTempSig and TileTempSig set to 0 as inputs, and the output is an array tempPredL2Residuals of size (nTbS) x (nTbS).

[0384] If the variables temporal_enabled_flag and temporal_refresh_bit_flag are both equal to 1, then the array tempPredL2Residuals of size (nTbS) x (nTbS) is set to contain only zeros.

[0385] If the variable temporal_enabled_flag is equal to 1, then the array of tempPredL2Residuals of size (nTbS)x(nTbS) is stored in the array resL2Residuals of size (nTbS)x(nTbS), and the array resL2Residuals is stored in temporalBuffer at the brightness position (xTbY, yTbY).

[0386] Otherwise, the array resL2Residuals of size (nTbS)x(nTbS) is set to contain only zeros.

[0387] The image reconstruction process for each plane specified in the sub-clause is incorrect! Reference source not found. It is invoked with the transform block position (xTb0, yTb0), transform block size nTbS, variable IdxPlanes, (nTbS)x(nTbS) array resL2Residuals, and (xTbY)x(yTbY)recL2ModifiedUpsampledSamples as input.

[0388] Decoding process used for dequantization

[0389] Each group of transform coefficients passed to this process belongs to a specific plane and enhancement sublayer. It has been scaled with dead-zone using a uniform quantizer. The quantizer can use a non-centered dequantization offset.

[0390] Scaling process for transform coefficients

[0391] The input for this process is:

[0392] The variable nTbS specifies the size of the current transform block (nTbS=2 if transform_type is zero, and nTbS=4 if transform_type is 1).

[0393] TransformCoeffQ is an array containing entropy-decoded quantized transform coefficients of size (nTbS) x (nTbS).

[0394] The variable stepWidth specifies the step width value parameter.

[0395] The variable levelIdx specifies the index of the enhanced sublayer (levelIdx=1 for enhanced sublayer 1 and levelIdx=2 for enhanced sublayer 2).

[0396] The variables dQuantOffset and dequant_offset are specified as the dequantization offset.

[0397] If quant_matrix_mode is not 0, then the array QmCoeff0 of size 1 x nTbS2 is equal to the array variable qm_coefficient_0. In addition, if quant_matrix_mode is equal to 4, then the array QmCoeff1 of size 1 x nTbS2 is equal to the array qm_coefficient_1.

[0398] If nTbS==2, then there is an array QuantScalerDDBuffer containing the size (3*nTbS)x(nTbS) of the scaling parameter array used in the previous image.

[0399] If nTbS==4, then there is an array QuantScalerDDSBuffer containing the size (3*nTbS)x(nTbS) of the scaling parameter array used in the previous image.

[0400] The output of this process is an array d of (nTbS)x(nTbS) with elements d[x][y] and dequantized transform coefficients of the updated array QuantMatrixBuffer.

[0401] To derive the scaled transformation coefficients d[x][y], where x = 0...nTbS−1, y = 0...nTbS−1, and given the matrix qm[x][y] as specified in subclause 8.6.2, use the following formula:

[0402] d[x][y] = (TransformCoeffQ[x][y] * ((qm[x + (levelIdxSwap * nTbS)][y]+ stepWidthModifier[x][y]) + appliedOffset [x][y]) (1)

[0403] Derivation of dequantization offset and step width modifier

[0404] The variables appliedOffset[x][y] and stepWidthModifier[x][y] are exported as follows:

[0405] if (dequant_offset_signalled_flag == 0) {

[0406] stepWidthModifier [x][y] = ((((Floor(−Cconst * Ln (qm[x +(levelIdxSwap * nTbS)][y]))) + Dconst) *

[0407] (qm[x + (levelIdxSwap * nTbS)][y]2))) / 32768) >> 16

[0408] if (TransformCoeffQ[x][y] < 0)

[0409] appliedOffset [x][y] = (−1 * (−deadZoneWidthOffset [x][y]))

[0410] else if (TransformCoeffQ [x][y] > 0)

[0411] appliedOffset [x][y] = −deadZoneWidthOffset [x][y]

[0412] else

[0413] appliedOffset [x][y] = 0

[0414] } else if (dequant_offset_signalled_flag == 1) && (dequant_offset_mode_flag ==1) {

[0415] stepWidthModifier [x][y] = 0

[0416] if (TransformCoeffQ[x][y] < 0)

[0417] appliedOffset = (−1 * (dQuantOffsetActual [x][y] −deadZoneWidthOffset [x][y]))

[0418] else if (TransformCoeffQ [x][y] > 0)

[0419] appliedOffset [x][y] = dQuantOffsetActual [x][y] − deadZoneWidthOffset [x][y]

[0420] else

[0421] appliedOffset [x][y] = 0

[0422] } else if ((dequant_offset_signalled_flag == 1) && (dequant_offset_mode_flag == 0)) {

[0423] stepWidthModifier [x][y] = (Floor((dQuantOffsetActual [x][y]) * (qm[x + (levelIdxSwap * nTbS)][y]))

[0424] / 32768)

[0425] if (TransformCoeffQ[x][y] < 0)

[0426] appliedOffset = (−1 * (−deadZoneWidthOffset [x][y]))

[0427] else if (TransformCoeffQ [x][y] > 0)

[0428] appliedOffset [x][y] = −deadZoneWidthOffset [x][y]

[0429] else

[0430] appliedOffset [x][y] = 0

[0431] }

[0432] Where, if stepWidth > 16, then deadZoneWidthOffset is derived as follows:

[0433] deadZoneWidthOffset [x][y] = (((1 << 16) − ((Aconst * (qm[x +(levelIdxSwap * nTbs)][y] + stepWidthModifier [x][y])) + Bconst) >> 1) * (qm[x + (levelIdxSwap * nTbs)][y] + stepWidthModifier [x][y])) >> 16

[0434] If stepWidth <= 16, then the deadZoneWidthOffset is exported as follows:

[0435] deadZoneWidthOffset [x][y] = stepWidth >> 1

[0436] in:

[0437] Aconst = 39

[0438] Bconst = 126484

[0439] Cconst = 5242

[0440] Dconst = 99614

[0441] The calculation of dQuantOffsetActual[x][y] is as follows:

[0442] if (dequant_offset == 0)

[0443] dQuantOffsetActual[x][y] = dQuantOffset

[0444] else {

[0445] if (dequant_offset_mode_flag == 1)

[0446] dQuantOffsetActual [x][y] = ((Floor(−Cconst * Ln(qm[x + (levelIdxSwap* nTbs)][y]) +

[0447] (dQuantOffset << 9) + Floor(Cconst * Ln(StepWidth)))) * (qm[x +(levelIdxSwap * nTbs)][y])) >> 16

[0448] else if (dequant_offset_mode_flag == 0)

[0449] dQuantOffsetActual [x][y] = ((Floor(−Cconst * Ln(qm[x + (levelIdxSwap* nTbs)][y]) +

[0450] (dQuantOffset << 11) + Floor(Cconst * Ln(StepWidth)))) * (qm[x +(levelIdxSwap * nTbs)][y]))

[0451] >>16

[0452] }

[0453] The levelIdxSwap export is as follows:

[0454] if (levelIdx == 2)

[0455] levelIdxSwap = 0

[0456] else

[0457] levelIdxSwap = 1

[0458] Derivation of the quantization matrix

[0459] The quantization matrix qm[x][y] contains the actual quantization step width to be used for decoding each coefficient group.

[0460] if (levelIdx == 2) {

[0461] if (scaling_mode_level2 == 1) {

[0462] for (x = 0; x < nTbS; x++) {

[0463] for (y = 0; y < nTbs; y++)

[0464] qm[x][y] = qm_p[x][y]

[0465] }

[0466] } else {

[0467] for (x = 0; x < nTbS; x++) {

[0468] for (y = 0; y < nTbS; y++)

[0469] qm[x][y] = qm_p[x + nTbS][y]

[0470] }

[0471] }

[0472] } else {

[0473] for (x = 0; x < nTbS; x++) {

[0474] for (y = 0; y < nTbs; y++)

[0475] qm[x][y] = qm_p[x + (2 * nTbS)][y]

[0476] }

[0477] }

[0478] where qm_p[x][y] is calculated as follows:

[0479] if (nTbs == 2) {

[0480] for (x = 0; x < 6; x++) {

[0481] for (y = 0; y < nTbs; y++)

[0482] qm_p[x][y] = (Clip3(0, (3 << 16), [(QuantScalerDDBuffer[x][y] * stepWidth) + (1 << 16)]) *

[0483] stepWidth) >> 16

[0484] }

[0485] } else {

[0486] for (y = 0; y < 12; y++) {

[0487] for (x = 0; x < nTbs; x++)

[0488] qm_p[x][y] = (Clip3 (0, (3 << 16),[(QuantScalerDDSBuffer [x][y] *stepWidth) + (1 << 16)]) *

[0489] stepWidth) >> 16

[0490] }

[0491] }

[0492] Furthermore, QuantScalerDDBuffer[x][y] is exported elsewhere, and QuantScalerDDSBuffer[x][y] is exported elsewhere.

[0493] Derivation of scaling parameters for 2×2 transformation

[0494] If the variable nTbS equals 2, the default scaling parameters are as follows:

[0495] default_scaling_dd[x][y] =

[0496] {

[0497] { 0, 2}

[0498] { 0, 0}

[0499] { 32, 3 )

[0500] { 0, 32}

[0501] { 0, 3}

[0502] { 0, 32}

[0503] }

[0504] As a first step, the array QuantScalerDDBuffer[x][y] is initialized as follows:

[0505] If the current image is an IDR image, then QuantScalerDDBuffer[x][y] is initialized to be equal to default_scaling_dd[x][y]. If the current image is not an IDR image, then the QuantScalerDDBuffer[x][y] matrix remains unchanged.

[0506] After initialization, based on the value of quant_matrix_mode, the array QuantScalerDDBuffer[x][y] is processed as follows:

[0507] If quant_matrix_mode equals 0 and the current image is not an IDR image, then QuantScalerDDBuffer[x][y] remains unchanged.

[0508] If quant_matrix_mode equals 1, then QuantScalerDDBuffer[x][y] equals default_scaling_dd[x][y].

[0509] If quant_matrix_mode equals 2, then QuantScalerDDBuffer[x][y] is modified as follows:

[0510] for (MIdx = 0; MIdx < 3; MIdx++)

[0511] for (x = 0; x < 2; x++)

[0512] for (y = 0; y < 2; y++)

[0513] QuantScalerDDBuffer [x + (MIdx * 2)][y] = QmCoeff0[(x * 2) + y]

[0514] If quant_matrix_mode equals 3, then QuantScalerDDBuffer[x][y] is modified as follows:

[0515] for (MIdx = 0; MIdx < 2; MIdx++)

[0516] for (x = 0; x < 2; x++)

[0517] for (y = 0; y < 2; y++)

[0518] QuantScalerDDBuffer [x + (MIdx * 2)][y] = QmCoeff0 [(x * 2) + y]

[0519] If quant_matrix_mode equals 4, then QuantScalerDDBuffer[x][y] is modified as follows:

[0520] for (x = 0; x < 2; x++)

[0521] for (y = 0; y < 2; y++)

[0522] QuantScalerDDBuffer [x + 4][y] = QmCoeff1 [(x * 2) + y]

[0523] If quant_matrix_mode equals 5, then QuantScalerDDBuffer is modified as follows:

[0524] for (MIdx = 0; MIdx < 2; MIdx ++)

[0525] for (x = 0; x < 2; x++)

[0526] for (y = 0; y < 2; y++)

[0527] QuantScalerDDBuffer [x + (MIdx * 2)][y] = QmCoeff0[(x * 2) + y]

[0528] for (x = 4, x < 6; x++)

[0529] for (y = 0; y < 2; y++)

[0530] QuantScalerDDBuffer [x][y] = QmCoeff1[(x * 2) + y]

[0531] Derivation of scaling parameters for 4×4 transformation

[0532] If the variable nTbS equals 4, the default scaling parameters are as follows:

[0533] default_scaling_dds[x][y] =

[0534] {

[0535] { 13, 26, 19, 32}

[0536] { 52, 1, 78, 9}

[0537] { 13, 26, 19, 32}

[0538] { 150, 91, 91, 19}

[0539] { 13, 26, 19, 32}

[0540] { 52, 1, 78, 9}

[0541] { 26, 72, 0, 3}

[0542] { 150, 91, 91, 19}

[0543] { 0, 0, 0, 2}

[0544] { 52, 1, 78, 9}

[0545] { 26, 72, 0, 3}

[0546] { 150, 91, 91, 19}

[0547] }

[0548] As a first step, the array QuantScalerDDSBuffer[][] is initialized as follows:

[0549] If the current image is an IDR image, then QuantScalerDDSBuffer[x][y] is initialized to be equal to default_scaling_dds[x][y]. If the current image is not an IDR image, then the QuantScalerDDSBuffer[x][y] matrix remains unchanged.

[0550] After initialization, based on the value of quant_matrix_mode, the array QuantScalerDDSBuffer[x][y] is processed as follows:

[0551] If quant_matrix_mode equals 0 and the current image is not an IDR image, then QuantScalerDDSBuffer remains unchanged.

[0552] If quant_matrix_mode equals 1, then QuantScalerDDSBuffer equals default_scaling_dds[x][y].

[0553] If quant_matrix_mode equals 2, then QuantScalerDDSBuffer is modified as follows:

[0554] for (MIdx = 0; MIdx < 3; MIdx++)

[0555] for (x = 0; x < 4; x++)

[0556] for (y = 0; y < 4; y++)

[0557] QuantScalerDDSBuffer [x + (MIdx * 4)][y] = QmCoeff0[(x * 4) + y]

[0558] If quant_matrix_mode equals 3, then QuantScalerDDSBuffer is modified as follows:

[0559] for (MIdx = 0; MIdx < 2; MIdx++)

[0560] for (x = 0; x < 4; x++)

[0561] for (y = 0; y < 4; y++)

[0562] QuantScalerDDSBuffer [x + (MIdx * 4)][y] = QmCoeff0[(x * 4) + y]

[0563] If quant_matrix_mode equals 4, then QuantScalerDDSBuffer is modified as follows:

[0564] for (x = 0; x < 4; x++)

[0565] for (y = 0; y < 4; y++)

[0566] QuantScalerDDSBuffer [x + 8][y] = QmCoeff1[(x * 4) + y]

[0567] If quant_matrix_mode equals 5, then QuantScalerDDSBuffer is modified as follows:

[0568] for (MIdx = 0; MIdx < 2; MIdx++)

[0569] for (x = 0; x < 4; x++)

[0570] for (y = 0; y < 4; y++)

[0571] QuantScalerDDSBuffer [x + (MIdx * 4)][y] = QmCoeff0[(x * 4) + y]

[0572] for (x = 8, x < 12; x++)

[0573] for (y = 0; y < 4; y++)

[0574] QuantScalerDDSBuffer [x][y] = qm_coefficient_1[(x * 4) + y]

Claims

1. A method for encoding an input signal into a plurality of coded streams, wherein the coded streams can be combined to reconstruct the input signal, the method comprising: Receive input signals; The input signal is downsampled to create an downsampled signal; The instruction is to use a base encoder to encode the downsampled signal to create a base encoded stream; Instructs the use of a basic decoder to decode the underlying encoded stream to generate the reconstructed signal; The reconstructed signal is compared with the input signal to create a residual set; and... The residual set is encoded to create an encoded stream, comprising: The transformation is applied to the residual set to create a set of transformed coefficients; The quantization operation is applied to the set of transformed coefficients to create a set of quantized coefficients; as well as The encoding operation is applied to the quantized coefficients. The quantization operation is performed based on time information associated with the set of transformed coefficients.

2. The method according to claim 1, wherein, The method further includes determining the temporal information associated with the set of coefficients by deriving the correlation between sets of coefficients co-located at different samples.

3. The method according to claim 1, wherein, The time information includes whether the set or subset of coefficients is one of the following: static, quasi-static, and dynamic.

4. The method according to claim 3, wherein, A given set or subset of coefficients in a layer is static if the difference between the set or subset of corresponding co-located coefficients in previous and / or subsequent samples has an estimated information entropy that is substantially zero.

5. The method according to claim 1, wherein, If the difference between the set or subset of coefficients corresponding to the coefficients in the previous and / or subsequent samples has an estimated information entropy lower than the estimated information entropy of the given set or subset of coefficients in the layer, then the given set or subset of coefficients is quasi-static.

6. The method according to claim 1, wherein, The given set or subset of coefficients is dynamic if the difference between the corresponding co-located set or subset of coefficients in previous and / or subsequent samples has an estimated information entropy that is substantially the same as or higher than that of the estimated information entropy of the given set or subset of coefficients in the layer.

7. The method according to claim 1, wherein, The residual set is a first residual set, and the comparison step includes comparing the reconstructed signal with the downsampled signal to create the first residual set, such that the encoded stream is a first-level encoded stream, and the method further includes: Decode the first residual set to generate the decoded first residual set; The reconstructed signal is corrected using the decoded first residual set to generate a corrected reconstructed signal; The corrected reconstructed signal is upsampled to generate an upsampled reconstructed signal; The upsampled reconstructed signal is compared with the input signal to create a second set of residuals; and Encode the second residual set to create a second-level encoded stream, comprising: The transformation is applied to the second residual set to create a second set of coefficients; Apply the quantization operation to the second set of coefficients to create a second set of quantized coefficients; and The encoding operation is applied to the second set of quantized coefficients.

8. The method according to claim 1, wherein, The method further includes deriving one or more quantization parameters based on the time information.

9. The method according to claim 1, wherein, The method further includes determining a first quantization parameter of a first subset of the set of coefficients associated with the first time information.

10. The method according to claim 9, wherein, The method further includes determining a second quantization parameter for a second subset of the set of coefficients associated with the second time information.

11. The method according to claim 1, wherein, The quantization operation includes quantizing the coefficients using a linear quantizer, wherein the linear quantizer uses a variable-size dead zone.

12. The method according to claim 1, wherein, The quantization operation is performed based on one or more quantization parameters.

13. The method according to claim 12, wherein, The one or more quantization parameters are set to perform at least one of the following operations: controlling and providing the desired bit rate in one or more encoded streams.

14. The method according to claim 12, wherein, The desired bit rate is a common bit rate for all streams to generate a common encoded stream, or a different bit rate is provided for different encoded streams.

15. The method according to claim 12, wherein, Set one or more quantization parameters to provide the desired quality level or maximize the quality level within a set of predefined bit rate constraints.

16. The method according to claim 12, wherein, The method includes receiving the state of the buffers that receive the one or more encoded streams and the underlying encoded stream; And the quantization parameters are determined using the state.

17. The method according to claim 16, wherein, The buffer is used for at least one of the following operations: storing and combining the encoded base stream and the encoded enhancement stream, and is configured to receive input at a variable bit rate while reading the output at a constant rate, wherein the rate controller reads the state from the buffer to ensure that the buffer does not overflow or become empty, and that data is always available to be read at its output, and wherein the state of the buffer is used to generate the one or more quantization parameters.

18. The method according to claim 17, wherein, The one or more quantization parameters are controlled based on the amount of data in the buffer.

19. The method according to claim 18, wherein, The value of the quantization parameter is negatively correlated with the amount of data in the buffer.

20. The method according to claim 12, wherein, The quantization parameter is determined for at least one of the following: each frame, residual, and residual group.

21. The method according to claim 20, wherein, The quantization parameters of the frame are determined using a previously set of quantization parameters based on the target data size of the frame and the current data size of the frame.

22. The method according to claim 20, wherein, The quantization parameters are based on a previous set of quantization parameters.

23. The method according to claim 12, wherein, The method further includes defining a set of curves to map the normalized size to the one or more quantization parameters, wherein each curve includes one or more of a multiplier and an offset depending on the nature of the current frame.

24. The method according to claim 23, wherein, The multiplier is applied to a size normalization function, which is a function of the quantization parameter Q.

25. The method according to claim 24, wherein, Use Q t-1 The current size of the encoded frame t and Q t-1 Points within the space of the set of curves are defined, and the set of points closest to the curves is selected from the set of curves using the points.

26. The method of claim 25, wherein, The set of closest curves, together with the points, is used in an interpolation function to determine a new curve associated with the points, wherein a multiplier and offset of the determined new curve are determined, and the method further includes using the values ​​of the multiplier and offset of the determined new curve values, together with a received target size, to determine Q. t The value of .

27. The method according to claim 23, wherein, The set of curves is stored in accessible memory and updated based on the set of curves determined for a previous frame.

28. The method according to claim 1, wherein, The step width used in the quantization operation varies according to a step width parameter, wherein the step width parameter is based on the time information.

29. A method for decoding an encoded stream into a reconstructed output signal, the method comprising: Receive a first output signal decoded from a first basic encoded stream according to a first codec; Receive the coded stream at the layer level; Decode the hierarchical encoded stream to obtain the residual set; as well as, The residual set is combined with the first output signal to generate the reconstructed signal. Decoding the hierarchical encoded stream includes: Decode the set of quantized coefficients from the hierarchical encoded stream; Dequantize the set of quantized coefficients, wherein the dequantization of the set of quantized coefficients is based on time information associated with the hierarchical coded stream.

30. The method according to claim 29, wherein: The hierarchical encoded stream is a first-level encoded stream; The set of quantized coefficients is the first set of quantized coefficients; as well as, The residual set is a first residual set, and the method further includes: Receive the second-level encoded stream; Decode the second-level encoded stream to obtain the second residual set; and The second residual set is combined with the upsampling pattern of the reconstructed signal to generate a reconstruction of the original resolution input signal. Decoding the second-level encoded stream includes: Decode the second set of quantized coefficients from the second-level encoded stream; Dequantize the second set of quantized coefficients.

31. The method according to claim 30, wherein, The second set of quantized coefficients is dequantized based on time information associated with the second-level encoded stream.

32. The method according to claim 29, wherein, The method further includes deriving one or more quantization parameters from the encoded stream based on time information associated with the residual.

33. The method according to claim 32, wherein, Deriving the quantization parameters further includes determining from the encoded stream whether a default value should be used.

34. The method according to claim 32, wherein, Deriving the quantization parameters further includes determining from the encoded stream whether a signal value should be used instead of the default value.

35. The method according to claim 29, wherein, The method further includes determining a first quantization parameter for a first subset of data associated with the first time information.

36. The method according to claim 35, wherein, The method further includes determining a second quantization parameter for a second subset of data associated with the second time information.

37. The method according to claim 29, wherein, The method further includes determining time information associated with the residual.

38. The method according to claim 29, wherein, The dequantization is performed based on one or more quantization parameters, which are generated based on time information associated with the encoded stream.

39. The method according to claim 38, wherein, The method includes decoding a data layer by combining one or more sets of decoded data with corresponding data contained in a time buffer, and dequantizing decoded data combined with corresponding data in the time buffer with different quantization parameters relative to decoded data not combined with corresponding data in the time buffer.

40. The method according to claim 39, wherein, The decoder receives a single quantization parameter for the entire data layer, and the decoder generates the different quantization parameters at least in part based on the single quantization parameter for datasets that will be combined with corresponding data in the time buffer and datasets that will not be combined with corresponding data in the time buffer.

41. The method according to claim 40, wherein, The formula used by the decoder to derive the quantization parameters of the dataset that will not be combined with the corresponding data in the time buffer is at least partially based on default parameters, which override the default parameters if configuration parameters contained in the encoded stream are specified.

42. The method according to claim 29, wherein, The step width used in the dequantization operation varies according to a step width parameter, wherein the step width parameter is based on the time information.

43. An encoder for encoding an input signal, the encoder being configured to perform the method according to any one of claims 1 to 28.

44. A decoder for decoding an encoded stream into a reconstructed output signal, the decoder being configured to perform the method according to any one of claims 29 to 42.

45. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 42.

Images