Method and apparatus for quantization matrix prediction for video encoding and decoding

Abstract

The invention relates to a method and apparatus for quantization matrix prediction for video encoding and decoding. The method for decoding and / or encoding video information contained in a bitstream may involve a predictive quantization matrix associated with at least a portion of the video information and a syntax element indicating that at least one coefficient of the predictive quantization matrix is to be interpreted as a residual, such that the decoding and / or encoding of at least a portion of the video information is based on a combination of the predictive quantization matrix and the residual, where the syntax element is obtained from the bitstream during decoding or included in the bitstream during encoding.

Description

[0001] This application is a divisional application of the Chinese invention patent application with application date of August 6, 2020, application number 202080057371.7, and invention name “Quantization matrix prediction for video encoding and decoding”. Technical Field

[0002] The present disclosure relates to video compression, and more particularly, to quantization step size for video compression. Background Art

[0003] In order to achieve high compression efficiency, image and video coding schemes usually use prediction and transformation to exploit spatial and temporal redundancy in video content. Typically, intra-frame or inter-frame prediction is used to exploit intra-frame or inter-frame correlation, and then the difference between the original image and the predicted image, which is usually represented as a prediction error or prediction residual, is transformed, quantized, and entropy decoded. In order to reconstruct the video, the compressed data is decoded by the inverse process corresponding to prediction, transformation, quantization, and entropy decoding.

[0004] HEVC (High Efficiency Video Coding) is an example of a compression standard. HEVC consists of Joint Collaboration on Video Coding (JCT-VC) Development (see, for example, "ITU-T H.265 ITU Telecommunication Standards Section (10 / 2014), Series H: Audiovisual and Multimedia Systems, Infrastructure for Audiovisual Services - Coding for Mobile Video, High Efficiency Video Coding, ITU-T H.265 Recommendation"). Another example of a compression standard is the compression standard being developed by the Joint Video Experts Group (JVET) and is associated with the development work specified by Versatile Video Coding (VVC). VVC is intended to provide improvements over HEVC. Summary of the invention

[0005] Generally, at least one example of an embodiment may relate to a method comprising: obtaining information representing at least one coefficient of a quantization matrix and a syntax element from a bitstream containing encoded video information; determining based on the syntax element that the information representing the at least one coefficient is to be interpreted as a residual; and decoding at least a portion of the encoded video information based on a combination of a prediction of the quantization matrix and the residual.

[0006] Generally, at least one example of an embodiment may be directed to an apparatus comprising:

[0007] One or more processors configured to obtain information representing at least one coefficient of a quantization matrix and a syntax element from a bitstream containing encoded video information; determine based on the syntax element that the information representing the at least one coefficient is to be interpreted as a residual; and decode at least a portion of the encoded video information based on a combination of a prediction of the quantization matrix and the residual.

[0008] Generally, at least one example of an embodiment may relate to a method comprising: obtaining video information and information representing at least one coefficient of a predicted quantization matrix associated with at least a portion of the video information; determining that the at least one coefficient is to be interpreted as a residual; and encoding the at least a portion of the video information based on a combination of the predicted quantization matrix and the residual, and encoding a syntax element indicating that the at least one coefficient is to be interpreted as a residual.

[0009] Generally, at least one example of an embodiment may relate to a device comprising one or more processors configured to: obtain video information and information representing at least one coefficient of a predicted quantization matrix associated with at least a portion of the video information; determine that the at least one coefficient is to be interpreted as a residual; and encode the at least a portion of the video information based on a combination of the predicted quantization matrix and the residual, and encode a syntax element indicating that the at least one coefficient is to be interpreted as a residual.

[0010] In general, another example of an embodiment may involve a bitstream formatted to contain picture information, wherein the picture information is encoded by processing the picture information based on any one or more of the examples of embodiments of the method disclosed according to the present invention.

[0011] In general, one or more other examples of embodiments may also provide a computer-readable storage medium, for example, a non-volatile computer-readable storage medium, on which instructions for encoding or decoding picture information such as video data according to the method or apparatus described herein are stored. One or more embodiments may also provide a computer-readable storage medium on which a bitstream generated according to the method or apparatus described herein is stored. One or more embodiments may also provide methods and apparatus for transmitting or receiving a bitstream generated according to the method or apparatus described herein.

[0012] As explained below, various modifications and embodiments are contemplated which may provide improvements to video encoding and / or decoding systems including, but not limited to, one or more of increased compression efficiency and / or coding efficiency and / or processing efficiency and / or reduced complexity.

[0013] A simplified overview of the subject matter is provided above in order to provide a basic understanding of some aspects of the present disclosure. This overview is not an exhaustive review of the subject matter. It is not intended to identify the key / important elements of each embodiment or to describe the scope of the subject matter. Its sole purpose is to present some concepts of the subject matter in a simplified form as a preface to the more detailed description provided below. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The present disclosure may be better understood by considering the following detailed description in conjunction with the accompanying drawings, in which:

[0015] Figure 1 shows an example of transform / quantization coefficients being inferred to be zero for block sizes greater than 32 in VVC;

[0016] Figure 2 A comparison of the intra and inter default quantization matrices in H.264 is shown (top: inter (solid line) and intra (grid); bottom: inter (solid line), intra scaling (grid));

[0017] Figure 3 A comparison of intra and inter default quantization matrices in HEVC is shown (top: inter (solid line) and intra (grid); bottom: inter (solid line), intra scaling (grid));

[0018] Figure 4 An example of a quantization matrix (QM) decoding workflow is shown;

[0019] Figure 5 An example of an embodiment of a modified QM decoding workflow is shown;

[0020] Figure 6 Another example of an embodiment of a modified QM decoding workflow is shown;

[0021] Figure 7 Another example of an embodiment of a QM decoding workflow involving prediction (e.g., replication) and variable length residuals is shown;

[0022] Figure 8 Another example of an embodiment of a QM decoding workflow involving prediction (e.g., scaling) and variable length residuals is shown;

[0023] Fig. 9 Another example of an embodiment involving a QM decoding workflow that always uses prediction (e.g., scaling) and variable length residuals is shown;

[0024] Fig.10 An example of an embodiment of an encoder (e.g., a video encoder) suitable for implementing various aspects, features, and embodiments described herein is shown in block diagram form;

[0025] Fig.11 An example of an embodiment of a decoder (e.g., a video decoder) suitable for implementing various aspects, features, and embodiments described herein is shown in block diagram form;

[0026] Fig.12 An example of a system suitable for implementing the various aspects, features, and embodiments described herein is shown in block diagram form;

[0027] Fig.13 Another example of an embodiment of a decoder according to the present invention is shown; and

[0028] Fig.14 Another example of an embodiment of an encoder described according to the present invention is shown.

[0029] It should be understood that the drawings are for purposes of illustrating examples of various aspects and embodiments and are not necessarily the only possible configurations. In all drawings, the same reference numerals represent the same or similar features. DETAILED DESCRIPTION

[0030] The HEVC specification allows the use of quantization matrices in the dequantization process, where the coefficients of the frequency transform of the decoded block are scaled by the current quantization step size and further scaled by the quantization matrix (QM), as shown below:

[0031] d[x][y]=Clip3(coeffMin,coeffMax,((TransCoeffLevel[xTbY][yTbY][cIdx][x][y]

[0032] *m[x][y]*levelScale[qP%6]<<(qP / 6))+(1<<(bdShift-1)))>>bdShift)

[0033] in:

[0034] TransCoeffLevel[…] is the absolute value of the transform coefficient of the current block identified by the spatial coordinates xTbY, yTbY and its component index cIdx of the current block.

[0035] x and y are the horizontal / vertical frequency indices.

[0036] qP is the current quantization parameter.

[0037] Multiplying by levelScale[qP%6] and shifting left by (qP / 6) is equal to multiplying by the quantization step size qStep=(levelScale[qP%6]<<(qP / 6)).

[0038] ·m[…][…] is a two-dimensional quantization matrix.

[0039] bdShift is an additional scaling factor that takes into account the image sampling bit depth. The term (1<<(bdShift-1)) is used for rounding purposes to the nearest integer.

[0040] ·d[…] is the resulting absolute value of the dequantized transform coefficient.

[0041] The syntax used by HEVC to transmit the quantization matrix is ​​as follows:

[0042]

[0043] It can be noted that:

[0044] Specify a different matrix for each transform size (sizeId).

[0045] For a given transform size, six matrices are specified for intra / inter decoding and Y / Cb / Cr components.

[0046] The matrix can be one of the following:

[0047] o If scaling_list_pred_mode_flag is zero (reference matrixId is obtained as matrixId for scaling_list_pred_matrix_id_delta), the matrix is ​​copied from a previously transmitted matrix of the same size.

[0048] o Copy the matrix from the default value specified in the standard (if scaling_list_pred_mode_flag and scaling_list_pred_matrix_id_delta are both zero).

[0049] o The matrix is ​​fully specified in DPCM coding mode in top-right diagonal scan order using exp-Golomb entropy coding.

[0050] • For block sizes larger than 8x8, only 8x8 coefficients are transmitted for signaling the quantization matrix in order to save decoding bits. The coefficients are then interpolated using zero-holding (ie, repetition), except for the DC coefficient which is explicitly transmitted.

[0051] In VVC draft 5, the use of quantization matrices similar to HEVC has been adopted based on the contribution of JVET-N0847 (see O. Chubach, T. Toma, SCLim et al., "CE7 related: Support of quantization matrices for VVC", JVET-N0847, Geneva, CH, March 2019). The scaling_list_data syntax has been adapted to the VVC codec as shown below.

[0052]

[0053]

[0054] Compared to HEVC, VVC requires more quantization matrices due to the larger number of block sizes.

[0055] In VVC draft 5 (adopting JVET-N0847), as in HEVC, the QM is identified by two parameters matrixId and sizeId. This is illustrated in the following two tables.

[0056] Table 1: Block size identifiers (JVET-N0847)

[0057]

[0058] Table 2: QM Type Identifiers (JVET-N0847)

[0059]

[0060] Note: MODE_INTER QM is also used for IBC (Intra Block Copy) mode (MODE_IBC). The combination of the two identifiers is shown in the following table:

[0061] Table 3: (matrixId, sizeId) combinations (JVET-N0847)

[0062]

[0063] As in HEVC, for block sizes larger than 8×8, only the 8×8 coefficients + DC are transmitted. Zero-preserving interpolation is used to reconstruct the QM of the correct size. For example, for a 16×16 block, each coefficient is repeated twice in both directions, and then the DC coefficient is replaced by the transmitted coefficient.

[0064] For rectangular blocks, the reserved size (sizeId) selected for the QM is the larger size, i.e., the maximum of the width and height. For example, for a 4×16 block, a QM of 16×16 block size is selected. The reconstructed 16×16 matrix is ​​then vertically decimated by a factor of 4 to obtain the final 4×16 quantization matrix (i.e., three of the four rows are skipped).

[0065] For the following, we refer to the QM for a given family of block sizes (square or rectangular) as size-N, which is related to the sizeId and the square block size it is used for: for example, for block sizes 16×16 or 16×4, the QM is identified as size-16 (sizeId 4 in VVC draft 5). The sign of size-N is used to distinguish the exact block shape, as well as to distinguish the number of QM coefficients signaled (restricted to 8×8, as shown in Table 3).

[0066] Furthermore, in VVC draft 5, for the size-64 case, the QM coefficients in the lower right quadrant are not transmitted (they are inferred to be 0, hereinafter referred to as "zero-out"). This is achieved by the "x>=4&&y>=4" condition in the scaling_list_data syntax. This avoids transmitting QM coefficients that are never used by the transform / quantization process. In fact, in VVC, for transform block sizes greater than 32 in any dimension (64×N, N×64, where N<=64), any transform coefficient with an x / y frequency coordinate greater than or equal to 32 is not transmitted and is inferred to be zero. Therefore, no quantization matrix coefficients are required to quantize them. This is in Figure 1 , where the shaded areas correspond to transform coefficients that are inferred to be zero.

[0067] Generally, an aspect of at least one embodiment described herein may involve saving bits in the transmission of a customized QM while keeping the specification as simple as possible. Generally, another aspect of the disclosure may involve improving QM prediction so that simple adjustments can be made at low bit cost.

[0068] For example, two common adjustments applied to QM are:

[0069] - control of the dynamic range (the ratio between the lowest value (usually for low frequencies) and the highest value (usually for high frequencies)), which is easily achieved by globally scaling the matrix coefficients around a chosen neutral value; and

[0070] - General offsets for fine tuning bitrate, extending QP range, or balancing quality between different transform sizes.

[0071] An example is Figure 2 The "intra" and "inter" 8×8 default matrices for h264 shown in . Figure 2 Shown is a comparison of the default h264 matrix for intra and inter. Figure 2 The upper diagram in shows inter-frame (solid line) and intra-frame (grid). Figure 2 The bottom diagram in shows inter-frame (solid line) and intra-frame scaling (grid). Figure 2 In , the "inter" matrix is ​​just a scaled version of the "intra" matrix, where "intra scale" is (intra_QM-16)*11 / 16+17.

[0072] A similar trend is observed for the HEVC 8×8 default matrix, such as Figure 3 As shown in . Figure 3 Shown is a comparison of the default h264 matrix for intra and inter. Figure 3 The upper diagram in shows inter-frame (solid line) and intra-frame (grid). Figure 3 The bottom diagram in shows inter (solid line) and intra scaling (grid), where "intra scaling" is (intra_QM-16)*12 / 16+16).

[0073] This simple adjustment requires a complete re-encoding of the matrices in HEVC and VVC, which can be expensive (e.g., 240 bits to decode the scaled HEVC intra matrix shown above). Furthermore, in both HEVC and VVC, prediction (= copying) and explicit QM coefficients are mutually exclusive. That is, refinement of prediction is not possible.

[0074] In the context of VVC, modifications to QM syntax and prediction have been proposed. For example, in one proposal (JVET-O0223), QMs are transmitted in order from larger to smaller, and prediction from all previously transmitted QMs is allowed, including prediction from QMs intended for larger block sizes. Prediction in this context means copying, or means extraction if the reference is larger. This exploits the similarity between QMs of the same type (Y / Cb / Cr, intra / inter), but intended for different block sizes. The QM index matrixId included in the described proposal is a combination of a size identifier sizeId (shown in Table 4) and a type identifier matrixTypeId (shown in Table 5), which are combined using the following equation, which produces the unique identifier matrixId shown in Table 6.

[0075] matrixId=6 * sizeId+matrixTypeId (3-1)

[0076] Table 4: Size Identifiers (JVET-O0223)

[0077] brightness Chroma sizeId 64×64 32×32 0 32×32 16×16 1 16×16 8×8 2 8×8 4×4 3 4×4 2×2 4

[0078] Table 5: Matrix Type Identifiers (JVET-O0223)

[0079]

[0080] Table 6: Unified matrixId (JVET-O0223)

[0081]

[0082] And the syntax is modified, as indicated by the following highlighting:

[0083]

[0084] Figure 4 The QM decoding workflow is shown in FIG.

[0085] exist Figure 4 middle:

[0086] -The input is the decoded bit stream.

[0087] - The output is a ScalingMatrix array.

[0088] - "Decoding QM prediction patterns" : Get the prediction flag from the bitstream.

[0089] - "Predicted?": Whether the QM is inferred (predicted) or signaled in the bitstream depends on the aforementioned flag.

[0090] - "Decoding QM prediction data": Get prediction data from the bitstream when not signaled

[0091] When the QM needs to be inferred, for example, the QM index difference scaling_list_pred_matrix_id_delta. - " is the default of": Determine whether the QM is predicted from a default value (eg, whether scaling_list_pred_matrix_id_delta is zero) or from a previously decoded QM.

[0092] - "The reference QM is the default QM": Select the default QM as the reference QM. There can be several default QMs to choose from, e.g. depending on the parity of the matrixId.

[0093] - "Get Reference QM": The previously decoded QM is selected as the reference QM. The index of the reference QM is derived from the matrixId and the above index difference.

[0094] - "Copy or reduce reference QM":Predicts a QM from a reference QM. The prediction consists of simple copying if the reference QM is the correct size, or decimation if it is larger than expected. The result is stored in ScalingMatrix[matrixId].

[0095] - "Get the number of coefficients": Depends on matrixId determines the number of QM coefficients that will be decoded from the bitstream. For example: if matrixId is less than 20, then 64, if matrixId is between 20 and 25, then 16, otherwise 4.

[0096] - "Decoding QM coefficients": Decode the associated number of QM coefficients from the bitstream.

[0097] - "Diagonal Scan": Organize the decoded QM coefficients into a 2D matrix. The result is stored in ScalingMatrix[matrixId].

[0098] - "The Last QM": Loop or stop when all QMs are decoded from the bitstream.

[0099] - Details about DC values ​​are omitted for clarity.

[0100] The above proposal (JVET-00223) attempts to reduce the bit cost of decoding customized QMs in the context of VVC by allowing prediction from QMs with different sizeIds. However, the prediction is limited to copying (or decimation).

[0101] In the HEVC context, JCTVC-A114 (a response to the HEVC call for proposals) has proposed a QM inter-prediction technique with scalability. The following syntax and semantics are found in JCTVC-A114 Appendix A, with the most relevant parts shown in bold:

[0102]

[0103] in:

[0104] -qscalingmatrix_update_type_flag: Indicates the method of updating the scaling matrix. A value of 0 means adapting the new matrix from the existing matrix, while a value of 1 means retrieving the new matrix from the header. -delta_scale: Indicates the incremental scaling value. The delta_scale value is decoded in zig-zag scan order. If not present, the default value of delta_scale is equal to zero.

[0105] -matrix_scale : Parameter indicating scaling of all matrix values.

[0106] -matrix_slope: Indicates the parameter used to adjust the gradient of a non-flat matrix.

[0107] - num_delta_scale: indicates the number of delta_scale values ​​to be decoded when qscalingmatrix_update_type_flag is equal to 0.

[0108] - Depending on qscalingmatrix_update_type_flag, the value of the scaling matrix ScaleMatrix is ​​defined as follows:

[0109] - If qscalingmatrix_update_type_flag is equal to 0,

[0110] ScaleMatrix[i]=(ScaleMatrix[i]*(matrix_scale+16)+matrix_slope*

[0111] (ScaleMatrix[i]-16)+8)>>4)+delta_scale,

[0112] Where i=0…MaxCoeff.

[0113] - Otherwise (qscalingmattrix_update_type_flag is equal to 1)

[0114] ScaleMatrix[i] is derived as shown in Section 5.10.

[0115] Note - ScaleMatrix[i] is equal to 16 by default.

[0116] Although the method of JCTVC-A114 enables QM prediction using scale factors and variable length residuals, there are two redundant scale factors and no simple way to specify a global offset (all delta_scale values ​​must be coded since they are not DPCM coded).

[0117] In general, at least one aspect of the present disclosure may be directed to one or more of extending the prediction of QM by applying a scaling factor in addition to replication and downsampling, providing a simple way to specify a global offset, or allowing refined prediction with variable length residuals.

[0118] In general, at least one aspect of this disclosure may involve combining improved prediction with residuals to further refine the QM, potentially at a lower cost than full transcoding.

[0119] In general, at least one example of an embodiment according to the present disclosure may include QM prediction with offset in addition to replication / reduction.

[0120] In general, at least one example of an embodiment according to the present disclosure may include QM prediction plus a (variable length) residual.

[0121] In general, at least one example of an embodiment according to the present disclosure may include QM prediction with a scaling factor in addition to replication / downscale.

[0122] In general, at least one example of an embodiment according to the present disclosure may include a combination of a scaling factor and an offset or residual.

[0123] In general, at least one example of an embodiment disclosed herein may provide for reducing the number of bits required to transmit a QM and allowing a user-defined tradeoff between accuracy and bit cost while keeping the specification simple.

[0124] In general, at least one example of an embodiment disclosed herein may include one or more of the following features to reduce the number of bits required to transmit a QM and allow a user-defined tradeoff between accuracy and bit cost while keeping the specification simple:

[0125] Add residuals to QM forecasts

[0126] Add scaling factors to QM forecasts

[0127] Combining residuals and scaling

[0128] Always use prediction (remove prediction mode flag)

[0129] Examples of various embodiments are provided below, which combine one or more of the described features and aspects. The following examples are presented in the form of a grammatical and textual description based on VVC Draft 5 including proposal JVET-O0223. However, this context is merely an example selected for ease and clarity of description, and does not limit the scope of the present disclosure or the scope of application of the principles described herein. That is, the same principles can be applied to many different contexts, which is obvious to those skilled in the art, for example, to HEVC or VVC Draft 5, to QMs interpreted as QP offsets instead of scaling factors, to QM coefficients decoded with 7 bits instead of 8 bits, and so on.

[0130] Typically, at least one example of an embodiment includes a modification on top of a copy for QM prediction that includes adding a global offset. As an example, the global offset may be explicitly specified, as indicated in the following modification based on the syntax and specification text of VVC draft 5 including JVET-00223, where additions are highlighted with shading and deletions are underlined.

[0131]

[0132] An example of a modified QM decoding workflow corresponding to the aforementioned embodiment is shown in Figure 5 , where the changes from JVET-00223 are bold text and thick outlines.

[0133] Alternatively, the global offset may be specified as the first coefficient of the DPCM coded residual (described below).

[0134] In general, at least one example of an embodiment includes adding a residual to the prediction. For example, it may be desirable to transmit the residual on top of the QM prediction to refine the prediction or provide an optimized decoding scheme for QM (prediction plus residual rather than direct decoding). When QM is predicted, the syntax used to transmit QM coefficients when not predicted can be reused to transmit the residual.

[0135] In addition, the number of coefficients of the residual can be variable. When these coefficients are DPCM decoded as in HEVC, VVC and JVET-00223, and the lost coefficients are inferred to be zero, this means that the trailing coefficients of the residual are repetitions of the last transmitted residual coefficients. Then, transmitting a single residual coefficient is equivalent to applying a constant offset to the entire QM.

[0136] In general, at least one variation of the described embodiments includes adding residuals, which may further include using a flag when in prediction mode to indicate that coefficients may still be transmitted as in non-prediction mode but interpreted as residuals. When not in prediction mode, for uniform design, coefficients may still be interpreted as residuals over the default full 8 QM predictions (the value 8 is to mimic HEVC and VVC behavior, but is not limited to this value). An example of syntax and semantics based on JVET-00223 is shown below.

[0137]

[0138] scaling_list_pred_mode_flag[matrixId] equal to 0 specifies that the scaling matrix is ​​derived from the value of a reference scaling matrix. The reference scaling matrix is ​​specified by scaling_list_pred_matrix_id_delta[matrixId].

[0139] scaling_list_pred_mode_flag[matrixId] equal to 1 specifies that the values ​​of the scaling list are explicitly signaled. When scaling_list_pred_mode_flag[matrixId] is equal to 1, all values ​​of the (matrixSize)×(matrixSize) array predScalingMatrix[matrixId] and the value of the variable predScalingDC[matrixId] are inferred to be equal to 8. scaling_list_pred_matrix_id_delta[matrixId] specifies the reference scaling matrix used to derive the predicted scaling matrix as follows. The value of scaling_list_pred_matrix_id_delta[matrixId] should be in the range of 0 to matrixId, inclusive.

[0140] When scaling_list_pred_mode_flag[matrixId] is equal to zero:

[0141] -The variables refMatrixSize and the array refScalingMatrix are first derived as follows:

[0142] ° If scaling_list_pred_matrix_id_delta[matrixId] is equal to zero, the following applies to setting the default value:

[0143] ■ refMatrixSize is set equal to 8,

[0144] ■If matrixId is an even number,

[0145] refScalingMatrix = (6-1)

[0146] {

[0147] {16,16,16,16,16,16,16,16} / / default value placeholder in the frame

[0148] {16,16,16,16,16,16,16,16}

[0149] {16,16,16,16,16,16,16,16}

[0150] {16,16,16,16,16,16,16,16}

[0151] {16,16,16,16,16,16,16,16}

[0152] {16,16,16,16,16,16,16,16}

[0153] {16,16,16,16,16,16,16,16}

[0154] {16,16,16,16,16,16,16,16}

[0155] },

[0156] ■Otherwise

[0157] refScalingMatrix = (6-2)

[0158] {

[0159] {16,16,16,16,16,16,16,16} / / Inter-frame default value placeholder

[0160] {16,16,16,16,16,16,16,16}

[0161] {16,16,16,16,16,16,16,16}

[0162] {16,16,16,16,16,16,16,16}

[0163] {16,16,16,16,16,16,16,16}

[0164] {16,16,16,16,16,16,16,16}

[0165] {16,16,16,16,16,16,16,16}

[0166] {16,16,16,16,16,16,16,16}

[0167] },

[0168] ° Otherwise (if scaling_list_pred_matrix_id_delta[matrixId] is greater than zero), the following applies:

[0169] refMatrixId=matrixId - scaling_list_pred_matrix_id_delta[matrixId](6-3)

[0170] refMatrixSize=(refMatrixId < 20)? 8: (refMatrixId < 26)? 4:2)(6-4)

[0171] refScalingMatrix=ScalingMatrix[refMatrixId] (6-5)

[0172] -Then export the array predScalingMatrix[Matrix]Id as follows:

[0173] predScalingMatrix[Matrix Id][x][y]=refScalingMatrix[i][j] (6-6)

[0174] Wherein, matrixSize = (matrixId<20)?8:matrixId<26)?4:2)

[0175] x=0…matrixSize-1, y=0…matrixSize-1,

[0176] i=x<<(log2(refMatrixSize)-log2(matrixSize)), and

[0177] j=y<<(log2(refMatrixSize)-log2(matrixSize))

[0178] - When matrixId is lower than 14, if scaling_list_pred_matrix_id_delta[matrixId] is equal to zero,

[0179] Then set the variable predScalingDC[matrixId] to 16, otherwise set it to

[0180] ScalingDC[refMatrixId].

[0181] scaling_list_coef_present_flag equal to 1 specifies that the scaling list coefficients are transmitted and interpreted as residuals to be added to the prediction quantization matrix. scaling_list_coef_present_flag equal to 0 specifies that no residuals are added. When not present, scaling_list_coef_present_flag is inferred to be equal to 1.

[0182] Used to derive ScalingDC[matrixId], which specifies the first value of the scaling matrix when relevant, as described in clause xxx. The value of scaling_list_dc_coef[matrixId] should be in the range -128 to 127, inclusive. When not present, the value of scaling_list_dc_coef[matrixId] is inferred to be equal to 0.

[0183] When matrixId is less than 14, the variable ScalingDC[matrixId] is derived as follows:

[0184] ScalingDC[matrixId]=(predScalingDC[matrixId]+scaling_list_dc_coef[matrixId]+256)%256(6-7)

[0185]

[0186] scaling_list_delta_coef specifies the difference between scaling_list_matrixId][i] and the previous matrix coefficient scaling_list[matrixId][i-1], when scaling_list_coef_pref_flag When equal to 1, the value of scaling_list_delta_coef should be in the range of -128 to 127, including the end value. When scaling_list_coef_present_flag is equal to 0, all values ​​of the inference array ScalingList[matrixId] are equal to zero.

[0187] The (matrixSize)×(matrixSize) array ScalingMatrix[matrixId] is derived as follows:

[0188] ScalingMatrix[i][j]=(predScalingMatrix[i][j]+ScalingList[matxId][k]+256)% 256 (6-8)

[0189] Where k = 0…CoefNum-1,

[0190] i = diagScanOrder[log2(coefNum) / 2][log2(coefNum) / 2][k][0], and

[0191] j=diagScanOrder[log2(coefNum) / 2][log2(coefNum) / 2][k][1]

[0192] Additionally, in the "Scaling Matrix Export Process" section, the text changes as follows:

[0193] If matrixId is lower than 14, m[0][0] is further modified as follows:

[0194]

[0195] An example of a modified QM decoding workflow corresponding to the aforementioned embodiment is shown in Figure 6 , where the changes from JVET-00223 are bold text and thick outlines.

[0196] In general, at least one example of an embodiment may include using a variable length residual on top of the prediction. Variable length residuals may be useful because, for example:

[0197] The low frequency coefficients, which are transmitted first due to diagonal scanning, have the greatest impact,

[0198] Since the coefficients are DPCM-coded, if the missing delta coefficient is inferred to be zero, this results in the last residual coefficient being repeated. This means that the last transmitted residual is reused for the part of the QM where no residual is transmitted. In this way, transmitting a single residual coefficient is equivalent to applying an offset to the entire QM, without using specific additional syntax.

[0199] The number of residual coefficients can be indicated by, for example:

[0200] ·Explicit number of coefficients

[0201] o coefficients using Exponential Golomb coding or fixed-length encoding (length depends on matrixId), or

[0202] a flag indicating the presence of residuals, followed by the number of residual coefficients minus 1, which may be limited to powers of 2,

[0203] o or equivalently, 1 + log2 of the number of residual coefficients, where zero indicates no residual.

[0204] This can be decoded as fixed length (length potentially depends on matrixId) or Exponential Golomb decoding, or

[0205] - An index into a list containing the number of coefficients. It is recommended that this list be fixed by a standard (does not need to be transmitted), e.g. {0, 1, 2, 4, 8, 16, 32, 64}.

[0206] The number of residual coefficients need not count the potential additional DC residual coefficient.This additional DC residual coefficient may be transmitted once at least one regular residual coefficient is to be sent.

[0207] An example of syntax and semantics based on the immediately preceding example embodiment is provided below.

[0208]

[0209] scaling_list_pred_mode_flag[matrixId] equal to 0 specifies that the scaling matrix is ​​derived from the value of the reference scaling matrix. The reference scaling matrix is ​​specified by scaling_list_pred_matrix_id_delta[matrixId]. scaling_list_pred_mode_flag[matrixId] equal to 1 specifies that the value of the scaling list is explicitly signaled.

[0210] When scaling_list_pred_mode_flag[matrixId] is equal to 1, all values ​​of the matrixSize×matrixSize array predScalingMatrix[matrix] and the value of the variable predScalingDC[matrixId] are inferred to be equal to 8.

[0211] scaling_list_pred_matrix_id_delta[matrixId] specifies the reference scaling matrix used to derive the predicted scaling matrix as follows: The value of scaling_list_pred_matrix_id_delta[matrixId] should be in the range of 0 to matrixId, inclusive.

[0212] When scaling_list_pred_mode_flag[matrixId] is equal to zero:

[0213] -The variables refMatrixSize and the array refScalingMatrix are first derived as follows:

[0214] o If scaling_list_pred_matrix_id_delta[matrixId] is equal to zero, the following applies to setting the default value:

[0215] ■ refMatrixSize is set equal to 8,

[0216] ■If matrixId is an even number,

[0217] refScalingMatrix = (6-10)

[0218] {

[0219] {16,16,16,16,16,16,16,16} / / default value placeholder in the frame

[0220] {16,16,16,16,16,16,16,16}

[0221] {16,16,16,16,16,16,16,16}

[0222] {16,16,16,16,16,16,16,16}

[0223] {16,16,16,16,16,16,16,16}

[0224] {16,16,16,16,16,16,16,16}

[0225] {16,16,16,16,16,16,16,16}

[0226] {16,16,16,16,16,16,16,16}

[0227] ■Otherwise

[0228] refScalingMatrix = (6-11)

[0229] {

[0230] {16, 16, 16, 16, 16} / / Inter-frame default value placeholder

[0231] {16, 16, 16, 16, 16, 16, 16, 16}

[0232] {16, 16, 16, 16, 16, 16, 16, 16}

[0233] {16, 16, 16, 16, 16, 16, 16, 16}

[0234] {16, 16, 16, 16, 16, 16, 16, 16}

[0235] {16, 16, 16, 16, 16, 16, 16, 16}

[0236] {16, 16, 16, 16, 16, 16, 16, 16}

[0237] {16, 16, 16, 16, 16, 16, 16, 16}

[0238] },

[0239] o Otherwise (if scaling_list_pred_matrix_id_delta[matrixId] is greater than zero), the following applies:

[0240] refMatrixId=matrixId-scaling_list_pred_matrix_id_delta[matrixId] (6-12)

[0241] refMatrixSize=(refMatrixId<20)? 8: (refMatrixId<26)? 4:2) (6-13)

[0242] refScalingMatrix = ScalingMatrix[refMatrixId] (6-14) - then derive the array predScalingMatrix[matrixId] as follows:

[0243] predScalingMatrix[matrixId][x][y]=refScalingMatrix[i][j] (6-15)

[0244] in x=0…matrixSize–1, y=0…matrixSize–1,

[0245] i=x<<(log2(refMatrixSize)–log2(matrixSize)), and

[0246] j=y<<(log2(refMatrixSize)–log2(matrixSize))

[0247] - When matrixId<14, the variable predScalingDC[matrixId] is set equal to 16 if scaling_list_pred_matrix_id_delta[matrixId] is equal to zero, otherwise it is set equal to ScalingDC[refMatrixId].

[0248] coefNum specifies the number of residual coefficients to be transmitted. The value of coefNum should be in the range of 0 to maxCoefNum, inclusive.

[0249]

[0250] scaling_list_dc_coef[matrixId] is used to derive ScalingDC[matrixId], which specifies the first value of the scaling matrix when relevant, as described in clause xxx. The value of scaling_list_dc_coef[matrixId] should be in the range -128 to 127, inclusive. When not present, the value of scaling_list_dc_coef[matrixId] is inferred to be equal to 0.

[0251] When matrixId is less than 14, the variable ScalingDC[matrixId] is derived as follows:

[0252] ScalingDC[matrixId]=(predScalingDC[matrixId]+scaling_list_dc_coef[matrixId]+256)% 256 (6-16)

[0253] scaling_list_delta_coef specifies the difference between the current matrix coefficient ScalingList[matrixId][i] and the previous matrix coefficient ScalingList[matrixId][i-1], i between 0 and CoefNum-1, including the end value. For i between CoefNum and maxCoefNum-1 (including the end value), scaling_list_delta_coef is inferred to be equal to 0. The value of scaling_list_dc_coef shall be in the range -128 to 127, inclusive.

[0254]

[0255] The (matrixSize)×(matrixSize) array ScalingMatrix[matrixId] is derived as follows:

[0256] ScalingMatrix[i][j]=(predScalingMatrix[i][j]+ScalingList[matxId][k]+256)%256 (6-17)

[0257] Where k = 0…maxCoefNum-1,

[0258] i=diagScanOrder[log2(matrixSize)][log2(matrixSize)][k][0], and

[0259] j=diagScanOrder[log2(MatrixSize)][log2(MatrixSize)][k][1]

[0260] Figure 7 An example of a modified QM decoding workflow is shown. Figure 7 The example of the embodiment shown in involves: prediction = copy; variable length residual. Figure 7 In, yes Figure 6 The changes to the example embodiment shown in are bold text and thick outlines.

[0261] Typically, at least one example of an embodiment includes adding a scaling factor to the prediction. For example, the prediction of the QM can be enhanced by adding a scaling factor to the replication / decimation. When combined with the previous embodiment examples, this enables two common adjustments to the QM as explained above, namely scaling and offset, and also utilizes variable length residuals to achieve a user-defined trade-off between precision and bit cost. The scaling factor can be an integer, standardized to a number N (e.g., 16). The scaling formula can scale the reference QM around a selected neutral value (e.g., 16 for HEVC, VVC, or JVET-O0223; but if the QM is interpreted as a QP offset, it will be zero). An example formula with N=(1<<shift)=16 is shown below:

[0262] scaledQM=((refQM-16)×scale+rnd)>>shift)+16 (6-18)

[0263] in:

[0264] shift=4 and rnd=8, which means N=(1<<4)=16

[0265] refQM is the input QM to be scaled

[0266] scale is the scaling factor

[0267] ·scaledQM is the output scaled QM

[0268] As a result, scaledQM should be further clipped to the QM range (eg, 0...255 or 1...255).

[0269] The scaling factor scale may be transmitted as an offset of N (in the example above, the transmitted value would be: scale - (1 << shift)), such that a value of 0 represents a neutral value (no scaling) with minimal bit cost: 1 bit when decoding as exponential Golomb. Furthermore, the value may be clamped to a practical range: e.g. a number between -32 and 32 means a scaling between -1 and +3, or a number between -16 and 16 means a scaling between 0 and +2 (thus limiting the scaling to positive values).

[0270] An example of the syntax is shown below. Note that the scaling factor does not have to be combined with the offset or residual; it can be applied alone on JVET-O0223, VVC, or HEVC.

[0271]

[0272] scaling_list_pred_mode_flag[matrixId] equal to 0 specifies that the scaling matrix is ​​derived from the values ​​of the reference scaling matrix. The reference scaling matrix is ​​specified by scaling_list_pred_matrix_id_delta[matrixId]. scaling_list_pred_mode_flag[matrixId] equal to 1 specifies the values ​​of the scaling list that are explicitly signaled. When scaling_list_pred_mode_flag[matrixId] is equal to 1, all values ​​of the matrixSize×matrixSize array predScalingMatrix[matrix] and the value of the variable predScalingDC[matrixId] are inferred to be equal to 8. scaling_list_pred_matrix_id_delta[matrixId] specifies the reference scaling matrix used to derive the predicted scaling matrix as follows. The value of scaling_list_pred_matrix_id_delta[matrixId] should be in the range of 0 to matrixId, inclusive.

[0273] scaling_list_pred_scale_minus16[matrixId] plus 16 specifies the scaling factor to be applied to the reference scaling matrix used to derive the prediction scaling matrix as follows: The value of scaling_list_pred_scale_minus16[matrixId] should be in the range of -16 to +16, inclusive.

[0274] When scaling_list_pred_mode_flag[matrixId] is equal to zero:

[0275] -The variables refMatrixSize and the array refScalingMatrix are first derived as follows:

[0276] o If scaling_list_pred_matrix_id_delta[matrixId] is equal to zero, the following applies to setting the default value:

[0277] ■ refMatrixSize is set equal to 8,

[0278] ■If matrixId is an even number,

[0279] refScalingMatrix = (6-19)

[0280] {

[0281] {16, 16, 16, 16, 16} / / default value placeholder in the frame

[0282] {16, 16, 16, 16, 16, 16, 16, 16}

[0283] {16, 16, 16, 16, 16, 16, 16, 16}

[0284] {16, 16, 16, 16, 16, 16, 16, 16}

[0285] {16, 16, 16, 16, 16, 16, 16, 16}

[0286] {16, 16, 16, 16, 16, 16, 16, 16}

[0287] {16, 16, 16, 16, 16, 16, 16, 16}

[0288] {16, 16, 16, 16, 16, 16, 16, 16}

[0289] },

[0290] ■Otherwise

[0291] refScalingMatrix = (6-20)

[0292] {

[0293] {16, 16, 16, 16, 16} / / Inter-frame default value placeholder

[0294] {16, 16, 16, 16, 16, 16, 16, 16}

[0295] {16, 16, 16, 16, 16, 16, 16, 16}

[0296] {16, 16, 16, 16, 16, 16, 16, 16}

[0297] {16, 16, 16, 16, 16, 16, 16, 16}

[0298] {16, 16, 16, 16, 16, 16, 16, 16}

[0299] {16, 16, 16, 16, 16, 16, 16, 16}

[0300] {16, 16, 16, 16, 16, 16, 16, 16}

[0301] },

[0302] o Otherwise (if scaling_list_pred_matrix_id_delta[matrixId] is greater than zero), the following applies:

[0303] refMatrixId=matrixId-scaling_list_pred_matrix_id_delta[matrixId] (6-21)

[0304] refMatrixSize=(refMatrixId<20)? 8: (refMatrixId<26)? 4:2) (6-22)

[0305] refScalingMatrix=ScalingMatrix[refMatrixId] (6-23)

[0306] -Then export the array predScalingMatrix[matrixId] as follows:

[0307] predScalingMatrix[matrixId][x][y]=

[0308] Clip3(1, 255, (((refScalingMatrix[i][j]–16) * scale+8) >> 4)+16) (6-24)

[0309] Where x=0…matrixSize–1, y=0…matrixSize–1,

[0310] i=x<<(log2(refMatrixSize)-log2(matrixSize))

[0311] j=y<<(log2(refMatrixSize)-log2(matrixSize)), and

[0312] scale=scaling_list_pred_scale_minus16[matrixId]+16

[0313] When matrixId < 14, if scaling_list_pred_matrix_id_delta[matrixId] is equal to zero, the variable predScalingDC[matrixId] is set to 16, otherwise Export as follows

[0314] predScalingDC[matrixId]=

[0315] Clip3(1, 255, (((ScalingDC[refMatrixId]–16) * scale+8) >> 4)+16) (6-25)

[0316] where scale=scaling_list_pred_scale_minus16[matrixId]+16

[0317] coefNum specifies the number of residual coefficients to be transmitted. The value of coefNum should be in the range of 0 to maxCoefNum, inclusive.

[0318] scaling_list_dc_coef[matrixId] is used to derive ScalingDC[matrixId], which specifies the first value of the scaling matrix when relevant, as described in clause xxx. The value of scaling_list_dc_coef[matrixId] should be in the range -128 to 127, inclusive. When not present, the value of scaling_list_dc_coef[matrixId] is inferred to be equal to 0.

[0319] When matrixId is less than 14, the variable ScalingDC[matrixId] is derived as follows:

[0320] ScalingDC[matrixId]=(predScalingDC[matrixId]+scaling_list_dc_coef[matrixId]+256)%256 (6-26)

[0321] scaling_list_delta_coef specifies the current matrix coefficient ScalingList[matrixId][i] and the previous matrix coefficient

[0322] The difference between ScalingList[matrixId][i-1], i between 0 and CoefNum-1, inclusive. For i between CoefNum and maxCoefNum-1 (inclusive), scaling_list_delta_coef is inferred to be equal to 0. The value of scaling_list_delta_coef should be in the range of -128 to 127, inclusive.

[0323] The (matrixSize)×(matrixSize) array ScalingMatrix[matrixId] is derived as follows:

[0324] ScalingMatrix[i][j]=(predScalingMatrix[i][j]+ScalingList[matrixId][k]+256)% 256

[0325] (6-27)

[0326] where k = 0…maxCoefNum–1,

[0327] i=diagScanOrder[log2(matrixSize)][log2(matrixSize)][k][0], and

[0328] j=diagScanOrder[log2(matrixSize)][log2(matrixSize)][k][1]

[0329] Figure 8 An example of a modified QM decoding workflow is shown. Fig. 9 The example of the embodiment shown involves: prediction = scaling; variable length residual. Figure 8 In, yes Figure 7 The changes to the example embodiment shown in are bold text and thick outlines.

[0330] Typically, at least one example of an embodiment includes removing the prediction mode flag. For example, as long as a full residual can be transmitted, such as in the example of the above embodiment, since the QM can be fully standardized regardless of the scaling_list_pred_mode_flag flag, the flag can be removed, which simplifies the specification text and has a smaller impact on bit cost.

[0331] An example of the syntax and semantics of the current embodiment is provided below, based on changes to the previous embodiment.

[0332]

[0333]

[0334] scaling_list_pred_matrix_id_delta[matrixId] specifies the reference scaling matrix used to derive the predicted scaling matrix. The value of scaling_list_pred_matrix_id_delta[matrixId] should be in the range of 0 to matrixId, inclusive.

[0335] scaling_list_pred_scale_minus16[matrixId] plus 16 specifies the scaling factor to be applied to the reference scaling matrix used to derive the prediction scaling matrix as follows: The value of scaling_list_pred_scale_minus16[matrixId] should be in the range of -16 to +16, inclusive.

[0336]

[0337] -The variables refMatrixSize and the array refScalingMatrix are first derived as follows:

[0338] o If scaling_list_pred_matrix_id_delta[matrixId] is equal to zero, the following applies to setting the default value:

[0339] ■ refMatrixSize is set equal to 8,

[0340] ■If matrixId is an even number,

[0341] refScalingMatrix = (6-28)

[0342] {

[0343] {16, 16, 16, 16, 16} / / default value placeholder in the frame

[0344] {16, 16, 16, 16, 16, 16, 16, 16}

[0345] {16, 16, 16, 16, 16, 16, 16, 16}

[0346] {16, 16, 16, 16, 16, 16, 16, 16}

[0347] {16, 16, 16, 16, 16, 16, 16, 16}

[0348] {16, 16, 16, 16, 16, 16, 16, 16}

[0349] {16, 16, 16, 16, 16, 16, 16, 16}

[0350] {16, 16, 16, 16, 16, 16, 16, 16}

[0351] },

[0352] ■Otherwise

[0353] refScalingMatrix = (6-29)

[0354] {

[0355] {16, 16, 16, 16, 16} / / Inter-frame default value placeholder

[0356] {16, 16, 16, 16, 16, 16, 16, 16}

[0357] {16, 16, 16, 16, 16, 16, 16, 16}

[0358] {16, 16, 16, 16, 16, 16, 16, 16}

[0359] {16, 16, 16, 16, 16, 16, 16, 16}

[0360] {16, 16, 16, 16, 16, 16, 16, 16}

[0361] {16, 16, 16, 16, 16, 16, 16, 16}

[0362] {16, 16, 16, 16, 16, 16, 16, 16}

[0363] },

[0364] o Otherwise (if scaling_list_pred_matrix_id_delta[matrixId] is greater than zero), the following applies:

[0365] refMatrixId=matrixId-scaling_list_pred_matrix_id_delta[matrixId] (6-30)

[0366] refMatrixSize=(refMatrixId<20)? 8: (refMatrixId<26)? 4:2) (6-31)

[0367] refScalingMatrix=ScalingMatrix[refMatrixId] (6-32)

[0368] -Then export the array predScalingMatrix[matrixId] as follows:

[0369] predScalingMatrix[matrixId][x][y]=

[0370] Clip3(1, 255, (((refScalingMatrix[i][j]–16) * scale+8) >> 4)+16) (6-33)

[0371] Where x=0…matrixSize–1, y=0…matrixSize–1,

[0372] i=x<<(log2(refMatrixSize)-log2(matrixSize))

[0373] j=y<<(log2(refMatrixSize)-log2(matrixSize)), and

[0374] scale=scaling_list_pred_scale_minus16[matrixId]+16

[0375] - When matrixId < 14, if scaling_list_pred_matrix_id_delta[matrixId] is equal to zero, then the variable predScalingDC[matrixId] is set equal to 16, otherwise it is derived as follows:

[0376] predScalingDC[matrixId]=

[0377] Clip3(1, 255, (((ScalingDC[refMatrixId]–16) * scale+8) >> 4)+16) (6-34)

[0378] Where scale = scaling_list_pred_scale_minus16[matrixId] + 16coefNum specifies the number of residual coefficients transmitted. The value of coefNum should be in the range of 0 to maxCoefNum, including the end value.

[0379] scaling_list_dc_coef[matrixId] is used to derive ScalingDC[matrixId], which specifies the first value of the scaling matrix when relevant, as described in clause xxx. The value of scaling_list_dc_coef[matrixId] should be in the range -128 to 127, inclusive. When not present, the value of scaling_list_dc_coef[matrixId] is inferred to be equal to 0.

[0380] When matrixId is less than 14, the variable ScalingDC[matrixId] is derived as follows:

[0381] ScalingDC[matrixId] = (predScalingDC[matrixId] + scaling_list_dc_coef[matrixId] + 256) % 256 (6-35) scaling_list_delta_coef specifies the difference between the current matrix coefficient ScalingList[matrixId][i] and the previous matrix coefficient ScalingList[matrixId][i-1], i between 0 and CoefNum-1, including the end value. For i between CoefNum and maxCoefNum-1 (including the end value), scaling_list_delta_coef is inferred to be equal to 0. The value of scaling_list_delta_coef should be in the range of -128 to 127, including the end value.

[0382] The (matrixSize)×(matrixSize) array ScalingMatrix[matrixId] is derived as follows:

[0383] ScalingMatrix[i][j]=(predScalingMatrix[i][j]+ScalingList[matrixId][k]+256)%256(6-36)

[0384] where k = 0…maxCoefNum–1,

[0385] i=diagScanOrder[log2(matrixSize)][log2(matrixSize)][k][0], and

[0386] j=diagScanOrder[log2(matrixSize)][log2(matrixSize)][k][1]

[0387] Fig. 9 An example of a modified QM decoding workflow is shown. Fig. 9 The examples of embodiments shown involve: always using prediction; prediction = scaling; variable length residual. Fig. 9 In, yes Figure 8 Variations of the example embodiments shown in include Figure 8 Deletion of features.

[0388] At least a few examples of various other embodiments are described below, which are related to several contributions submitted for changes to the 16th JVET meeting that are considering changes to QMs, with the above-mentioned current VVC technology as reference. Most can be considered equivalent changes to the technology proposed in this disclosure, without changing its essence, but affecting the example syntax and semantics by affecting the number of QMs, QM identifier mapping, QM matrix size formula, QM selection and / or resizing process. Examples of embodiments that provide adaptation to changes are described in detail in the following subsections.

[0389] At least one example of an embodiment may involve removing some or all of the 2x2 chroma QMs. VVC recently removed the 2x2 chroma transform blocks in intra mode, thus making it useless to transmit 2x2 QM intra QMs, although the waste can be limited to 2x3=6 bits (3 bits are needed to signal the use of the default matrix with the above syntax example).

[0390] Removing the Luminance 2×2 QM will affect the number of QMs transmitted, with a simple syntax change:

[0391]

[0392] And the identifier mapping will also be affected (id 26 and 28 from Table 6 become invalid), which can be reflected by adjusting the matrixId selection formula in the "Quantization Matrix Derivation Procedure" from JVET-O0223 (or the update in JVET-P0110), for example:

[0393] matrixId=6*sizeId+matrixTypeId

[0394] with log2TuWidth=log2(blkWidth)+(cIdx>0)? log2(SubWidthC):0,

[0395] log2TuHeight=log2(blkHeight)+(cIdx>0)? log2(SubHeightC):0,

[0396] sizeId=6–max(log2TuWidth,log2TuHeight),and

[0397] matrixTypeId=(2*cIdx+(predMode==MODE_INTRA?0:1))

[0398] matrixId-=(matrixId>25)? (matrixId>27?:2:1):0

[0399] It would be less complex to remove all 2×2 QMs and disable QMs for 2×2 chroma inter blocks by reducing the number of QMs to 26, as shown in the following table:

[0400]

[0401] And instead of enlisting the "scaling matrix derivation process" in the "transform coefficient scaling process" of VVC, this case can be added to the exception list instead:

[0402] – The intermediate (nTbW) × (nTbH) scaling factor array m is derived as follows:

[0403] - m[x][y] is set equal to 16 if one or more of the following conditions are true:

[0404] -sps_scaling_list_enabled_flag is equal to 0.

[0405] -transform_skip_flag[xTbY][yTbY] is equal to 1.

[0406] - nTbW and nTbH are both equal to 2

[0407] Otherwise, m is the output of the derivation process of the scaling matrix as specified in clause 8.7.1, which is called with the prediction mode CuPredMode[xTbY][yTbY], the color component variables cIdx, the block width nTbW and the block height nTbH as input.

[0408] Or alternatively, when the block size is 2x2 or similar, all 16 values ​​are assigned to the m[][] array within the "scaling matrix derivation process".

[0409] At least one other embodiment involves a larger size for a luma QM of size-64. The matrix size formula may be changed to enable, for example, 16×16 coefficients for a luma QM of size-64; an example syntax is shown below:

[0410]

[0411] It will be clear to those skilled in the art that refMatrixSize in the example semantics of the above embodiments follows the same rules and will change accordingly.

[0412] At least one other embodiment involves adding joint CbCr QMs. Specific QMs may be needed for blocks coded with joint CbCr residuals. These can be added after the regular chroma one, resulting in the following new identification mapping table:

[0413] Table 7: MatrixId mapping with additional joint CbCr QM

[0414]

[0415] This affects the number of QMs, the matrix size formula, the DC coefficient existence condition (moves from <14 to <18), and the matrixId formula in the QM selection process as follows:

[0416]

[0417] matrixId=86*sizeId+matrixTypeId

[0418] with log2TuWidth=log2(blkWidth)+(cIdx>0)? log2(SubWidthC):0,

[0419] log2TuHeight=log2(blkHeight)+(cIdx>0)? log2(SubHeightC):0,

[0420] sizeId=6–max(log2TuWidth,log2TuHeight),and

[0421] matrixTypeId=(2*cIdx+(predMode==MODE_INTRA?0:1))

[0422] Other relevant locations in the semantics are affected (refMatrixSize etc.) with corresponding changes that will be clear to those skilled in the art.

[0423] At least one other embodiment involves adding QMs for LNFST. LNFST is a secondary transform used in VVC that changes the spatial frequency meaning of the final transform coefficients, thereby also affecting the meaning of the QM coefficients. Specific QMs may be required. Since the LNFST output is limited to 4×4 and has two alternating transform cores, these QMs can be added as pseudo-TU sizes inserted before or after the regular size-4 (keeping the QMs in descending size order), as shown in the table below. Since LNFST is only used in intra-frame (INTRA) mode, the INTRA / INTER line will be used to distinguish the core transforms.

[0424] Table 6: MatrixId mapping with additional QMs for LNFST

[0425]

[0426] As in the previous examples, the syntax and semantics should be modified accordingly to reflect the new matrixld mapping: number of QMs, matrix size, DC coefficient condition, etc.

[0427] The derivation of matrixId used to select the correct QM in the "Quantization Matrix Derivation Process" should also be updated by table description as described above or by changing the equation / pseudocode as in the following example:

[0428] if(lfnst_idx[xTbY][yTbY]!=0)

[0429] matrixId=24+2*cIdx+lfnst_idx[xTbY][yTbY]

[0430] else{

[0431] matrixId=[…]

[0432] if(matrixId>23)

[0433] matrixId+=6

[0434] }

[0435] At least one other embodiment involves processing QM with LNFST by forcing the use of 4×4 QM. This can be achieved by forcing a specific matrixId in the "quantization matrix derivation process" when LNFST is used for the current block, for example:

[0436] if(lfnst_idx[xTbY][yTbY]!=0)

[0437] matrixId=20+((cIdx==0)?10:2*cIdx)+lfnst_idx[xTbY][yTbY]

[0438] else

[0439] matrixId=[…]

[0440] This application describes various aspects, including tools, features, embodiments, models, methods, etc. For example, various aspects include but are not limited to:

[0441] - Save bits in the transmission of custom QMs while keeping the specification as simple as possible;

[0442] - Improved QM prediction, making simple adjustments possible at low bit cost;

[0443] - Provides a simple way to specify global offsets;

[0444] - one or more of the following: extending the prediction of the QM by applying a scaling factor in addition to replication and downsampling; providing a simple way to specify a global offset; and allowing refined prediction with variable length residuals;

[0445] - Combine the improved prediction with the residual to further refine the QM at potentially lower cost than full decoding;

[0446] - Provides QM forecast with offset in addition to copy / reduce;

[0447] - Provides QM predictions plus (variable length) residuals;

[0448] - Provides QM forecasts with scaling factors in addition to copy / reduce;

[0449] - providing a QM prediction comprising a combination of a scaling factor and an offset or residual;

[0450] - Provides a reduction in the number of bits required to transmit a QM and allows a user-defined trade-off between accuracy and bit cost while keeping the specification simple.

[0451] - Provides a reduction in the number of bits required to transmit a QM and allows a user-defined tradeoff between accuracy and bit cost while keeping the specification simple based on one or more of the following features:

[0452] oAdd the residuals to the QM forecast;

[0453] oAdd scaling factors to QM forecasts;

[0454] oMerge residuals and scale;

[0455] oAlways use prediction (remove prediction mode flag).

[0456] Many of these aspects are described as having specificity, and at least to illustrate individual characteristics, are often described in a manner that may sound limited. However, this is for the purpose of describing clearly, and does not limit the application or scope of those aspects. In fact, all different aspects can be combined and interchanged to provide additional aspects. In addition, these aspects can also be combined and interchanged with aspects described in earlier documents.

[0457] The aspects described and contemplated in this application can be implemented in many different forms. Fig.10 , Fig.11 and Fig.12 Some embodiments are provided, but other embodiments are contemplated and are not intended to be construed in detail. Fig.10 , Fig.11 and Fig.12 The discussion does not limit the breadth of implementation. At least one aspect generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, computer-readable storage media having stored thereon instructions for encoding or decoding video data according to any of the methods described, and / or computer-readable storage media having stored thereon a bitstream generated according to any of the methods described.

[0458] In this application, the terms "reconstruction" and "decoding" may be used interchangeably, the terms "pixel" and "sample" may be used interchangeably, and the terms "image", "picture" and "frame" may be used interchangeably.

[0459] Various methods are described herein, and each method includes one or more steps or actions for implementing the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined.

[0460] Various methods and other aspects described in this application can be used to modify modules, such as Fig.10 and Fig.11The quantization module and the inverse quantization module (130, 140, 240) of the video encoder 100 and the decoder 200 are shown. In addition, aspects of the present invention are not limited to VVC or HEVC, and can be applied to (for example) other standards and recommendations (whether previously existing or developed in the future), as well as extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise specified or technically excluded, the aspects described in this application can be used alone or in combination.

[0461] Various numerical values ​​are used in this application, such as the size of the maximum quantization matrix considered, the number of block sizes, etc. The specific values ​​are for example purposes, and the described aspects are not limited to these specific values.

[0462] Fig.10 A video encoder 100 is shown. Variations of the encoder 100 are contemplated, but for the sake of clarity, the encoder 100 is described below without describing all contemplated variations.

[0463] Before being encoded, the video sequence may undergo pre-encoding processing (101), for example, applying a color transform to the input color picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of the input picture components in order to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and appended to the bitstream.

[0464] In encoder 100, a picture is encoded by encoder elements as described below. The picture to be encoded is divided (102) and processed in units such as CUs. Each unit is encoded using, for example, intra or inter mode. When the unit is encoded in intra mode, it performs intra prediction (160). In inter mode, motion estimation (175) and motion compensation (170) are performed. The encoder decides (105) whether to encode the unit using one of intra mode or inter mode, and indicates the intra / inter decision by, for example, a prediction mode flag. The prediction residual is calculated, for example, by subtracting (110) the predicted block from the original image block.

[0465] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, along with motion vectors and other syntax elements, are entropy coded (145) to output a bitstream. The encoder may also skip the transform and apply quantization directly to the untransformed residual signal. The encoder may bypass both the transform and quantization, i.e., directly code the residual without applying the transform or quantization process.

[0466] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are inverse quantized (140) and inverse transformed (150) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (155) to reconstruct the image block. An in-loop filter (165) is applied to the reconstructed picture to perform, for example, deblocking / SAO (sample adaptive offset) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (180).

[0467] Fig.11 2 shows a block diagram of a video decoder 200. In the decoder 200, the bitstream is decoded by the decoder elements as described below. The video decoder 200 generally performs the same operations as described above. Fig.10 The encoding pass is described as the inverse of the decoding pass. Encoder 100 also typically performs video decoding as part of encoding the video data.

[0468] Specifically, the input to the decoder includes a video bitstream that may be generated by the video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other decoding information. Picture segmentation information indicates how the picture is segmented. Thus, the decoder can divide (235) the picture according to the decoded picture segmentation information. The transform coefficients are inverse quantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual is combined (255) with the prediction block to reconstruct the image block. The prediction block can be obtained (270) from intra-frame prediction (260) or motion compensated prediction (i.e., inter-frame prediction) (275). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0469] The decoded picture may further undergo post-decoding processing (285), such as inverse color conversion (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or performing an inverse remapping of the remapping process performed in the pre-encoding process (101). The post-decoding process may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0470] Fig.12A block diagram of an example of a system in which various aspects and embodiments are implemented is shown. System 1000 can be implemented as a device including various components described below, and is configured to perform one or more aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected household appliances and servers. The elements of system 1000 can be implemented individually or in combination in a single integrated circuit (IC), multiple ICs and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed on multiple ICs and / or discrete components. In various embodiments, system 1000 is coupled to one or more other systems or other electronic devices via, for example, a communication bus or by a dedicated input and / or output port communication. In various embodiments, system 1000 is configured to implement one or more aspects described in this document.

[0471] The system 1000 includes at least one processor 1010 configured to execute instructions loaded therein for implementing, for example, various aspects described in this document. The processor 1010 may include embedded memory, input-output interfaces, and various other circuits known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 1000 includes a storage device 1040, which may include a non-volatile memory and / or a volatile memory, including but not limited to an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), a programmable read-only memory (PROM), a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, a magnetic disk drive, and / or an optical disk drive. As a non-limiting example, the storage device 1040 may include an internal storage device, an attached storage device (including a removable and non-removable storage device), and / or a network-accessible storage device.

[0472] The system 1000 includes an encoder / decoder module 1030, which is configured to, for example, process data to provide encoded video or decoded video, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents a module (or modules) that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of the encoding and decoding modules. In addition, the encoder / decoder module 1030 may be implemented as a separate element of the system 1000 or may be incorporated into the processor 1010 as a combination of hardware and software as known to those skilled in the art.

[0473] Program code to be loaded onto the processor 1010 or the encoder / decoder 1030 to perform various aspects described in this document may be stored in the storage device 1040 and subsequently loaded onto the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of the various items during the execution of the processes described in this document. These stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0474] In some embodiments, memory within the processor 1010 and / or encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be memory 1020 and / or a storage device 1040, such as a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store, for example, an operating system for a television. In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as working memory for video coding and decoding operations such as MPEG-2 (MPEG refers to Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard developed by JVET, Joint Video Experts Team).

[0475] As shown in block 1130, input to the elements of system 1000 may be provided through various input devices. Such input devices include, but are not limited to: (i) an RF section that receives a radio frequency (RF) signal transmitted over the air, for example, by a broadcaster; (ii) a component (COMP) input terminal (or a set of COMP input terminals); (iii) a universal serial bus (USB) input terminal; and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Fig.10 Other examples not shown include composite video.

[0476] In various embodiments, the input device of block 1130 has associated corresponding input processing elements known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also referred to as selecting a signal, or band limiting a signal to a frequency band); (ii) down-converting the selected signal; (iii) again band-limiting to a narrower frequency band to select a signal frequency band that may be referred to as a channel in some embodiments; (iv) demodulating the down-converted and band-limited signal; (v) performing error correction; and (vi) demultiplexing to select a desired data packet stream. The RF section of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a near-baseband frequency) or baseband. In a set-top box embodiment, the RF part and its associated input processing element receive the RF signal transmitted by wired (for example, cable) medium, and filter to the desired frequency band again by filtering, down-conversion and perform frequency selection. Various embodiments rearrange the order of above-mentioned (and other) elements, remove some in these elements, and / or add other elements that perform similar or different functions. For example, adding element can include inserting element between existing element, for example, inserting amplifier and analog-to-digital converter. In various embodiments, the RF part includes antenna.

[0477] In addition, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 1000 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, may be implemented as needed, for example, in a separate input processing IC or processor 1010. Similarly, various aspects of USB or HDMI interface processing may be implemented as needed, either in a separate interface IC or in processor 1010. The demodulated, error corrected, and demultiplexed streams are provided to various processing elements, including, for example, processor 1010 and encoder / decoder 1030, which operate in conjunction with memory and storage elements to process the data streams as needed for presentation on output devices.

[0478] The various components of system 1000 may be provided in an integrated housing. Within the integrated housing, the various components may be interconnected and transfer data between them using a suitable connection arrangement 1140, such as an internal bus (including an inter-IC (I2C) bus), wiring, and printed circuit boards as known in the art.

[0479] The system 1000 includes a communication interface 1050 that enables communication with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data through the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or a network card, and the communication channel 1060 may be implemented, for example, within a wired and / or wireless medium.

[0480] In various embodiments, a wireless network, such as a Wi-Fi network, such as IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers), is used to stream data or otherwise provide it to the system 1000. The Wi-Fi signals of these embodiments are received by a communication channel 1060 and a communication interface 1050 suitable for Wi-Fi communication. The communication channel 1060 of these embodiments is usually connected to an access point or a router, which provides access to an external network including the Internet to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box to provide streaming data to the system 1000, which passes data through the HDMI connection of the input block 1130. Still other embodiments use the RF connection of the input block 1130 to provide streaming data to the system 1000. As described above, various embodiments provide data in a non-streaming manner. In addition, various embodiments use a wireless network other than Wi-Fi, such as a cellular network or a Bluetooth network.

[0481] System 1000 can provide output signals to various output devices, including display 1100, speaker 1110 and other peripheral devices 1120. Display 1100 of various embodiments includes one or more of, for example, a touch screen display, an organic light emitting diode (OLED) display, a curved display and / or a foldable display. Display 1100 can be used for a television, a tablet computer, a laptop computer, a cellular phone (mobile phone) or other devices. Display 1100 can also be integrated with other components (e.g., as in a smart phone), or applied separately (e.g., an external monitor for a laptop computer). In each example of each embodiment, other peripheral devices 1120 include one or more of an independent digital video disk (or digital versatile disk) (DVR, two items), a disk player, a stereo system and / or a lighting system. Various embodiments use one or more peripheral devices 1120, which provide functions based on the output of system 1000. For example, a disk player performs the function of playing the output of system 1000.

[0482] In various embodiments, control signals are transmitted between the system 1000 and the display 1100, speaker 1110, or other peripheral device 1120 using signaling such as AV.Link, consumer electronics control (CEC), or other communication protocols, which enables device-to-device control with or without user intervention. Output devices can be communicatively coupled to the system 1000 via dedicated connections through corresponding interfaces 1070, 1080, and 1090. Alternatively, the output device can be connected to the system 1000 via the communication interface 1050 using the communication channel 1060. For example, the display 1100 and the speaker 1110 can be integrated into a single unit in an electronic device (e.g., a television) along with other components of the system 1000. In various embodiments, for example, the display interface 1070 includes a display driver, such as a timing controller (T Con) chip.

[0483] For example, if the RF portion of input 1130 is part of a separate set-top box, the display 1100 and speaker 1110 may alternatively be separate from one or more of the other components. In various embodiments where the display 1100 and speaker 1110 are external components, the output signal may be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0484] These embodiments may be performed by computer software implemented by processor 1010, hardware, or a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. Memory 1020 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology, such as, as a non-limiting example, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. Processor 1010 may be of any type suitable for the technical environment and may include, as a non-limiting example, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0485] Fig.13 Another example of an embodiment is shown in FIG. Fig.13In , input information, such as a bitstream, includes coded video information and other information, such as information associated with the coded video information (e.g., one or more quantization matrices) and control information, such as one or more syntax elements. At 1310, information representing at least one coefficient of a quantization matrix and syntax elements are obtained from the input, such as an input bitstream. At 1320, it is determined based on the syntax elements that the information representing the at least one coefficient is to be interpreted as a residual. Next, at 1330, at least a portion of the coded video information is decoded based on a combination of a prediction of the quantization matrix and the residual. Fig.13 One or more examples of the features shown and described hereinbefore, for example, with respect to Figure 6 Show and describe.

[0486] Fig.14 Another example of an embodiment is shown. Fig.14 In the method of claim 14, at 1410, an input including video information is processed to obtain video information and information representing at least one coefficient of a predictive quantization matrix associated with at least a portion of the video information. At 1420, it is determined that at least one coefficient is to be interpreted as a residual. For example, the processing of the video information may indicate the use of the residual to provide advantageous processing of certain video information, such as providing improved compression efficiency when encoding the video information, or providing improved video quality when decoding the encoded video information. Then, at 1430, at least a portion of the video information is encoded based on a combination of the predictive quantization matrix and the residual. The encoding at 1430 also encodes a syntax element indicating that the at least one coefficient is to be interpreted as a residual.

[0487] The examples of various implementations and embodiments described herein relate to decoding. As used in the present application, "decoding" may include, for example, all or part of a process performed on a received coded sequence to produce a final output suitable for display. In various embodiments, such a process includes one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such a process also or alternatively includes a process performed by a decoder of the various implementations described in the present application.

[0488] As a further example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, and in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or generally to a broader decoding process will be clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0489] Various implementations involve encoding. In a manner similar to the discussion above about "decoding", "encoding" as used in this application can include, for example, all or part of a process performed on an input video sequence to produce an encoded bitstream. In various embodiments, such processes include one or more processes typically performed by an encoder, such as segmentation, differential encoding, transforms, quantization, and entropy encoding.

[0490] As a further example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in another embodiment, "encoding" refers to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or generally to a broader encoding process will become clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0491] Note that the syntax elements used herein, such as scaling_list_pred_mode_flag, scaling_list_pred_matrix_id_delta, scaling_list_dc_coef_minus8, are descriptive terms. Therefore, they do not exclude the use of other syntax element names.

[0492] When a figure is presented as a flow chart, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a figure is presented as a block diagram, it should be understood that it also provides a flow chart of the corresponding method / process.

[0493] The implementations and aspects described herein may be implemented in, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method), the implementation of the features discussed may also be implemented in other forms (e.g., an apparatus or program). For example, an apparatus may be implemented with appropriate hardware, software, and firmware. The method may be implemented in, for example, a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes a communication device, such as a computer, a cellular phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate information communication between end users.

[0494] References to "one embodiment" or "an embodiment" or "an implementation" or "implementations" and other variations thereof mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in an implementation" or "in an implementation" and any other variations thereof in various places throughout this application are not necessarily all referring to the same embodiment.

[0495] Additionally, the present application may refer to “determining” various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.

[0496] Furthermore, the present application may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0497] Additionally, the present application may involve "receiving" various information. Like "accessing," receiving is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, during operations such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information, "receiving" is generally involved in one way or another.

[0498] It should be understood that, for example, in the case of "A / B", "A and / or B", and "at least one of A and B", the use of any of the following " / ", "and / or", and "at least one of..." is intended to cover the selection of only the first listed option (A), or only the second listed option (B), or both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C", such wording is intended to include the selection of only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A and B and C). This can be extended to many of the listed items, as will be clear to one of ordinary skill in the art and related arts.

[0499] As will be appreciated by those skilled in the art, implementations may generate various signals formatted to carry information that may be stored or transmitted, for example. The information may include, for example, instructions for executing a method, or data generated by one of the described implementations. For example, a signal may be formatted to carry a bitstream of the described embodiments. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal may be transmitted via a variety of different wired or wireless links. The signal may be stored on a processor readable medium.

[0500] Various generalized and specialized embodiments are also supported and considered in the present disclosure.Examples of embodiments disclosed according to the present invention include but are not limited to the following.

[0501] Generally, at least one example of an embodiment may relate to a method comprising: obtaining information representing at least one coefficient of a quantization matrix and a syntax element from a bitstream containing encoded video information; determining based on the syntax element that the information representing the at least one coefficient is to be interpreted as a residual; and decoding at least a portion of the encoded video information based on a combination of a prediction of the quantization matrix and the residual.

[0502] Generally, at least one example of an embodiment may relate to a device comprising one or more processors configured to: obtain information representing at least one coefficient of a quantization matrix and syntax elements from a bitstream comprising encoded video information; determine based on the syntax elements that the information representing the at least one coefficient is to be interpreted as a residual; and decode at least a portion of the encoded video information based on a combination of a prediction of the quantization matrix and the residual.

[0503] Generally, at least one example of an embodiment may relate to a method comprising: obtaining video information and information representing at least one coefficient of a predicted quantization matrix associated with at least a portion of the video information; determining that the at least one coefficient is to be interpreted as a residual; and encoding the at least a portion of the video information based on a combination of the predicted quantization matrix and the residual, and encoding a syntax element indicating that the at least one coefficient is to be interpreted as a residual.

[0504] Generally, at least one example of an embodiment may involve a device comprising one or more processors configured to: obtain video information and information representing at least one coefficient of a predicted quantization matrix associated with at least a portion of the video information; determine that the at least one coefficient is to be interpreted as a residual; and encode the at least a portion of the video information based on a combination of the predicted quantization matrix and the residual, and encode a syntax element indicating that the at least one coefficient is to be interpreted as a residual.

[0505] In general, at least one example of an embodiment may be directed to a method or apparatus as described herein comprising a residual, wherein the residual comprises a variable length residual.

[0506] Generally, at least one example of an embodiment may be directed to a method or apparatus as described herein, wherein at least one coefficient comprises a plurality of predicted quantization matrix coefficients, and combining comprises adding the residual to the plurality of predicted quantization matrix coefficients.

[0507] Generally, at least one example of an embodiment may be directed to a method or apparatus as described herein, wherein combining further comprises applying a scaling factor.

[0508] Generally, at least one example of an embodiment may be directed to a method or apparatus as described herein, wherein combining comprises applying a scaling factor to a plurality of predicted quantization matrix coefficients and then adding a residual.

[0509] Generally, at least one example of an embodiment may be directed to a method or apparatus as described herein, wherein the combining further comprises applying an offset.

[0510] Generally, at least one example of an embodiment may relate to a method or apparatus as described herein, wherein the bitstream includes one or more syntax elements for transmitting at least one coefficient when the quantization matrix is ​​not predicted, and the one or more syntax elements can be used again to transmit the residual when the quantization matrix is ​​predicted.

[0511] In general, at least one example of an embodiment may involve a method or apparatus as described herein, which generates a bitstream comprising one or more syntax elements for transmitting at least one coefficient when a quantization matrix is ​​not predicted, which bitstream can be used again to transmit a residual when the quantization matrix is ​​predicted.

[0512] In general, one or more other examples of embodiments may also provide a computer-readable storage medium, for example, a non-volatile computer-readable storage medium, on which instructions for encoding or decoding picture information such as video data according to the method or apparatus described herein are stored. One or more embodiments may also provide a computer-readable storage medium on which a bitstream generated according to the method or apparatus described herein is stored. One or more embodiments may also provide methods and apparatus for transmitting or receiving a bitstream generated according to the method or apparatus described herein.

[0513] In general, another example of an embodiment may include a signal including data generated according to any of the methods described herein.

[0514] In general, another example of an embodiment may relate to a signal or a bitstream as described herein and comprising at least one syntax element for representing at least one coefficient of a quantization matrix when the quantization matrix is ​​not predicted, and wherein the at least one syntax element is used to represent a residual when the quantization matrix is ​​predicted.

[0515] Generally, another example of an embodiment may include a device comprising an apparatus according to any example of an embodiment described herein, and at least one of: (i) an antenna configured to receive a signal, the signal including data representing image information; (ii) a frequency band limiter configured to limit the received signal to include data representing image information; and (iii) a display configured to display an image from the image information.

[0516] In general, another example of an embodiment may involve a device as described herein and include one of a television, a television signal receiver, a set-top box, a gateway device, a mobile device, a cellular phone, a tablet computer, or other electronic devices.

[0517] Various embodiments are described herein. Features of these embodiments may be provided individually or in any combination across various claim categories and types. In addition, embodiments may include one or more of the following features, devices, or aspects, individually or in any combination across various claim categories and types: · Providing video encoding and / or decoding, including bit saving in transmission of customized QMs, while keeping the specification as simple as possible;

[0518] Providing video encoding and / or decoding, including improved QM prediction, so that simple adjustments can be made at low bit cost;

[0519] Provide video encoding and / or decoding, including a simple way to specify global offsets;

[0520] Provide video encoding and / or decoding, including one or more of the following: in addition to replication and downsampling,

[0521] It also scales the QM's predictions by applying a scaling factor; provides a simple way to specify a global offset;

[0522] as well as allowing the use of variable length residuals to refine the predictions;

[0523] Providing video encoding and / or decoding including combining improved prediction with residual to further refine the QM with potentially lower cost than full encoding;

[0524] Providing video encoding and / or decoding including QM prediction with offset in addition to copying / downscaling; Providing video encoding and / or decoding including QM prediction plus (variable length) residual;

[0525] Providing video encoding and / or decoding including QM prediction with scaling factors in addition to copying / downscaling;

[0526] Providing video encoding and / or decoding including QM prediction, the QM prediction including a combination of a scaling factor and an offset or a residual;

[0527] Provide video encoding and / or decoding, including reducing the number of bits required to deliver QM and allowing user-defined tradeoffs between accuracy and bit cost while keeping the specification simple;

[0528] Provide video encoding and / or decoding including reducing the number of bits required to deliver QM and allowing a user-defined tradeoff between accuracy and bit cost while keeping the specification simple based on including one or more of the following features:

[0529] oAdd residuals to QM forecasts;

[0530] oAdd scaling factors to QM forecasts;

[0531] oCombining residuals and scaling;

[0532] oAlways use prediction (remove prediction mode flag);

[0533] Providing video encoding and / or decoding, including modifications over replication for QM prediction, the modifications comprising adding a global offset, wherein the global offset can be explicitly specified;

[0534] Providing video encoding and / or decoding, including a global offset, the global offset being assignable as a first coefficient of a DPCM coded residual;

[0535] Providing video encoding and / or decoding including adding a residual to a prediction;

[0536] Providing video encoding and / or decoding, including including residual on top of QM prediction to refine the prediction or providing an optimized decoding scheme for QM (prediction plus residual instead of direct decoding);

[0537] Providing video encoding and / or decoding including syntax for transmitting QM coefficients when not predicted, said syntax being reusable for transmitting residuals when QM is predicted;

[0538] Providing video encoding and / or decoding including adding a residual, wherein the number of coefficients of the residual may be variable;

[0539] Providing video encoding and / or decoding including adding a residual, wherein adding the residual may involve using a flag when in a prediction mode to indicate that coefficients may still be transmitted with a non-prediction mode but interpreted as a residual;

[0540] Providing video encoding and / or decoding, including when not in prediction mode, coefficients can still be interpreted as residuals above a default QM prediction;

[0541] Providing video encoding and / or decoding including using variable length residuals over prediction;

[0542] Video encoding and / or decoding is provided, including using variable length residuals on top of prediction, wherein the number of residual coefficients can be indicated by one or more of the following:

[0543] o an explicit number of coefficients; or

[0544] o a flag indicating the presence of residuals, followed by the number of residual coefficients minus 1, which may be clamped to a power of 2; or

[0545] o 1 + log2 of the number of residual coefficients, where zero indicates no residual, where this can be coded as fixed length (length potentially dependent on matrixId) or Exponential Golomb coding; or

[0546] o an index into a list, including the number of coefficients the list contains, e.g. the list may be fixed by the standard and need not be transmitted, such as {0, 1, 2, 4, 8, 16, 32, 64};

[0547] or

[0548] o As long as at least one normal residual coefficient is to be sent, the number of residual coefficients need not count the potential additional DC residual coefficients that may be transmitted;

[0549] Providing video encoding and / or decoding, including adding a scaling factor to a prediction, such as adding a scaling factor to a copy / decimation;

[0550] Providing video encoding and / or decoding including QM prediction including a variable length residual and a scaling factor, thereby enabling adjustment of the QM based on scaling and / or offset and enabling a trade-off between user defined accuracy and bit cost based on the variable length residual;

[0551] Providing video encoding and / or decoding including QM prediction including a variable length residual and a scaling factor, thereby enabling adjustment of the QM based on a scale and / or an offset, and enabling a trade-off between user defined accuracy and bit cost based on the variable length residual, wherein the scaling factor may be an integer, normalized to a number N (e.g., 16), and wherein the scaling formula scales a reference QM around a neutral value;

[0552] Providing video encoding and / or decoding, comprising adding a scaling factor to a prediction, wherein the scaling factor may be an integer, normalized to a number N, and wherein the scaling factor may be transmitted as an offset of N;

[0553] Providing video encoding and / or decoding, including removing prediction mode flags;

[0554] a bitstream or signal comprising one or more of the described syntax elements or variations thereof;

[0555] A bitstream or signal including syntax conveying information generated according to any of the described embodiments;

[0556] inserting syntax elements in the signaling that enable the decoder to operate in a manner corresponding to that used by the encoder;

[0557] Creating and / or transmitting and / or receiving and / or decoding a bitstream or signal comprising one or more of the described syntax elements or variations thereof;

[0558] Creating and / or transmitting and / or receiving and / or decoding according to any of the described embodiments;

[0559] A method, process, apparatus, medium storing instructions, medium storing data or signal according to any of the described embodiments;

[0560] TV, set-top box, cellular phone, tablet computer, or other electronic device provided for implementing any of the described embodiments;

[0561] A TV, set-top box, cellular phone, tablet computer, or other electronic device that provides for implementing any of the described embodiments and displaying (e.g., using a monitor, screen, or other type of display) the resulting images;

[0562] A TV, set-top box, cellular phone, tablet computer, or other electronic device that selects (e.g., using a tuner) a channel to receive a signal including an encoded image and provides for implementing any of the described embodiments;

[0563] A TV, set-top box, cellular phone, tablet computer, or other electronic device receives over the air (eg, using an antenna) a signal including an encoded image and is provided for implementing any of the described embodiments.

Claims

1. A method comprising: Obtaining information representing at least one coefficient of a quantization matrix and syntax elements from a bitstream containing coded video information; Based on the syntax element, interpreting the information representative of the at least one coefficient as a residual; as well as At least a portion of the encoded video information is decoded based on a combination of the prediction of the quantization matrix and the residual.

2. A device comprising: One or more processors configured to: Obtaining information representing at least one coefficient of a quantization matrix and syntax elements from a bitstream containing coded video information; Based on the syntax element, interpreting the information representative of the at least one coefficient as a residual; as well as At least a portion of the encoded video information is decoded based on a combination of the prediction of the quantization matrix and the residual.

3. A method comprising: obtaining information representing at least one coefficient of a quantization matrix associated with at least a portion of the video information; encoding a syntax element indicating that the at least one coefficient is to be interpreted as a residual; The at least a portion of the video information is encoded based on a combination of a prediction of the quantization matrix and the residual.

4. A device comprising: One or more processors configured to: obtaining information representing at least one coefficient of a quantization matrix associated with at least a portion of the video information; encoding a syntax element indicating that the at least one coefficient is to be interpreted as a residual; The at least a portion of the video information is encoded based on a combination of a prediction of the quantization matrix and the residual.

5. A method according to any one of claims 1 or 3 or an apparatus according to any one of claims 2 or 4, wherein the residual comprises a variable length residual. 6 . The method of claim 5 , wherein the variable length of the residual depends on the syntax element when the quantization matrix is ​​in a prediction mode in which the quantization matrix is ​​predicted from a reference quantization matrix according to the quantization matrix.

7. The apparatus of claim 5, wherein when the quantization matrix is ​​in a prediction mode in which the quantization matrix is ​​predicted from a reference quantization matrix according to the quantization matrix, the variable length of the residual depends on the syntax element.

8. The method of claim 1, wherein when the quantization matrix is ​​in a prediction mode in which the quantization matrix is ​​predicted from a reference quantization matrix according to the quantization matrix, the amount of information representing at least one coefficient obtained from the bitstream depends on the syntax element.

9. The method of claim 1, wherein the quantization matrix is ​​obtained by adding elements of a reference quantization matrix to elements of the residual.

10. The method of claim 9, wherein when the quantization matrix is ​​in a prediction mode in which the quantization matrix is ​​predicted from a reference quantization matrix, all elements of the residual are inferred to be equal to 0 for a given value of the syntax element.

11. A method according to claim 1 or an apparatus according to claim 2, wherein the bitstream containing the information representing at least one coefficient of the quantization matrix is ​​obtained by decoding one or more syntax elements, the same one or more syntax elements being used to transmit the at least one coefficient when the quantization matrix is ​​not predicted and being used to transmit the residual when the quantization matrix is ​​predicted.

12. The method of claim 3, wherein the encoding, or the apparatus of claim 4, wherein the at least one processor is configured to encode: A bitstream is generated, the bitstream comprising one or more syntax elements that signal the information representing at least one coefficient of the quantization matrix, the same one or more syntax elements being used to transmit the at least one coefficient when the quantization matrix is ​​not predicted and being used to transmit the residual when the quantization matrix is ​​predicted.

13. The method of claim 1, comprising decoding a flag having a first value specifying that coefficients of the quantization matrix are explicitly signaled in the bitstream or a second value specifying that the quantization matrix is ​​derived from a reference quantization matrix.

14. The method of claim 13, comprising obtaining a further syntax element indicating the reference quantization matrix.

15. The method of claim 13, wherein when the decoded flag has the first value, all elements of the reference quantization matrix are inferred to be equal to 8.

16. The method of claim 14, wherein all elements of the reference quantization matrix are inferred to be equal to 16 when other syntax elements are 0.

17. The method of claim 14, wherein the reference quantization matrix is ​​derived from a previously transmitted matrix identified by other syntax elements when other elements are not zero.

18. A non-transitory computer-readable medium storing executable program instructions, the executable program instructions causing a computer executing the instructions to perform the method according to any one of claims 1 or 3.

19. A non-transitory computer-readable medium storing a bitstream formatted to include syntax elements of the method according to claim 3 and coded image information.

20. The non-transitory computer-readable medium of claim 19, comprising one or more syntax elements for transmitting the at least one coefficient of the quantization matrix when the quantization matrix is ​​not predicted, and wherein the same one or more syntax elements are used to transmit the residual when the quantization matrix is ​​predicted.

Images