Encoding and decoding quantization matrices using parameterized models
Through parameterized models and mononomial polynomial representation, the problem of inefficient quantized matrix encoding and decoding in video encoding and decoding is solved, and compact representation and efficient encoding and decoding are realized, which are suitable for the needs of future video standards.
Patent Information
- Application Number
- CN201980052630.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-09-17
- Filing Date
- 2019-08-08
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2039-08-08
AI Technical Summary
Existing video encoding and decoders have inefficient problems when processing quantization matrix encoding and decoding, especially when processing large quantization matrices, traditional methods are complex and have high bit costs, making it difficult to meet the requirements of future video standard bit rate and complexity. .
Using a parametric model, the quantization matrix is represented by a mononomial polynomial, the number of polynomial parameters is controlled to reduce complexity and bit cost, and the calculation efficiency is improved through integer calculation and normalization processing. It is suitable for the quantization and quantization matrix in video encoding and decoding. Dequantization operation.
The compact representation and efficient encoding and decoding of the quantized matrix are realized, reducing bit cost and computational complexity, suitable for future video standards, while maintaining high accuracy and flexibility.
Smart Images

Figure CN112771865B_ABST
Abstract
Description
Technical Field
[0001] The present embodiment generally relates to a method and apparatus for video encoding or decoding, and more particularly, to a method and apparatus for encoding and decoding a quantization matrix in video encoding or decoding. Background Art
[0002] In order to achieve high compression efficiency, image and video coding schemes usually use prediction and transformation to fully utilize the spatial and temporal redundancy in the video content. In general, intra-frame or inter-frame prediction is used to exploit intra-frame or inter-frame correlation, and then the difference between the original block and the predicted block (often expressed as a prediction error or prediction residual) is transformed, quantized, and entropy encoded and decoded. In order to reconstruct the video, the compressed data is decoded by the inverse process corresponding to entropy encoding and decoding, quantization, transformation, and prediction. Summary of the invention
[0003] According to an embodiment, a method for video decoding is provided, comprising: accessing a parameterized model based on a parameter sequence; determining a plurality of parameters corresponding to a subset of the parameter sequence; associating each of the plurality of parameters with a corresponding parameter of the subset of the parameter sequence to represent a quantization matrix; dequantizing transform coefficients of a block of an image based on the quantization matrix; and reconstructing the block of the image in response to the dequantized transform coefficients.
[0004] According to another embodiment, a method for video encoding is provided, comprising: accessing a parameterized model based on a parameter sequence; determining a plurality of parameters corresponding to a subset of the parameter sequence; associating each of the plurality of parameters with a corresponding parameter of the subset of the parameter sequence to represent a quantization matrix; quantizing transform coefficients of a block of an image based on the quantization matrix; and entropy encoding the quantized transform coefficients.
[0005] According to another embodiment, a device for video decoding is provided, comprising: a component for accessing a parameterized model based on a parameter sequence; a component for determining a plurality of parameters corresponding to a subset of the parameter sequence; a component for associating each of the plurality of parameters with a corresponding parameter of the subset of the parameter sequence to represent a quantization matrix; a component for dequantizing transform coefficients of a block of an image based on the quantization matrix; and a component for reconstructing the block of the image in response to the dequantized transform coefficients.
[0006] According to another embodiment, a device for video encoding is provided, comprising: a component for accessing a parameterized model based on a parameter sequence; a component for determining a plurality of parameters corresponding to a subset of the parameter sequence; a component for associating each of the plurality of parameters with a corresponding parameter of the subset of the parameter sequence to represent a quantization matrix; a component for quantizing transform coefficients of a block of an image based on the quantization matrix; and a component for entropy encoding the quantized transform coefficients.
[0007] According to another embodiment, an apparatus for video decoding is provided, comprising one or more processors, wherein the one or more processors are configured to: access a parameterized model based on a parameter sequence; determine a plurality of parameters corresponding to a subset of the parameter sequence; associate each of the plurality of parameters with a corresponding parameter of the subset of the parameter sequence to represent a quantization matrix; dequantize transform coefficients of a block of an image based on the quantization matrix; and reconstruct the block of the image in response to the dequantized transform coefficients. The apparatus may also include one or more memories coupled to the one or more processors.
[0008] According to another embodiment, a device for video encoding is provided, comprising one or more processors, wherein the one or more processors are configured to: access a parameterized model based on a parameter sequence; determine a plurality of parameters corresponding to a subset of the parameter sequence; associate each of the plurality of parameters with a corresponding parameter of the subset of the parameter sequence to represent a quantization matrix; quantize transform coefficients of a block of an image based on the quantization matrix; and entropy encode the quantized transform coefficients. The device may also include one or more memories coupled to the one or more processors.
[0009] According to another embodiment, a signal is formatted to include: a value indicating the number of parameters in a plurality of parameters; the plurality of parameters, wherein each parameter in the plurality of parameters is associated with a corresponding parameter of a subset of a parameter sequence to represent a quantization matrix, wherein a parameterized model is based on the parameter sequence; and transform coefficients of a block of an image quantized based on the quantization matrix. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] Figure 1 A block diagram of an embodiment of a video encoder is illustrated.
[0011] Figure 2 A block diagram of an embodiment of a video decoder is illustrated.
[0012] Figure 3 is a pictorial example illustrating a 2D separable transform.
[0013] Figure 4The default intra and inter 8x8 quantization matrices (QM) defined in HEVC are illustrated.
[0014] Figure 5 A process for determining QM in HEVC is illustrated.
[0015] Figure 6 Illustration of upright diagonal scan and default intra 8x8 QM in HEVC.
[0016] Figure 7 Illustration of the different transformations used in JEM (Joint Exploration Model).
[0017] Figure 8 Example matrices found in standards and encoders are illustrated.
[0018] Fig. 9 Two matrices of complex shapes are shown.
[0019] Fig.10 AVC type (left) and HVS (right) matrices for sizes 8, 16 and 32 are illustrated.
[0020] Fig.11A An example matrix layer for 8x8 size is illustrated, and Fig. 11B Four areas are shown.
[0021] Fig.12 Figure 2. The fitting performance of the [JCTVC-H0460] quadratic model for symmetric (left) or asymmetric (right) test intra matrices of size 8x8.
[0022] Fig.13 An example of using an approximate quantization matrix is illustrated.
[0023] Fig.14 The figure shows the 16x16 AV1 quantization matrix.
[0024] Fig.15 Quantization matrix encoding and decoding performance is illustrated.
[0025] Fig.16 A system for decoding a quantization matrix according to an embodiment is illustrated.
[0026] Fig.17 A method for decoding a quantization matrix according to an embodiment is illustrated.
[0027] Fig.18 A method for normalizing matrix coordinates according to an embodiment is illustrated.
[0028] Fig.19 The dynamic range of the coefficients is illustrated.
[0029] Fig. 20 Plot the effect of coefficient accuracy on the quality of the fit.
[0030] Fig.21 Plotting the effect of coefficient bounds on the quality of the fit.
[0031] Fig. 22 A quantization matrix generator with integer calculations according to an embodiment is illustrated.
[0032] Fig.23 A method for generating a quantization matrix using integer calculation according to an embodiment is illustrated.
[0033] Fig.24 A default HEVC intra matrix with a 10-parameter fit is illustrated.
[0034] Fig.25 The effect of clipping on the fitting error is shown.
[0035] Fig.26 The default HEVC intra matrix with a 6-parameter fit is illustrated.
[0036] Fig. 27 A method for generating a QM in an encoder according to an embodiment is illustrated.
[0037] Fig.28 A method for generating a QM in a decoder according to an embodiment is illustrated.
[0038] Fig.29 A block diagram of a system is illustrated in which aspects of the present embodiments may be implemented. DETAILED DESCRIPTION
[0039] Figure 1 An exemplary video encoder 100 , such as a High Efficiency Video Coding (HEVC) encoder, is illustrated. Figure 1 It is also possible to illustrate an encoder in which the HEVC standard is improved or an encoder adopting a technique similar to HEVC, such as the VVC (Versatile Video Coding) encoder being developed by JVET (Joint Video Exploration Team).
[0040] In this application, the terms "reconstruction" and "decoding" can be used interchangeably, the terms "encoding" or "codec" can be used interchangeably, and the terms "image", "picture" and "frame" can be used interchangeably. Usually, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side.
[0041] Before being encoded, the video sequence may undergo a pre-encoding process (101), for example, applying a color transform to an input color picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of input picture components in order to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and appended to the bitstream.
[0042] In HEVC, to encode a video sequence with one or more pictures, the pictures are partitioned (102) into one or more slices, where each slice may include one or more slice segments. The slice segments are organized into codec units, prediction units, and transform units. The HEVC specification distinguishes between "blocks" and "units", where a "block" addresses a specific area in the sample array (e.g., luma, Y), and a "unit" includes a concatenated block of all coded color components (Y, Cb, Cr, or monochrome), syntax elements, and prediction data (e.g., motion vectors) associated with the block.
[0043] For coding and decoding, the picture is divided into square codec tree blocks (CTBs) of configurable size, and a continuous set of codec tree blocks is grouped into slices. The codec tree unit (CTU) contains the CTB of the coded color component. The CTB is the root of the quadtree divided into codec blocks (CBs), and the codec block can be divided into one or more prediction blocks (PBs) and forms the root of the quadtree divided into transform blocks (TBs). Corresponding to the codec block, prediction block and transform block, the codec unit (CU) includes a prediction unit (PU) and a tree-structured transform unit (TU) set, the PU includes prediction information for all color components, and the TU includes a residual codec syntax structure for each color component. The size of the CB, PB and TB of the luminance component is applicable to the corresponding CU, PU and TU. In the present application, the term "block" can be used to refer to any one of CTU, CU, PU, TU, CB, PB and TB, for example. Furthermore, "block" may also be used to refer to macroblocks and partitions specified in H.264 / AVC or other video codec standards, and more generally to data arrays of various sizes.
[0044] In an exemplary encoder 100, a picture is encoded by an encoder element as described below. The picture to be encoded is processed in units of CUs. Each CU is encoded using either intra or inter mode. When a CU is encoded in intra mode, it performs intra prediction (160). In inter mode, motion estimation (175) and compensation (170) are performed. The encoder decides (105) which of intra mode or inter mode to use to encode the CU, and indicates the intra / inter decision through a prediction mode flag. The prediction residual is calculated by subtracting (110) the predicted block from the original image block.
[0045] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients as well as the motion vectors and other syntax elements are entropy encoded (145) to output a bitstream. The encoder can also skip the transform and apply quantization directly to the untransformed residual signal on a 4x4 TU basis. The encoder can also bypass both the transform and quantization, i.e., the residual is directly encoded and decoded without applying the transform or quantization process. In direct PCM encoding and decoding, no prediction is applied and the codec unit samples are encoded and decoded directly into the bitstream.
[0046] The encoder decodes the encoded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150) to decode the prediction residual. The decoded prediction residual and the predicted block are combined (155) to reconstruct the image block. A loop filter (165) is applied to the reconstructed picture, for example to perform deblocking / SAO (sample adaptive offset) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (180).
[0047] Figure 2 A block diagram of an exemplary video decoder 200, such as a HEVC decoder, is shown. In the exemplary decoder 200, a bitstream is decoded by decoder elements as described below. The video decoder 200 generally performs the same operations as described below. Figure 1 The encoding pass shown in corresponds to a decoding pass that performs video decoding as part of the encoded video data. Figure 2 It is also possible to illustrate a decoder in which the HEVC standard is improved or a decoder adopting a technique similar to HEVC (such as a VVC decoder).
[0048] In particular, the input to the decoder includes a video bitstream that may be generated by the video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, picture partitioning information, and other codec information. The picture partitioning information indicates the size of the CTU and the manner in which the CTU is split into CUs (and possibly PUs when applicable). Thus, the decoder can divide (235) the picture into CTUs and each CTU into CUs based on the decoded picture partitioning information. The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residuals.
[0049] The decoded prediction residual and the predicted block are combined (255) to reconstruct the image block. The predicted block can be obtained (270) from intra-frame prediction (260) or motion compensated prediction (i.e., inter-frame prediction) (275). A loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280). The decoded picture may also undergo post-decoding processing (285), such as an inverse color transform (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or an inverse remapping that performs the inverse of the remapping process performed in the pre-encoding process (101). The post-decoding processing may use metadata derived in the pre-encoding process and signaled in the bitstream.
[0050] As described above, the prediction residual is transformed and quantized. Consider an MxN (M columns x N rows) residual block ([U]) input to a 2D MxN forward transform. MXN ), a 2D transform is usually implemented by applying an N-point transform to each column (i.e., vertical transform) and an M-point transform to each row (i.e., horizontal transform), as Figure 3 As shown in . Mathematically, the forward transform can be expressed as:
[0051] [C] MxN =[A] T NxN x[U] MxN x[B] MxM
[0052] Where [A] NxN is the N-point transformation matrix applied vertically, [B] MxM is an M-point transformation matrix applied horizontally, and "T" (superscript) is the matrix transpose operator. Thus, a separable transformation consists in applying the horizontal and vertical transformations successively to each row and column of the 2D residual block.
[0053] In HEVC, the two-dimensional transform uses the same separable transform for both horizontal and vertical filtering. In other words, matrix B is the same as matrix A:
[0054] [C] MxN =[A] T NxN x[U] MxN x[A] MxM
[0055] Note that in HEVC, only square transforms are supported, which means N=M.
[0056] Then the resulting MxN transform coefficients ([C] MXN ) is quantized to obtain a quantized transform coefficient ([CQ] MXN). In HEVC, unified reconstruction quantization (URQ) is used for quantization, which is conceptually equivalent to dividing by the quantization step size (Qstep). More generally, for video coding, it should be noted that other quantization methods can be used, for example, by further considering quantization rounding offsets or using non-uniform quantization.
[0057] To quantize the transform coefficients, in HEVC, the encoder can specify a customized quantization (scaling) matrix (QM) for each transform block size and separately for intra-frame and inter-frame prediction for use by the decoder in inverse quantization scaling. The quantization matrix is generally designed to provide higher accuracy to coefficients corresponding to frequencies that are more sensitive to human perception. The goal is to adapt the quantization of transform coefficients, generally by taking into account the characteristics of the human visual system for different spatial frequencies of the signal, to improve the perceptual quality of the encoded and decoded video. For example, low frequencies are not as quantized as high frequencies.
[0058] In general, using quantization matrices does not improve objective fidelity as measured by mean squared error (or equivalently, PSNR), but often improves subjective fidelity. Default values for quantization scaling matrices are specified in the standard, and encoders can also choose to use customized values by sending a representation of those values at the sequence or picture level.
[0059] More specifically, the transform coefficients are quantized according to the scaling values provided in the quantization matrices. After the transform is applied to the residual values, the luma and chroma transform coefficients in the transform block are quantized according to the integer scaling values present in the intra and inter QMs for the specific component and block size: 6 matrices for each of the transform sizes 4x4, 8x8, 16x16 and 2 matrices for 32x32 (chroma in 4:4:4 sampling mode reuses the matrices for 16x16), forming a total of 20 matrices. To reduce memory storage requirements, the matrix definition is restricted to 8x8 resolution: the complete matrices for 16x16 and 32x32 transform sizes are obtained by 2x2 and 4x4 sample repetitions, respectively.
[0060] The HEVC specification defines default intra and inter QMs in HEVC based on the human vision model. If the default matrix indicated by the usage flag is not used, then a self-organizing (hoc) matrix explicitly signaled in the bitstream may be used.
[0061] The elements in the QM apply different quantization scaling to transform coefficients at different frequencies in the transform block. Therefore, these QMs have the ability to control the quantization step size and therefore the ability to adjust the quantization based on the frequency of the transform coefficients. A transform block contains DC and AC transform coefficients, where the DC transform coefficient is the lowest frequency component and the AC coefficients correspond to low, medium, and high frequency components. Because low-frequency transform coefficients are generally more important for video signal reconstruction, the default QM in HEVC applies coarser quantization to medium and high frequency AC transform coefficients.
[0062] Quantization on the encoder side is not normative and can be done in different ways. However, for HEVC encoders, it usually reproduces or approximates the following formula:
[0063] CQ[x][y]=sign(C[x][y])*(abs(C[x][y])*invScale[QP%6] / QM[x][y]+quantOffset)>>quantShift
[0064] in
[0065] -CQ is a block of quantized coefficients,
[0066] - C is the block of transform coefficients,
[0067] -QM is the quantization matrix,
[0068] - invScale[k] = {26214, 23302, 20560, 18396, 16384, 14564}, where k = 0, ..., 5,
[0069] -QP is the quantization parameter,
[0070] - quantShift is a parameter that depends on the scaling involved in the forward transform applied at the encoder, on the transform block size, on the signal bit depth and on the QP.
[0071] In the HEVC reference encoder, quantShift=14+QP+TransformShift, where TransformShift is related to the scaling involved in the forward transform,
[0072] - quantOffset is a parameter that depends on the parameter quantShift, for example, quantOffset = 1 << (quantShift 1),
[0073] -(x,y) is the position of the coefficient,
[0074] -Abs(r) is the absolute value of r,
[0075] -If r<0, then -sign(r)=-1, otherwise 1,
[0076] - "*" is scalar multiplication, " / " is integer division with the result truncated towards zero, and "%" is the modulus operator.
[0077] On the decoder side, QM can be applied in dequantization compliant with the HEVC specification based on the following formula:
[0078] C'[x][y]=(CQ[x][y]*QM[x][y]*
[0079] levScale[QP%6]<<(QP / 6))+(1<<(bdShift-1)))>>bdShift
[0080] in
[0081] -CQ is a block of quantized coefficients,
[0082] -C' is the block of dequantized coefficients,
[0083] -QM is the quantization matrix,
[0084] -levScale[k] = {40, 45, 51, 57, 64, 72}, where k = 0, ..., 5,
[0085] -QP is the quantization parameter,
[0086] bdShift is defined for the HEVC Normal profile as follows: bdShift=Max(20-bitDepth, 0), where bitDepth is the bit depth of the samples of the considered component (e.g., Y, Cb, or Cr),
[0087] -(x,y) is the position of the coefficient.
[0088] In HEVC, only default QMs are defined for 8x8 transform blocks to reduce memory storage requirements. The default intra and inter 8x8 QMs defined in HEVC are as follows and are Figure 4 (note that they are all symmetric). QMs for larger blocks are obtained by upsampling the 8x8 QM. To create a 16x16 QM, each entry in the 8x8 QM is copied into a 2x2 region. To create a 32x32 QM, each entry in the 8x8 QM is copied into a 4x4 region. For 4x4 transform blocks, the default QM is flat (all components equal to 16). The default QM depends only on the intra / inter mode, but is the same for Y, Cb, and Cr components. The copying rule can also be applied to rectangular blocks (e.g., those in JEM).
[0089]
[0090] The intra-frame default QM is based on the human visual system, as explained in the patent application of M. Haque et al. (U.S. Patent Application No. 13 / 597,131, Publication No. US2013 / 0188691, "Quantization matrix design for HEVC standard", hereinafter referred to as [US20130188691]) or in the article entitled "Designing JPEG quantum tables based on human visual system" by Long-Wen Chang et al. (hereinafter referred to as "Chang") in Signal Processing: Image Communication, Vol. 16, No. 5, pp. 501-506, January 2001, and can be derived as explained in the following procedure. First, the radial frequency f per degree of viewing angle corresponding to the coefficient at position (x, y) is defined in units of cycles as:
[0091]
[0092] in, and N are block width or height (u=0, ..., N-1, v=0, ..., N-1, typically N=8), Δ, K and s are constant parameters (to obtain HEVC 8x8 QM, Δ should be set to 0.00025, K to 0.00893588, and s to 0.7).
[0093] Then, the modulation transfer function H(f) is defined as
[0094]
[0095] where f max = 8 (cycles per degree). The QM value is calculated as Round(16 / H(f)), where Round(x) gives the nearest integer value to x.
[0096] The inter-frame default QM can be derived from the intra-frame default QM using the following process:
[0097] 1.QMinter(0,0)=QMintra(0,0)
[0098] 2. For n = 1…N-1
[0099] -QMinter(0,n)=QMinter(0,n-1)+int(S1*(QMintra(0,n)-QMintra(0,n-1))+0.5),
[0100] - Where S1 = 0.714285714
[0101] 3.QMinter(0,N-1)=QMintra(0,N-1)
[0102] 4. For m = 1…N-1
[0103] -QMinter(m,N-1)=QMinter(m-1,N-1)+int(S3*(QMintra(m,N-1)-QMintra(m-1,N-1))+0.5),
[0104] - where S3 = 0.733333333, where int(r) is the nearest integer value of r,
[0105] 5. For m = 1...N-1, for n = 0...N-2
[0106] -QMinter(m,n)=QMinter(m-1,n+1)
[0107] The coefficients of the quantization matrix are signaled in the bitstream using scaling lists. There is one scaling list per block size, indexed by the parameter sizeId, and one per mode, indexed by the parameter matrixId, both specified as follows:
[0108] sizeId = 0 blocks of 4x4, 1 block of 8x8, 2 blocks of 16x16, 3 blocks of 32x32
[0109] matrixId = 0 intra Y, 1 intra U, 2 intra V, 3 intra Y, 4 intra U,
[0110] 5 frames V
[0111] For 32x32 blocks, you can only set matrixId to 0 or 3.
[0112] The syntax used in the HEVC specification for signaling scaling lists is reproduced below.
[0113]
[0114] For each possible value of the two parameters sizeId and matrixId, apply Figure 5The process 500 shown in FIG. 5 determines the actual QM from either the default QM or the explicitly decoded value of the exemplary HEVC decoder. At step 510, the decoder checks whether the syntax element scaling_list_pred_mode_flag[sizeId][matrixId] is equal to 0. If not, then at step 550, the decoder decodes the scaling list value from the variable ScalingList[sizeId][matrixId][i], i=0, ..., coefNum-1, and constructs the QM in the upright diagonal scan order, as Figure 6 . If scaling_list_pred_mode_flag[sizeId][matrixId] is equal to 0, then at step 520, the decoder checks whether the syntax element scaling_list_pred_matrix_id_delta[sizeId][matrixId] is equal to 0. If not, then matrixId is modified (530) based on scaling_list_pred_matrix_id_delta[sizeId][matrixId]. At step 540, a scaling list value in the default scaling list value identified by sizeId and matrixId is obtained and a QM is constructed, where Figure 6 The right side shows the default matrix for an 8x8 frame.
[0115] As mentioned above, previous video codecs (such as HEVC compliant video codecs) are based on 2D separable transforms using the same vertical and horizontal transforms. Therefore, the derived QM is a generally symmetric 2D QM, applicable to these 2D transforms.
[0116] However, in the current JEM, five different horizontal / vertical transformations are defined, which are shown in Table 1 and in Figure 7 The DCT-II transforms are derived from the five transforms described in 4x4 for the 4x4 size. For sizes from 4x4 to 64x64, flags are used at the CU level to control the combination of transforms. When the CU flag is equal to 0, DCT-II is applied to both horizontal and vertical transforms. When the CU flag is equal to 1, two additional syntax elements are signaled to identify the horizontal and vertical transforms to be used. Note that other horizontal / vertical transforms may also be considered, such as the identity transform (which corresponds to skipping the transform in one direction).
[0117] Table 1. Transform basis functions of DCT-II / V / VIII and DST-I / VII for N-point input in JEM.
[0118]
[0119] For the intra case, the set of possible transforms depends on the intra mode. Three sets are defined as follows:
[0120] Set 0: DST-VII, DCT-VIII
[0121] ·Set 1: DST-VII, DST-I
[0122] Set 2: DST-VII, DCT-V
[0123] For each intra mode and each transform direction (horizontal / vertical), one of these three sets is enabled. For the inter case, only DST-VII and DCT-VIII are enabled, and the same transform is applied for both horizontal and vertical transforms.
[0124] As mentioned above, in many codecs, a quantization matrix is used. For example, Figure 8 Six example matrices found in standards and encoders are shown. From left to right and top to bottom, the matrices are from JPEG, MPEG2-intra, H264-intra, HEVC-intra, an actual DTT H.264 encoder, and a satellite HEVC broadcast stream. However, since many encoders do not use the default matrices specified in the codec specification, they need to be explicitly encoded and decoded. The cost of encoding and decoding such quantization matrices can become significant.
[0125] More specifically, one or more of the following issues may occur:
[0126] o There can be many quantization matrices: the matrices may depend on the horizontal / vertical size of the transform block, the horizontal / vertical transform type, and the codec mode of the codec unit containing the transform block; the number of possible combinations can become very large if 4, 8, 16, 32, 64 size transforms with all rectangular variants, intra / inter and Y / U / V are considered, which would result in 150 matrices (compared to 8 matrices for H.264 and 20 matrices for HEVC).
[0127] o Recent codecs tend to use larger transform sizes, which in turn require the transmission of larger quantization matrices, which means a larger number of coefficients.
[0128] o As indicated above, an alternative matrix may be selected that does not match the default matrix found in the specification for psychovisual improvement or other reasons.
[0129] o The quantization matrix can be asymmetric, for example to accommodate interlaced content or anamorphic content.
[0130] o It may be required to change the quantization matrix of each picture in order to better adjust the quantization parameter (e.g., to simulate a QP greater than 51, which is the current maximum specified in AVC and HEVC, or to simulate fractional QP).
[0131] o Bit rate regulation may require sacrificing some frequency; it usually occurs in "emergency mode" when the buffer is almost full; in this case, it is not advantageous to spend many bits encoding and decoding the quantization matrix.
[0132] o It may require to adjust the transform coefficients differently depending on the temporal distance of the temporal frames used to predict the block (the residual statistics strongly depend on this temporal distance), thus requiring frequent changes of the quantization matrix.
[0133] o Even simple changes can be expensive if transfer of the complete matrix is required.
[0134] o In current codecs (e.g. H.264 and HEVC), the encoding and decoding cost of the quantization matrix is somewhat related to its complexity due to DPCM encoding, but this may be pushed further. Similarly, if prediction is used, the encoding and decoding cost should be related to the complexity of the change compared to the predictor.
[0135] Another aspect addressed here is that in current codecs, the quantization matrix is used as a multiplier in the dequantization process, which increases the complexity compared to no matrix: it requires two multiplications instead of one, and increases the dynamic range of the intermediate results. Moreover, the HVS model that is the basis of the default HEVC matrix follows an exponential trend, which is difficult to match with a low complexity model and a small number of parameters.
[0136] During the development of HEVC, some issues have been studied in JCT-VC. For example, the increase in transform size and type has led to discussions and suggestions on the efficiency of quantization matrix encoding and decoding:
[0137] Evidence of the problem is provided in K. Sato, H. Sakurai, “Necessity of Quantization Matrices Compression in HEVC”, JCTVC-E056, 5th JCT-VC Meeting: Geneva, Switzerland, March 16-23, 2011 (hereinafter referred to as [JCTVC-E056]).
[0138] M. Zhou, V. Sze, “Compact representation of quantization matrices for HEVC,” JCTVC-D024, 4th JCT-VC Conference: Daegu, South Korea, January 20-28, 2011 (hereinafter referred to as [JCTVC-D024]), proposed a codec method involving symmetry, subsampling, and linear interpolation.
[0139] J. Tanaka, Y. Morigami, and T. Suzuki, “Quantization Matrix for HEVC,” JCTVC-E073, 5th JCT-VC Meeting: Geneva, Switzerland, March 16-23, 2011 [hereinafter “JCTVC-E073”], proposed a codec with optional prediction according to a linear model, prediction according to other matrices, symmetry, and non-uniform residual quantization.
[0140] G. Korodi and D. He, “QuYK: AUniversal, Lossless Compression Method for Quantization Matrices,” JCTVC-E435, Fifth JCT-VC Conference: Geneva, March 16-23, 2011 (hereinafter referred to as “JCTVC-[E435]”) proposed two methods, one with advanced entropy coding and the other with symmetry, diagonal affine or quadratic prediction and zerotree coding of the residual.
[0141] E. Maani, M. Haque, A. Tabatabai, “Parameterization of Default Quantization Matrices,” JCTVC-G352, 7th JCT-VC Conference: Geneva, Switzerland, November 21-30, 2011 (hereinafter referred to as [JCTVC-G352]), proposed a quadratic model.
[0142] Y. Wang, J. Zheng, X. Zheng, Yun He, “Layered quantization matrices compression”, JCTVC-G530, 7th JCT-VC Meeting: Geneva, November 21-30, 2011 (hereinafter referred to as [JCTVC-G530]) proposed a codec method with layered iterative refinement.
[0143] R. Joshi, J. S. Rojals, M. Karczewicz, “Compression and signaling of quantizer matrices”, JCTVC-G578, 7th JCT-VC Meeting: Geneva, Switzerland, November 21-30, 2011 (hereinafter referred to as [JCTVC-[G578]) proposed a raster scan coding mode with improved differential coding based on the symmetry rule (instead of diagonal lines).
[0144] M. Haque, E. Maani, A. Tabatabai, “High-level Syntaxes for the Scaling List Matrices Parameters and Parametric coding”, JCTVC-H0460, 8th JCT-VC Conference: San Jose, CA, USA, February 1-10, 2012 (hereinafter referred to as [JCTVC-H0460]) proposed a prediction using 3 parameterized models (quadratic or HVS).
[0145] S. Jeong, Hendry, B. Jeon, J. Kim, “HVS-based Generalized Quantization Matrices,” JCTVC-I0518, 9th JCT-VC Conference: Geneva, Switzerland, April 27-May 7, 2012 (hereinafter referred to as [JCTVC-I0518]) proposed an HVS-based model with a single parameter.
[0146] R. Joshi, JS Rojals, M. Karczewicz, “Quantization matrix entries as QP offsets,” JCTVC-I0284, 9th JCT-VC Meeting: Geneva, Switzerland, April 27-May 7, 2012 (hereinafter [JCTVC-[I0284]) recommends using quantization matrix as QP offsets.
[0147] Evidence of increased coding and decoding costs for quantization matrices has been shown in [JCTVC-E056]. This has encouraged the development of techniques for more efficient coding and decoding of quantization matrices. J. Tanaka, Y. Morigami, and T. Suzuki, entitled "Enhancement of quantization matrix coding for HEVC", 6th JCT-VC Meeting: Torino, Italy, July 14-22, 2011 (hereinafter referred to as [JCTVC-F475]) provides a paper on matrices of complex shapes (such as Fig. 9 to test the efficiency of various proposed codecs. Fig. 9 The quantization matrices provided in are all 16x16, where the ones on the left are symmetric and the ones on the right are asymmetric. They are not meant to be meaningful, but are designed for stress testing: difficult to encode, but not too much.
[0148] [JCTVC-D024] deals with x / y and center-point symmetry, as well as the representation of subsampling and linear interpolation reconstruction. Note that x / y symmetry refers to matrix symmetry, i.e., M(x,y)=M(y,x). x / y symmetry is referred to as "135 degree symmetry" in [JCTVC-D024]. Center-point symmetry is referred to as "45 degree symmetry" in [JCTVC-D024], and mathematically, M(x,y)+M(N-1-y,N-1-x)=C, where N is the size of the (square) matrix, C is a constant, and x, y=0, 1, 2, ...N-1. [JCTVC-D024] also proposed a restriction to positive-only DPCM encoding and decoding, but it has since been withdrawn.
[0149] Center-point symmetry is generally not relevant, and x / y symmetry is not always correct. One benefit of the [JCTVC-D024] approach comes from the subsampled representation, which results in a bit cost reduction of about 2.5x and small errors. However, since the upsampling is not completely regular, it increases the complexity of the specification.
[0150] [JCTVC-E073] provides several codec modes. First, one mode involves affine prediction for the horizontal, vertical, and diagonal axes, and linear interpolation for the rest; however, this mode was abandoned in the second version ([JCTVC-F475]). The others involve prediction from other matrices (with fixed trees and higher-order trees), non-uniform residual quantization (quantization matrix for quantization matrix), x / y symmetry, and various scanning and coding options (raster DPCM, sawtooth DPCM, or sawtooth RLE), as well as VLC entropy coding. It was suggested in the first version to interpret the quantization matrix as a QP offset.
[0151] In the [JCTVC-E073] approach, the main reduction in bit cost comes from quantization, which produces a modest error. However, there are many options, and the specification seems too complex for this purpose. A 3-axis affine model followed by linear interpolation is interesting, but probably no better than a single-level polynomial model with the same number of parameters. It was not until [JCTVC-I0284] (see below) that this was further discussed as an explanation for QP offsets.
[0152] [JCTVC-E435] proposes two methods: the first is an advanced entropy codec with string substitution and arithmetic coding; the second has three modes: an asymmetric mode, where each upright diagonal is predicted using a quadratic model with its own set of parameters; an x / y symmetry mode, where the quadratic model is replaced by an affine model; and an x / y+center point symmetry mode, where only half of the diagonal is coded. The parameters for each diagonal are transmitted as indices to a limited set of coefficients, as well as an offset. The residual is coded using a zerotree.
[0153] For [JCTVC-E435], the first approach is a new entropy codec that seems too complex for this purpose. In the second approach, modeling each diagonal with a coarse-grained low-degree polynomial is interesting, but may be too complex compared to the global parameterized model: the diagonals are obviously correlated in real life. The test matrix (the same as [JCTVC-D024]) is a bit too simple and biased towards semi-diagonal zigzag scan efficiency; it is not clear whether the compression performance comes from the per-diagonal affine model or the zerotree codec.
[0154] [JCTVC-G352] and [US20130188691] proposed a quadratic polynomial model for the default matrix. The matrix coefficients are approximately
[0155] QM(x,y)=(a(x 2 +y 2 )+bxy+c(x+y)+d) / 2 q
[0156] where x and y are the positions of the coefficients (from 0 to N-1 for NxN matrices), (a,b,c,d) are the parameters of the model (as shown in Table 2), and q is the bit precision which is typically 10. It is said that simple scaling of (a,b,c,d) will provide straightforward extension to other matrix sizes: for a 2Nx2N matrix, they are divided by (4,4,2,1), respectively.
[0157] Table 2. Recommended parameters for AVC type and HVS default matrix
[0158]
[0159] This model provides a good fit for AVC-like matrices (since they are almost flat, hence easy), but not for HVS, and is limited to convex shapes. Also, the dynamic range of the parameters varies a lot with the x or y index, especially for large dimensions; this will cause accuracy issues for some lower bit-precision q. Fig.10 AVC type (left) and HVS (right) matrices for sizes 8, 16 and 32 are illustrated, with the black grid showing the model and the grey surface showing the fitted target.
[0160] In [JCTVC-G352], the work of Zhang et al. is cited, in which the encoder uses a quadratic model internally to optimize the quantization matrix, assuming that the optimized matrix is transmitted in a traditional manner (for example, see the patent application of H. Zhang et al., "Method and apparatus for modeling quantization matrices for image / video encoding", U.S. Patent No. US 8,326,068, hereinafter referred to as [US8326068]).
[0161] [JCTVC-G530] and Y. Wang, J. Zheng, Yun He, entitled "Layered quantization matrices representation and compression", JCTVC-H0314, 8th JCT-VC Conference: San Jose, California, USA, February 1-10, 2012 (hereinafter referred to as [JCTVC-H314]) proposed a new method. The method follows a layered approach, where a layer index is assigned to each position in the matrix (this assignment matrix is specific to each matrix size). Fig.11AAn example matrix layer for 8x8 size is shown. First, the coefficients at the root level are decoded, then the coefficients for the next layer are predicted using linear interpolation and corrected with the residual; the process continues for the next layer, and so on. The residual can be quantized at different scales for the 4 regions defined in [JCTVC-E073] (see Fig. 11B ) and the residual can be transmitted to a limited number of layers (e.g. only the first 2 layers), with the remaining layers being zero. There is an x / y symmetric mode where only half of the residual is transmitted. Depending on the layer, the residual is encoded or decoded using DPCM or RLE. In addition to explicit encoding and decoding as just described, matrices can also be copied from other matrices of the same picture (using a fixed tree, from larger size to smallest size, using regular subsampling), or differentially encoded from the same matrix of a previous picture.
[0162] This method outperforms some other methods with an interesting layering approach, but is generally too complex for this purpose. Specifically, linear interpolation using unevenly spaced anchor points is complex, with ratios that can be numbers like 1 / 3 or 3 / 7.
[0163] [JCTVC-G578] proposes a simple encoding and decoding method based on symmetry rules and subsampling: the coefficients are scanned in raster order, each coefficient is predicted with the maximum value of the neighbors above and to the left, and corrected with the residual. Next, the corrected coefficients are used for prediction. Since the matrix coefficients are usually increasing, the residuals are encoded and decoded using the Golomb index (exp-golomb, EG) with a signed-to-unsigned mapping that favors positive values. Residual transmission can be limited to a subblock of the matrix (e.g., the upper left corner), and the rest of the block can be inferred to be 0. The second version, as described in R. Joshi, JS Rojals, M. Karczewicz, “Coding of quantization matrices based on modified prediction and mapping to unsigned values,” JCTVC-H0451, 8th JCT-VC Conference: San Jose, California, USA, February 1-10, 2012 (hereinafter referred to as [JCTVC-H0451]), adds golomb-rice coding with variable degrees.
[0164] This approach has good performance in the asymmetric case (about 40% fewer bits) and very low complexity. However, this can be affected by the test matrix, which is quasi-monotonically increasing and therefore well suited for max(left, above) prediction, and outperforms the diagonal due to the significant asymmetry. Note that 15 of the 40% comes from the shift in the signed to unsigned mapping due to the increase in matrix values. The gain is much lower for the symmetric case, and is overall lower for the smaller size case than for the larger size case.
[0165] [JCTVC-H0460] proposed a coding and decoding method based on parameterized models. First, a parameterized model is selected from three possible models:
[0166] Table 3. Parameterized model proposed in [JCTVC-H0460]
[0167]
[0168] Then, optionally, the residual can be transmitted using conventional methods (e.g., zigzag scanning and symmetry rules). Test results are given for parameters and residuals using EG5 codec and normalized to 32. Matrix sizes larger than 8x8 are generated using 8x8 representation and then up-sampled.
[0169] The method is complicated by multiple parameterized models and other options such as residual encoding and decoding methods. Moreover, the HVS model is computationally demanding. Results with residuals did not show better performance than traditional methods, but it was argued that if residuals are not needed (i.e., the model is considered sufficient for this purpose), then the representation is very compact. However, this model may not be sufficient because it is limited to 2nd degree polynomials and the HVS version does not provide more degrees of freedom. Fig.12 In Figure 1, the fitting performance of the [JCTVC-H0460] quadratic model for symmetric (left) or asymmetric (right) test intra matrices of size 8x8 is shown using SSE optimization and positivity constraints, where the grey surface represents the target and the black grid represents the model. The results are 38 and 56 bits (anchors are in 334 and 380).
[0170] [JCTVC-I0518] proposed a general HVS model to transmit a scaling matrix with a single α parameter:
[0171]
[0172] H is the same as in [Chang] or equation (2). A simplified calculation method is also proposed, using an approximation of 1 / H as a power of 2. An example using the approximation is given in Fig.13 Shown in.
[0173] With only one parameter, this model is very compact and the exponential may make it well suited for intra-to-inter conversion and temporal distance adaptation. However, the underlying formula is rather complex and computationally expensive. Simplified computational techniques introduce crude approximations that may not be acceptable (e.g., the lower triangular half is completely flattened). Moreover, restricting the quantization matrix to such matrices with only one degree of freedom may be too severe.
[0174] [JCTVC-I0284] proposes to use the quantization matrix as QP offset (already proposed in [JCTVC-E073]), explaining that this makes the QP adjustment more uniform in both directions, and also simplifies the dequantization process and may save memory in some implementations. It calls for increasing the QP granularity locally by increasing the resolution of the levScale table during the dequantization process. The matrix will be encoded and decoded as offset binary (0-119, with a midpoint of 64).
[0175] During the standardization process, the JCT-VC group was very reluctant to increase the complexity of the matrix codec, and memory storage was seen as key to the cost of the codec. Therefore, matrix subsampling with repetitiveness was chosen because it achieved both goals without additional complexity and with little impact on quality, and furthermore, the design remained essentially unchanged compared to AVC. Various attempts to do non-uniform subsampling were rejected due to the added complexity and unclear benefits. Using symmetry also increased complexity and was therefore not considered worth the effort.
[0176] For larger matrix sizes, sample repeatability is better than linear interpolation because the dequantization process can use a reduced-size representation without any on-the-fly computation. Notably, subjective testing conducted during JCT-VC Conference H showed that small errors in the matrix coefficients have no visual impact, as described in T. Suzuki K. Sato, X. Zhang, R. Joshi, J. Zheng, M. Zhou, “Report on Quantization Matrices Subjective Viewing,” JCTVC-H0730, 8th JCT-VC Conference: San Jose, California, USA, February 1-10, 2012 (hereinafter [JCTVC-H0730]).
[0177] Since small errors seem acceptable and the quantization matrix data are usually smooth, a parameterized model would be a good choice. In fact, the data compression methods used to compare different JCT-VC proposals may be misleading because it does not focus on the actual requirements of the quantization matrix design. The concept of degrees of freedom may be a better match. Moreover, the expected increase in the number of quantization matrices in the next standard drives a more compact representation.
[0178] The emerging AV1 video codec standard also uses quantization matrices. Currently, it utilizes a set of 16 possible matrix sets that are hard-coded in both the encoder and decoder. One matrix set includes matrices for each block size, luma, and chroma. Like in HEVC, each matrix coefficient is used as a multiplication factor for the quantization step of the matching coefficients of the transformed block.
[0179] An index specifying which of the 16 sets to use can be signaled by the encoder at the picture level or 64x64 block level and for each color component separately. This index is called a "QM level" by AV1. Fig.14 A 16x16 AV1 quantization matrix is shown with luma and chroma indices 0 for the top row, and luma indices 4 and 8 for the bottom row.
[0180] According to the hardcoded table, increasing levels results in an increasingly flat matrix. The matrix for index 15 is completely flat (normalized value = 32). In fact, in that case, the decoder interprets index 15 as a "bypass" and skips the matrix during dequantization.
[0181] Here, relying on hard-coded matrices completely avoids the problem of transmitting many matrices. Flexibility is provided by choosing between 16 possibilities, but this is very limited: it can be compared to the c-factor in the HVS model of [JCTVC-H0460] or the alpha exponent of [JCTVC-I0518]. However, compared to HEVC, flexibility is improved by the ability to choose the matrix at the block level.
[0182] On the other hand, hard-coded matrices take up a lot of space in read-only memory, code, and specifications. In particular, there are currently 107,008 8-bit values for the decoder matrix in 6,767 lines of code, and just as many for the encoder.
[0183] The present application proposes a compact representation of a quantization matrix using a simple fixed polynomial model, whose monomials are ordered by increasing complexity, and with a variable number of parameters, thereby trading more bits for better accuracy and reducing the need to encode or decode more or fewer parameters. In the following, some advantages of the proposed technique are provided compared to other known works.
[0184]
[0185]
[0186] The proposed technique can be used as direct matrix coding, or as a residual of prediction, and either for a default matrix or for a custom (transmitted) matrix. In one embodiment, the residual using another coding method can be added to further adjust the QM coefficients.
[0187] As mentioned above, we expect that the bit cost of the quantization matrix will be under great pressure in future video standards. In various embodiments, parameterized models are used to provide a very compact representation, but at the expense of some degrees of freedom.
[0188] Since quantization matrices are usually very smooth, full control over each matrix coefficient is not required: it is clear from subjective observations at JCT-VC meeting H that no difference could be detected between lossy and lossless QM transmission techniques, and the final decision to represent large matrices at 8x8 resolution. Therefore, solutions that encode the quantization matrix can define the global shape with "enough" degrees of freedom.
[0189] In one embodiment, a polynomial model is proposed because among possible parametric surface models, polynomials may provide the highest flexibility for the lowest complexity.
[0190] Note that even though the proposed representation is independent of block size, it is compatible with extensions up to a maximum limit of 8x8 and is repeated for larger sizes as in HEVC. Fig.15 Quantization matrix codec performance is illustrated, where the simplest variant of the model proposed here (using 3, 4, 6, 8, 10, 11, 13, 15, 17, 19 or 21 parameters) is compared with the layered approach proposed by [JCTVC-G530] (the best "lossy" approach proposed for HEVC) for some 8x8 matrices, with the AVC / HEVC codec approach as reference (comparable results can be obtained for other matrices). The proposed model generally scales and outperforms [JCTVC-G530] on the low bitrate side.
[0191] Fig.16 A system 1600 for decoding a quantization matrix according to an embodiment is illustrated. A parameter decoder 1610 obtains model parameters of a current quantization matrix, such as polynomial coefficients {Pi}, from an input bitstream. Then, a quantization matrix generator 1620 generates a current quantization matrix based on the matrix size and the model parameters.
[0192] Fig.17A method 1700 for decoding a quantization matrix according to an embodiment is illustrated. The method 1700 begins at step 1705. The decoder 1710 obtains model parameters for a current quantization matrix, such as polynomial coefficients {Pi}, from an input bitstream. The decoder then generates 1720 a current quantization matrix based on the matrix size and the model parameters. The method 1700 ends at step 1799. The method 1700 can be implemented in the system 1600. In the following, the modeling of the quantization matrix will be described in further detail.
[0193] In one embodiment, the present technique uses a single fixed polynomial to represent the quantization matrix to keep complexity to a minimum. Modulation of bit cost and complexity can be achieved by specifying only the first n polynomial coefficients (also called "polynomial parameters"), with the remaining coefficients implicitly set to zero (or any relevant neutral value).
[0194] One form of a single fixed polynomial is a fully expanded polynomial in (x, y), where x, y indicate the coordinates of a given coefficient in the quantization matrix, with monomials (also called "terms") ordered by increasing exponents, as shown in equation (3), to express as polynomial parameter P i The function derives the quantization matrix coefficients M(x,y).
[0195]
[0196] Where P i are the polynomial coefficients, and is an item, and px i and py i is the monomial m i The x and y indices.
[0197] This form provides high flexibility with reasonable complexity, controlled by the number of polynomial coefficients specified. Since the higher exponents are the last, reducing the number of polynomial coefficients actually reduces the degree of the polynomial, thereby reducing its complexity. Note that the term (i.e., m i The exponent of a variable (i.e., x or y) in (x,y)) is called the degree of that variable in that term; the degree of a term is the sum of the degrees of the variables in that term, and the degree of a polynomial is the maximum degree of any one term with nonzero coefficients.
[0198] More specifically,
[0199]
[0200]
[0201] Here the monomials are sorted as follows:
[0202] Rule 1. Increase the number of items;
[0203] Rule 2. Increase the largest (x or y) index; and
[0204] Rule 3. Increase the y index.
[0205] In equation (4), different subsets of the polynomial coefficients define different degrees of M(x,y). For example, line 1 (“P 0 ”) defines a polynomial of degree 0 (i.e., a constant), and the first and second lines (“P 0 +P 1 x+P 2 y”) defines a first-degree polynomial, and lines 1-3 (P 0 +P 1 x+P 2 y+P 3 xy+P 4 x 2 +P 5 y 2 ) defines a polynomial of degree 2 (such as JCTVC-H0460), and so on. Each line adds terms of the next degree. Note that the maximum number of coefficients for degrees 0, 1, 2, 3, 4, and 5 is 1, 3, 6, 10, 15, and 21, respectively.
[0206] This series can be continued, but we recommend stopping at 4 (i.e., 15 parameters, P 0 -P 14 ), because higher degrees require more and more parameters, and the complexity and dynamic range of the intermediate calculations also increase.
[0207] Note that the polynomial can take different forms. For example, rule 2 can be ignored, or rule 3 can be modified to follow the increase of the x exponent. Although different forms of M(x,y) can be used, both the encoder and the decoder should understand the monomial m i An ordered sequence of (x, y), i = 0, 1, 2, .... Then, when sending or receiving polynomial parameters, the i-th polynomial coefficient P i and the ith monomial m i (x, y), we can associate the polynomial coefficients with the monomials of the same index (i.e., P i and m i (x,y) pairing) to construct a polynomial, such as M(x,y) = ∑ i P i ·m i (x,y).
[0208] Here, by using a well-defined transmission (or storage) order of the polynomial parameters, the complexity of the quantization matrix can be easily controlled by the number of polynomial parameters used to represent the quantization matrix. In one example, the number of polynomial parameters is explicitly or implicitly transmitted in the bitstream as part of the syntax.
[0209] Default polynomial coefficients may be used, or the polynomial coefficients may be transmitted in the bitstream. In one example, a limited number of polynomial parameters may be specified, and the remaining parameters may be inferred to be zero. Transmitting fewer parameters generally requires fewer bits and has simpler shapes that are easier to compute, while more parameters means more bits and more complex shapes that are difficult to compute.
[0210] For degree 4, the number of polynomial parameters can go from 0 to 15 and can be encoded using 4 bits if they need to be transmitted. 0 can be interpreted as the default matrix (or default parameters).
[0211] In one embodiment, a symmetry flag, such as sym, may be added to specify the same coefficients for x and y symmetric monomials, thereby forming symmetric polynomials, reducing the number of parameters (e.g., 9 instead of 15 for degree 4). The mapping for degree 4 is shown in Table 4.
[0212] Table 4. Mapping of symmetric polynomial parameters
[0213]
[0214] Normalization of x and y
[0215] In one embodiment, we propose to normalize the x and y variables to the following, which is not done in previous works such as [JCTVC-G352] or [JCTVC-H0460]:
[0216] 1- Equalize the dynamic range between the various polynomial coefficients. In Table 2, taken from previous work, the coefficients a and d have very different ranges, which can cause serious problems if the polynomial degree and / or matrix size are increased. By normalizing, this problem is solved and the polynomial coefficients can be transmitted with a fixed number of bits. For the same accuracy of the resulting matrix, this approach is simpler than Golomb exponential encoding of the parameters without x and y normalization, and proved to be more efficient in our tests.
[0217] 2- Use the same polynomial coefficients for different matrix sizes (including rectangular sizes). With this approach, using the same polynomial for a half-sized matrix is exactly equivalent to taking every other coefficient of the full-sized matrix; this applies to the x-direction, the y-direction, or both.
[0218] Let us define the normalization value N so that
[0219]
[0220] Where x', y' are intervals [0..size x / y -1] (i.e., the indices of the columns and rows of the matrix), where size x and size y are the horizontal and vertical dimensions of the matrix.
[0221] Fig.18 A method 1800 for normalizing matrix coordinates according to an embodiment is illustrated. The method 1800 may be implemented in the quantization matrix generation 1620 or may be used to perform step 1720.
[0222] Specifically, for a matrix coordinate x′, based on the horizontal size of the quantization matrix and the normalization value N, x normalization ( 1810 ) may be performed: Similarly, for matrix coordinate y′, based on the vertical dimension of the quantization matrix and the normalization value N, y normalization (1830) may be performed: The normalized matrix coordinates x and y may then be used to model (1820) a quantization matrix such as used in equation (4) to generate M(x, y).
[0223] The obvious choice for N is 1, since this puts each monomial in the range [0..1), so the maximum impact of a change (in accuracy) on each coefficient is the same. This is an important consideration when trying to define the number of significant digits required. However, the dynamic range of polynomial coefficients that match a given shape varies a lot, depending on the number of polynomial coefficients and on the rank (P i The coefficients of the 0th and 1st degree polynomials tend to be smaller than the other coefficients, and the larger the number of polynomial coefficients, the larger the dynamic range of all of them.
[0224] For example, unconstrained least squares fitting has been performed on a test set based on H.264, HEVC, and other test 8x8 matrices by minimizing the mean square error between the reconstructed matrix and the test matrix. Fig.19 , the left graph shows the maximum absolute value of each coefficient for several experiments with 6, 10, 15, and 21 polynomial coefficients; the right graph shows the maximum absolute value among all polynomial coefficients for different numbers of coefficients (3 to 21).
[0225] In theory, it is possible to transmit polynomial coefficients with unlimited range using Golomb exponential encoding. However, using a large number of polynomial coefficients for matrix calculations would be impractical and they must be fully specified, thus being bit-limited. This means defining the range of the polynomial coefficients and their accuracy.
[0226] Since quantization matrices are usually defined as 8-bit numbers, it makes sense to define the polynomial coefficients with similar bit depth. If the number of polynomial coefficients is limited to 15 (i.e., 4 times), then they can be constrained to the range [-512...511] (i.e., 10-bit signed), and dropping the 2 LSBs (least significant bits) will make them 8-bit signed. This will reduce the accuracy of the matrix (0,0) value by a factor of 4, but because each coefficient affects the matrix in different places at different levels, the overall accuracy can be satisfactory.
[0227] The normalization value N=1 is not the only possible choice, and other values will also produce other trade-offs between coefficient range and accuracy: for N>1, the accuracy of higher index coefficients has a greater impact, such as Fig. 20 The effect of coefficient bounds (i.e., dynamic range) on the quality of fit for various N is shown in Fig.21 As shown above, it is recommended that for N = 1, 2 is 512, 256 and 128.
[0228] The recommended N is 2, as it produces a stable low coefficient range and is a good tradeoff for overall accuracy, and retains full accuracy for the (0,0) matrix position, which is valuable.
[0229] Efficient computing
[0230] Due to size x and size y is usually a power of 2, and if N is also a power of 2 (we recommend N = 2), then the division in equation (5) simplifies to a bit shift. Let sx = log2(size x )-log2(N) and sy=log2(size y )-log2(N), then equation (5) can be written as:
[0231]
[0232] Reported in equation (3), it becomes:
[0233]
[0234] Now, if we define smax = max(sx.px + sy.py), then we can write
[0235]
[0236] pass
[0237]
[0238] Since smax ≥ 0 and smax - sx.px i -sy.py i ≥0, multiply by This simplifies to a left shift and a final division by 2 smax This can be achieved with a right shift, preferably with rounding:
[0239]
[0240] where s i =smax-(sx.px i +sy.py i ) and rnd=1<<(smax-1).
[0241] This approach maintains good accuracy through integer calculations. Fig. 22 A quantization matrix generator 2200 with integer calculations according to an embodiment is illustrated. The generator 2200 may be used as module 1620.
[0242] According to the matrix size x 、size y ) and normalized value N, sx = log2(size x )-log2(N) and sy=log2(size y )-log2(N), and the (2220) normalized data can be derived: s i =smax-(sx.px i +sy.py i ). The rounding and shifting values may be derived (2230) as: smax=max(sx.px+sy.py) and rnd=1<<(smax-1).
[0243] Based on the quantization matrix coordinates x′ and y′, the (2210) ith monomial can be derived: Combine the ith monomial with the ith polynomial parameter P i Pairing, forming (2240)P i m′ i (x′,y′). Then apply (2250) left shift: P i m′ i(x′,y′)<<s i , and sum the shifted results (2260): ∑ i P i m′ i (x′,y′)<<s i , rounded by addition (rnd) (2270), and right shifted by (2280) smax to form an element in the quantization matrix: M(x′, y′) = ((∑ i P i m′ i (x′,y′)<<s i )+rnd)>>smax.
[0244] Fig.23 A method 2300 for generating a quantization matrix by integer calculation according to an embodiment is illustrated. The generator 2300 may be used as in module 1620 or may be performed in step 1720.
[0245] Method 2300 begins at step 2305. Based on the matrix size (size x 、size y ) and normalized value N, sx = log2(size x )-log2(N) and sy=log2(size y )-log2(N), and the rounding and shifting values can be derived (2310) as: smax=max(sx.px+sy.py) and rnd=1<<(smax-1).
[0246] The decoder then initializes (2320) the variable i to 0, and M(x′, y′) = rnd. At step 2330, the normalized data can be derived: s i =smax-(sx.px i +sy.py i ). Based on the quantization matrix coordinates x′ and y′, the i-th monomial can be derived (2340) as: Combine the ith monomial with the ith polynomial parameter P i Pairing (2350) to form P i m′ i (x′, y′), and added to M(x, y) after left shift (2350): M(x′, y′) + = P i m′ i (x′,y′)<<s iAt step 2360, the variable i is incremented by 1. At step 2370, the decoder checks whether the last polynomial has been processed. If not, control returns to step 2330. Otherwise, at step 2380, M(x',y') is right-shifted by smax to form an element in the quantization matrix: M(x',y') >>=smax. Method 2300 ends at step 2399.
[0247] Most variables can be precomputed, since rnd and smax depend only on the matrix dimensions, s i Depends on the matrix size and the monomial index. Table 5 shows an example:
[0248] Table 5.s i Example calculation of , rnd, smax
[0249]
[0250] For 8x8 matrix and N=2: sx=2, sy=2, smax=8, rnd=128
[0251]
[0252] It can be pointed out that m′ i Do not depend on the matrix size, so they can be computed all at once, with sub-parts for smaller sizes. An efficient incremental implementation is possible with very few multiplications.
[0253] Alternatively, one can calculate p before computing the matrix i Applications i Shifting so that the latter is a series of multiply-accumulate operations, initialized by rnd and then shifted right is a very common operation.
[0254] Analysis of intermediate computational bit depth
[0255] For the example above (8x8 matrix, where N=2), with 8-bit signed P i , Table 6 shows m′ i , P i m′ i and P i m′ i <<s i bit depth.
[0256] Table 6
[0257]
[0258] However, after right shifting smax=8 bits, we expect a result of mbd(matrix bit depth)=8 bits (unsigned) for the matrix values. P must be constrained in the design i , making the result positive and in bounds (0-255). Furthermore, truncating (of the high order bits) after the shift (mbd = 8 bits) or before (mbd + smax = 16 bits) is equivalent. And, furthermore, truncating the operands makes no difference if the result is truncated to the same bit depth, and the two's complement representation of signed values makes it applicable to both signed and unsigned numbers.
[0259] This means that P i m′ i <<s i The accumulators can be limited to (mbd+smax)=16 bits. However, the multipliers cannot be reduced (but their results can be truncated to the working bit depth). In the 8x8 example, they must have 12-bit and 8-bit inputs and 16-bit results; for 256x256 matrices, they should have 32-bit and 8-bit inputs and 36-bit results.
[0260] The final right shift can be adjusted to i This provides some flexibility in the range and accuracy of . Then equation (10) is modified as follows:
[0261]
[0262] This is intended to be used with positive values to increase the accuracy of the parameter. If negative values are used to increase the dynamic range, then it should be noted that this will have an impact on the intermediate calculation bit depth (e.g., -1 will add 1 bit to the accumulator bit depth).
[0263] The rounding value rnd should be adjusted accordingly: rnd = 1 < < (smax-s base -1).
[0264] Lower clipping can be added on top of the polynomial representation, as shown in equation (12) (this also applies on top of equation (10) or (11)). This can improve the fit to conventional quantization matrices, since they often have a flat region at the lower end.
[0265]
[0266] like Fig.24 and Fig.25 As shown in (this was tested without symmetry markings and without custom shifts), this has a significant impact on the quality of the fit, especially for small numbers of parameters. In particular, Fig.24The left figure in shows the fit of the default HEVC intra matrix using 10 parameters without clipping, while the right figure shows it with clipping. Fig.25 The figures in show the effect of clipping on the fitting error of the default HEVC intra 8x8 matrix (left), the JCTVC-F475 8x8 asymmetric test matrix (center), and the H.264 matrix (right).
[0267] It should be noted that due to clipping, the discussion about the range of the polynomial result no longer applies on the lower side: the polynomial is now unbounded below zero, as clipping will correct for this. Either the polynomial coefficients should be constrained to give a result within the matrix range, or the result bit depth should be increased under the relaxed constraint to accommodate the larger range. The recommended option is to add 1 bit to allow negative polynomials (up to -256 for 8-bit matrix coefficients); clipping will make the result positive and discard the sign bit.
[0268] The current HEVC syntax is reproduced below (see HEVC standard version 3, April 2015):
[0269]
[0270] Based on the proposed model, this part of the grammar can be replaced, for example, by:
[0271]
[0272] scaling_poly_nb_param=0 means "default parameters". In that case, the following syntax elements are skipped. Otherwise, it defines the number of scaling_poly_param present in the syntax. Note that it can come from a list of predefined values. A lookup table can also be used to indicate the number of parameters, for example, the table nb_param[i][sym] can be known at both the encoder and the decoder, and the index i of the table is signaled in the bitstream. In one example, the number of parameters is limited to 1, 3, 4, 6, 8, 10, 11, 13, and 15.
[0273] scaling_poly_symmetric, if 1, indicates that scaling_poly_param should be repeated for x / y symmetric polynomial coefficients. If this feature is not desired, this flag can be removed. If removed, it should be inferred to be 0 in the following.
[0274] scaling_poly_shift selects s from a set of predefined values base For example, base=scaling_poly_shift offset, where the offset is chosen based on the normalization. For N=2, we recommend a range of 0 to 3, i.e., using s base =scale_poly_shift.
[0275] scaling_poly_clip defines the lower clipping value from a range of predefined values. It can take fewer bits than shown in the table, for example, it can use 2 or 3 instead of 4 bits. For example, this can be [1,4,8,16]. Note that the lower clipping value is usually no larger than 16 (a neutral value for 8-bit matrix coefficients used as quantized scaling factors like in HEVC).
[0276] scaling_poly_param is used to represent the polynomial coefficients for the selected matrix (scaling_poly_coef, which is the same as P in equations (3), (11), (12), etc. i match).
[0277] The size, order and format of the syntax elements may vary, and some of them may depend on scaling_poly_nb_param. For example, if scaling_poly_nb_param is 0 or 1, then scaling_poly_symmetric is not used, and the number of bits of scaling_poly_shift may increase with the number of parameters, etc. Furthermore, the syntax may vary, and the polynomial coefficients may be derived in an indirect way.
[0278] In HEVC, scaling lists are signaled in SPS (sequence parameter set) or PPS (picture parameter set). It should be noted that the present embodiment is not limited to transmitting quantization matrix information in SPS or PPS. For example, quantization matrix information can be transmitted in a parameter set dedicated to transmitting the matrix. In another example, quantization matrix information can be transmitted at the block level, and the number of possible parameters is variable.
[0279] The following algorithm can be used to replicate symmetric elements on the fly when needed:
[0280]
[0281]
[0282] With this approach, there is no need to save the symmetric flag, since all coefficients are derived on the fly. Note that when scaling_poly_symmetric is zero, this is a simple copy.
[0283] Note that unspecified parameters should have no effect: either scale_poly_coef should be initialized to zero, or n should be saved so that the matrix calculation can be terminated early.
[0284] This algorithm can be written in tabular form as follows:
[0285]
[0286] Residual
[0287] As described in [JCTVC-H0460], residuals can be added to further improve the fit to a given matrix shape. To optimize the encoding and decoding cost, the number of residuals should be variable, and their scanning order follows the order of low-frequency to high-frequency coefficients, such as the upper right diagonal (as in HEVC) or the zigzag shape (as in AVC), because low-frequency coefficients are often considered more critical. Unencoded residuals are inferred to be 0.
[0288] This can be encoded or decoded using the following syntax immediately after the polynomial definition:
[0289]
[0290] In another embodiment, the use of residuals may not be recommended due to complexity and memory considerations.
[0291] Matrix for QP offset
[0292] As in [JCTVC-I0284], the quantization matrix can be used as a QP offset (denoted as "QP-offset mode" or "log scale mode" because the QP represents the quantizer scale in a logarithmic scale instead of a scaling factor (denoted as "linear scale mode"), as it brings several advantages:
[0293] - Simpler dequantization by removing multiplications, which in turn reduces the computational dynamic range. This leaves room for improving the accuracy of levScale, which could allow levScale to be used as a normalizer for some non-square transforms. factor (using an additional QP offset), and completely remove multiplications except for levScale.
[0294] - Quantization is similarly simplified by removing the division of quantization matrix coefficients (usually implemented as multiplication with the inverse of the quantization matrix, with accuracy tradeoffs, memory impact, and even greater pressure on dynamic range than dequantization), leaving only multiplication by the inverse of levScale.
[0295] - It makes sense to unify the representation of all quantization adjustments
[0296] - Neutral value (flattened matrix) is zero
[0297] - Logarithmic scale naturally provides better accuracy for low frequency coefficients, which are more critical
[0298] -HEVC default matrix formulation uses exponentials; logarithmic scale representation should be easier to fit with polynomials. In general, fits are expected to be better in logarithmic scale, further reducing the need for residuals or large numbers of parameters or even clipping.
[0299] -Exponentiation becomes multiplication, which is much easier to implement
[0300] - Logarithmic scale representation is signed (e.g., from -128 to +127), and may not require the dynamic range expansion associated with clipping
[0301] - The increased QP granularity required for smooth matrix definition also provides better rate control
[0302] The disadvantage is that a finer QP granularity is required, which at least affects:
[0303] -levScale definition,
[0304] -Incremental-QP bit cost of slice header: will increase by only two bits
[0305] - Delta-QP bit cost per codec unit. As has been proposed in the past (e.g., [JCTVC-C135], D. Hoang, "Flexible scaling of quantization parameter", JCTVC-C135, 3rd JCT-VC Meeting: Guangzhou, China, October 7-15, 2010), this can be alleviated by introducing a delta-QP scale (in the PPS or slice header), which can actually reduce the delta-QP codec cost.
[0306] -Threshold table used by the deblocking filter
[0307] - Lagrange multipliers used in various places
[0308] Note that if the QP granularity is increased only for the quantization matrix, the impact can be limited to levScale, as described in [JCTVC-I0284].
[0309] We further recommend using a QP scale of 1 / 16 steps (as in [JCTVC-A114], J. Jung et al., "Description of video coding technology proposal by France Telecom, NTT, NTT DOCOMO, Panasonic and Technicolor", JCTVC-A114, First JCT-VC Meeting: Dresden, Germany, April 15-23, 2010), rather than 1 / 6 or 1 / 12 as in [JCTVC-I0284]. This further simplifies the quantization / dequantization operations by removing the division by 6 and modulo 6 operations, which are replaced by bit masks and shifts (no actual operations in hardware). This also simplifies the implementation of QP offsets.
[0310] Then, due to the increased accuracy, QP generally uses an additional two bits. An appropriate conversion formula can be defined to convert the QP value (which we call QP 6 ) into the new standard (QP 16 ). The quantization matrix bit depth can be reduced to 7 signed bits: the -64 to +63 range matches the 1 / 16 to 16 range defined by the linear scale matrix, with a 30% increase in resolution around neutral values. This in turn reduces the bit depth of the polynomial parameters to 7 bits, reducing the codec cost and reducing the matrix calculation dynamic range as discussed above.
[0311] The syntax will change to the following (scaling_poly_param is reduced to 7 bits):
[0312]
[0313] Fig.26 The results of the fit for the default HEVC intra matrix with 6 parameters are shown. The tests were performed without symmetry, shifting or shearing. The left figure shows the results of using the quantization matrix for linear scaling (51 bits for encoding the quantization matrix), and the right figure shows the results of using the quantization matrix for QP offset (46 bits for encoding the quantization matrix). Overall, in the various tests, the fit generally looks better in QP offset mode even when evaluated in the linear domain.
[0314] Default Parameters
[0315] Example parameters matching the default HEVC 8x8 intra matrix can be found below:
[0316] Linear scale mode:
[0317] clip=16, Sbase=0, sym=1
[0318] 5 parameter (8 bits) P i =[-53,94,-121,-29,36]
[0319] 7 parameter (8 bits) P i =[6,8,18,0,-17,1,21]
[0320] QP offset mode:
[0321] Clip=0, Sbase=0, sym=1
[0322] 4 parameters (7 bits) P i =[-2,-2,10,4]
[0323] 5 parameter (7 bits) P i =[-27,32,-26,-7,8]
[0324] 6 parameters (7 bits) P i =[-8,2,-3,11,3,-4]
[0325] 7 parameters (7 bits) P i =[1,-17,38,21,-16,-5,9]
[0326] 8 parameters (7 bits) P i =[-3,-1,6,3,4,1,6,-5]
[0327] 9 parameters (7 bits) P i =[0,-8,10,14,3,-7,6,-5,2]
[0328] Encoding and decoding the default HEVC intra 8x8 matrix with HEVC syntax will require 265 bits (1 bit for encoding and decoding the scaling_list_pred_mode_flag and 264 bits for encoding and decoding the scaling_list_delta_coef value).
[0329] For both linear scaling mode and QP offset mode, the codec bit cost and mean fit absolute error of the proposed parameters are analyzed as follows:
[0330] -4 bits for scaling_poly_nb_param,
[0331] -1 bit for scale_poly_symmetric,
[0332] -2 bits for scale_poly_shift,
[0333] -4 bits for scale_poly_clip, and
[0334] - 8 or 7 bits per parameter.
[0335] To enable error comparison, the error for the QP offset mode is calculated in the linear domain, i.e., the QP offset is first converted back to a scale factor. The LF error is the mean absolute error of the low-frequency half (x+y<8). As can be seen from Table 7, using 51 bits for the linear scale mode and 46 bits for the QP offset mode results in a small error in representing the quantization matrix. Therefore, the proposed method achieves good bit savings when encoding the quantization matrix without a large loss of accuracy, compared to the 265 bits used by the HEVC standard. In addition, as also shown in Table 7, the method can be easily scaled by using more polynomial parameters to improve the accuracy of representing the quantization matrix.
[0336] Table 7
[0337]
[0338] The polynomial coefficients can be obtained by doing a least squares fit to the existing 8x8 matrix and constraining the coefficient range and resulting matrix values. If relevant, then the lower clipping can be set manually.
[0339] In some embodiments, the polynomial may take other forms than those described above, or may be a combination of several sub-polynomials:
[0340] - x and y can be replaced by u and v, where u = (x + y) and v = (xy), or even u = (ax + by) and v = (bx - ay), where a + b = 1, and possibly a is transmitted. For a flat diagonal matrix, all coefficients in v will be zero, and for a symmetric matrix (with respect to the u axis), all coefficients with odd powers of v will be zero. However, these benefits may be difficult to exploit, and may require an additional parameter (a).
[0341] - Two separable polynomials can be combined to generate a matrix, such as M(x,y) = P 1 (x)·P 2 (y) or M(x,y)=P 1 (x)+P 2 (y), where x and y may be replaced by u and v as defined above, and P 1 and P 2 A different number of parameters are defined for each one.
[0342] The various methods described above can be used to modify, for example, Figure 1 and Figure 2 The quantization module and the dequantization module (130, 140, 240) of the encoder 100 and the decoder 200 shown in FIG. Moreover, the present embodiment is not limited to VVC or HEVC, and can be applied to other standards, recommendations and extensions thereof.
[0343] Various methods are described herein, and each method includes one or more steps or actions for implementing the described method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions can be modified or combined. Unless otherwise indicated or technically excluded, the various aspects described in this application can be used alone or in combination. Various numerical values are used in this application, for example, parameters used in the displacement of integer implementations and example polynomial parameters. Specific values are for example purposes, and the various aspects described are not limited to these specific values.
[0344] Fig. 27 A method (2700) for encoding video data according to an embodiment is illustrated. At step 2710, the encoder accesses a parameterized model based on a parameter sequence. At step 2720, the encoder determines a plurality of parameters to model a quantization matrix. At step 2730, the encoder associates each parameter of the plurality of parameters with a corresponding parameter of a subset of the parameter sequence to represent the quantization matrix. At step 2740, the encoder quantizes transform coefficients of a block of an image based on the quantization matrix. At step 2750, the encoder encodes the quantized transform coefficients, for example, using an entropy encoder.
[0345] Fig.28 A method (2800) for decoding video data according to an embodiment is illustrated. At step 2810, the decoder accesses a parameterized model based on a parameter sequence. At step 2820, the decoder determines a plurality of parameters to model a quantization matrix. At step 2830, the decoder associates each parameter of the plurality of parameters with a corresponding parameter of a subset of the parameter sequence to represent the quantization matrix. At step 2840, the decoder dequantizes transform coefficients of a block of an image based on the quantization matrix. At step 2850, the decoder reconstructs the block of the image in response to the dequantized transform coefficients.
[0346] Fig.29The block diagram of the example of the system in which various aspects and embodiments are implemented is illustrated.System 2900 can be implemented as the equipment including various components described below, and is configured to perform one or more aspects described in the present application.The example of such equipment includes but is not limited to various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, networked household appliances, and servers.The elements of system 2900 can be implemented in a single integrated circuit, multiple ICs and / or discrete components individually or in combination.For example, in at least one embodiment, the processing and encoder / decoder elements of system 2900 are distributed on multiple ICs and / or discrete components.In various embodiments, system 2900 is coupled to other systems or other electronic devices via, for example, a communication bus or by a dedicated input and / or output port communication.In various embodiments, system 2900 is configured to implement one or more aspects described in the present application.
[0347] System 2900 includes at least one processor 2910, which is configured to execute instructions loaded therein to implement various aspects described in the present application, for example. Processor 2910 may include embedded memory, input and output interfaces, and various other circuit systems known in the art. System 2900 includes at least one memory 2920 (e.g., volatile memory device, and / or non-volatile memory device). System 2900 includes storage device 2940, which may include non-volatile memory and / or volatile memory, including but not limited to EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, disk drive and / or optical drive. As a non-limiting example, storage device 2940 may include internal storage device, attached storage device and / or network accessible storage device.
[0348] The system 2900 includes an encoder / decoder module 2930, which is configured to, for example, process data to provide encoded video or decoded video, and the encoder / decoder module 2930 may include its own processor and memory. The encoder / decoder module 2930 represents a module (one or more) that can be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of the encoding and decoding modules. In addition, the encoder / decoder module 2930 may be implemented as a separate element of the system 2900, or may be combined in the processor 2910 as a combination of hardware and software, as known to those skilled in the art.
[0349] Program code to be loaded onto the processor 2910 or the encoder / decoder 2930 to perform various aspects described herein may be stored in the storage device 2940 and subsequently loaded onto the memory 2920 for execution by the processor 2910. According to various embodiments, during execution of the processes described herein, one or more of the processor 2910, the memory 2920, the storage device 2940, and the encoder / decoder module 2930 may store one or more of various items. Such stored items may include, but are not limited to, input video, decoded video or a portion of decoded video, bitstreams, matrices, variables, and intermediate or final results of equations, formulas, operations, and operation logic.
[0350] In several embodiments, memory within the processor 2910 and / or encoder / decoder module 2930 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be either the processor 2910 or the encoder / decoder module 2930) is used for one or more of these functions. The external memory may be a memory 2920 and / or a storage device 2940, such as a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, an external non-volatile flash memory is used to store the operating system of the television. In at least one embodiment, a fast external dynamic volatile memory (such as RAM) is used as working memory for video encoding and decoding operations, such as for MPEG-2, HEVC, or VVC (universal video codec).
[0351] As shown in block 2905, input to the elements of system 2900 may be provided through various input devices. Such input devices include, but are not limited to, (i) an RF section that receives an RF signal transmitted over the air, for example, by a broadcaster, (ii) a composite input terminal, (iii) a USB input terminal, and / or (iv) an HDMI input terminal.
[0352] In various embodiments, the input device of block 2905 has associated corresponding input processing elements, as known in the art. For example, the RF portion may be associated with elements suitable for: (i) selecting a desired frequency (also referred to as selecting a signal, or band limiting a signal within a frequency band), (ii) down-converting the selected signal, (iii) again band-limiting to a narrower frequency band to select a signal band that may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired data packet stream. The RF portion of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF portion may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or near baseband frequency) or baseband. In a set-top box embodiment, the RF part and its associated input processing element receive the RF signal transmitted on wired (for example, cable) medium, and filter to the frequency band of expectation again by filtering, down-conversion and perform frequency selection.Various embodiments rearrange the order of above-mentioned (and other) elements, remove some in these elements, and / or add other elements of similar or different functions.Adding element can include inserting element between existing element, for example, inserting amplifier and analog-to-digital converter.In various embodiments, the RF part comprises antenna.
[0353] In addition, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 2900 to other electronic devices across the USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented, for example, in a separate input processing IC or in the processor 2910 as needed. Similarly, various aspects of USB or HDMI interface processing may be implemented, for example, in a separate interface IC or in the processor 2910 as needed. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 2910, and an encoder / decoder 2930 operating in conjunction with memory and storage elements to process the data streams as needed for presentation on an output device.
[0354] The various elements of the system 2900 may be provided within an integrated housing. Within the integrated housing, the various elements may be interconnected and data may be transferred between them using a suitable connection arrangement 2915 (eg, an internal bus known in the art, including an I2C bus, wiring, and printed circuit boards).
[0355] The system 2900 includes a communication interface 2950 that enables communication with other devices via a communication channel 2990. The communication interface 2950 may include, but is not limited to, a transceiver configured to transmit and receive data through the communication channel 2990. The communication interface 2950 may include, but is not limited to, a modem or a network card, and the communication channel 2990 may be implemented, for example, within a wired and / or wireless medium.
[0356] In various embodiments, a Wi-Fi network such as IEEE 802.11 is used to stream data to the system 2900. The Wi-Fi signals of these embodiments are received on a communication channel 2990 and a communication interface 2950 suitable for Wi-Fi communication. The communication channel 2990 of these embodiments is typically connected to an access point or router that provides access to an external network including the Internet to allow streaming applications and other air communications. Other embodiments use a set-top box to provide streaming data to the system 2900, which delivers data via the HDMI connection of an input block 2905. Still other embodiments use the RF connection of an input block 2905 to provide streaming data to the system 2900.
[0357] The system 2900 can provide output signals to various output devices including a display 2965, a speaker 2975, and other peripherals 2985. In various examples of embodiments, the other peripherals 2985 include one or more of a stand-alone DVR, a disc player, a stereo system, a lighting system, and other devices that provide functions based on the output of the system 2900. In various embodiments, control signals are transmitted between the system 2900 and the display 2965, the speaker 2975, or other peripherals 2985 using signaling such as AV.Link, CEC, or other communication protocols that enable device-to-device control with or without user intervention. The output devices can be communicatively coupled to the system 2900 via dedicated connections through the respective interfaces 2960, 2970, and 2980. Alternatively, the output devices can be connected to the system 2900 using a communication channel 2990 via a communication interface 2950. In an electronic device (e.g., a television), the display 2965 and the speaker 2975 can be integrated into a single unit with the other components of the system 2900. In various embodiments, the display interface 2960 includes a display driver, such as a timing controller (T Con) chip.
[0358] For example, if the RF portion of input 2905 is part of a separate set-top box, the display 2965 and speaker 2975 may alternatively be separate from one or more of the other components. In various embodiments where the display 2965 and speaker 2975 are external components, the output signals may be provided via dedicated output connections including, for example, an HDMI port, a USB port, or a COMP output.
[0359] According to an embodiment, a method for video decoding is provided, comprising: accessing a parameterized model based on a parameter sequence; determining a plurality of parameters corresponding to a subset of the parameter sequence; associating each of the plurality of parameters with a corresponding parameter of the subset of the parameter sequence to represent a quantization matrix; dequantizing transform coefficients of a block of an image based on the quantization matrix; and reconstructing the block of the image in response to the dequantized transform coefficients.
[0360] According to another embodiment, a method for video encoding is provided, comprising: accessing a parameterized model based on a parameter sequence; determining a plurality of parameters corresponding to a subset of the parameter sequence; associating each of the plurality of parameters with a corresponding parameter of the subset of the parameter sequence to represent a quantization matrix; quantizing transform coefficients of a block of an image based on the quantization matrix; and entropy encoding the quantized transform coefficients.
[0361] According to another embodiment, a device for video decoding is provided, comprising one or more processors, wherein the one or more processors are configured to: access a parameterized model based on a parameter sequence; determine a plurality of parameters corresponding to a subset of the parameter sequence; associate each parameter of the plurality of parameters with a corresponding parameter of the subset of the parameter sequence to represent a quantization matrix; dequantize transform coefficients of a block of an image based on the quantization matrix; and reconstruct the block of the image in response to the dequantized transform coefficients. The device may also include one or more memories coupled to the one or more processors.
[0362] According to another embodiment, a device for video encoding is provided, comprising one or more processors, wherein the one or more processors are configured to: access a parameterized model based on a parameter sequence; determine a plurality of parameters corresponding to a subset of the parameter sequence; associate each of the plurality of parameters with a corresponding parameter of the subset of the parameter sequence to represent a quantization matrix; quantize transform coefficients of a block of an image based on the quantization matrix; and entropy encode the quantized transform coefficients. The device may also include one or more memories coupled to the one or more processors.
[0363] According to another embodiment, a signal is formatted to include: a value indicating the number of parameters in a plurality of parameters; the plurality of parameters, wherein each parameter in the plurality of parameters is associated with a corresponding parameter of a subset of a parameter sequence to represent a quantization matrix, wherein a parameterized model is based on the parameter sequence; and transform coefficients of a block of an image quantized based on the quantization matrix.
[0364] According to an embodiment, a value is encoded or decoded to indicate the number of parameters in the plurality of parameters.In general, as the number of parameters increases, the quantization matrix becomes more complex.
[0365] According to one embodiment, the parameterized model corresponds to a polynomial (x, y), where x and y indicate matrix coordinates, the parameter sequence corresponds to an ordered sequence of polynomial coefficients, and the polynomial is defined by the ordered sequence of polynomial coefficients and an ordered sequence of terms.
[0366] According to one embodiment, said ordered sequence of polynomial coefficients is symmetric in x and y.
[0367] According to one embodiment, a set of polynomial coefficients is decoded from a bitstream, wherein said ordered sequence of polynomial coefficients is determined in response to said decoded set of polynomial coefficients.
[0368] According to one embodiment, a value indicating the degree of the polynomial is encoded or decoded, wherein the value indicating the number of parameters is obtained from the value indicating the degree. In one example, the degree of the polynomial is set to four.
[0369] According to one embodiment, there is a one-to-one correspondence between the ordered sequence of polynomial coefficients and the ordered sequence of terms.
[0370] According to one embodiment, the polynomial is expressed as
[0371]
[0372] According to one embodiment, except for the subset of parameters, one or more parameters in the sequence of parameters are set to respective default values. In one example, the default value is set to zero.
[0373] According to one embodiment, the polynomial is a fully expanded polynomial.
[0374] According to one embodiment, said ordered sequence of terms is sorted by increasing degrees of the terms.
[0375] According to one embodiment, said ordered sequence of items is further sorted by the maximum values of increasing x and y indices.
[0376] According to one embodiment, said ordered sequence of items is further sorted by increasing x or y index.
[0377] According to one embodiment, the k-th parameter of said plurality of parameters is associated with the k-th item of said ordered sequence of items.
[0378] According to one embodiment, the matrix coordinates are normalized. Perform normalization, where x′ and y′ are the columns and rows of the quantization matrix, and size x is the horizontal size of the matrix, size y is the vertical dimension of the matrix, and N is a normalized value. In one example, N is set to 2.
[0379] According to one embodiment, the quantization matrix is used for quantization step size scaling or for quantizer parameter offset.
[0380] According to one embodiment, the matrix coefficients are calculated by an integer process that includes limiting the left-shifted polynomial terms to n bits by discarding the most significant bits (including the sign bit), then adding them using an adder that is also limited to n-bit output (still by discarding the MSB), and right-shifting the result by smax-sbase values to obtain the matrix coefficients, n being equal to the matrix coefficient bit depth (mbd) plus the maximum allowed value of (smax-sbase).
[0381] According to one embodiment, when the matrix coefficients are further clipped to a minimum value, the bit depth before clipping may be increased by one to keep the sign bit, and in that case, the n is increased by one.
[0382] According to one embodiment, the polynomial (x, y) corresponds to a polynomial denoted by x and a polynomial denoted by y.
[0383] According to one embodiment, the variables x and y are replaced by u and v, where u = (x + y) and v = (xy).
[0384] According to one embodiment, the variables x and y are replaced by u and v, where u=(ax+by) and v=(bx-ay), and a+b=1.
[0385] According to one embodiment, two separable polynomials are combined to generate a quantization matrix, such as M(x, y) = P 1 (x)·P 2 (y) or M(x,y)=P 1 (x)+P 2 (y).
[0386] An embodiment provides a computer program including instructions, which, when executed by one or more processors, causes one or more processors to perform an encoding method or a decoding method according to any of the above embodiments. One or more of the given embodiments also provide a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to the above method. One or more embodiments also provide a computer-readable storage medium having stored thereon a bitstream generated according to the above method. One or more embodiments also provide a method and apparatus for sending or receiving a bitstream generated according to the above method.
[0387] Various embodiments relate to decoding. As used in this application, "decoding" may encompass, for example, all or part of the processing performed on a received encoded sequence to produce a final output suitable for display. In various embodiments, such processing includes one or more of the processing typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. Based on the context of the particular description, whether the phrase "decoding processing" is intended to specifically refer to a subset of operations or to refer broadly to a broader decoding process will be apparent and will be well understood by those skilled in the art.
[0388] Various embodiments relate to encoding.In a similar manner to the discussion above regarding "decoding", as used in this application, "encoding" may encompass all or part of the processing performed on an input video sequence to produce an encoded bitstream, for example.
[0389] The embodiments and aspects described herein can be implemented, for example, in methods or processes, devices, software programs, data streams or signals. Even if only discussed in the context of a single form of embodiment (e.g., discussed only as a method), the embodiments of the features discussed can also be implemented in other forms (e.g., devices or programs). The device can be implemented, for example, with appropriate hardware, software and firmware. The method can be implemented in, for example, a device, such as a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit or a programmable logic device. The processor also includes communication equipment, such as a computer, a mobile phone, a portable / personal digital assistant ("PDA") and other equipment that facilitates information communication between end users.
[0390] References to "one embodiment" or "an embodiment" or "one implementation" or "an implementation" and other variations thereof mean that a particular feature, structure, characteristic, etc. described in conjunction with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" and any other variations throughout this application are not necessarily all referring to the same embodiment.
[0391] Additionally, this application may refer to "determining" various information. Determining information may include, for example, one or more of: estimating information, calculating information, predicting information, or retrieving information from a memory.
[0392] Additionally, this application may refer to "accessing" various information. Accessing information may include, for example, one or more of: receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.
[0393] Additionally, this application may refer to "receiving" various information. Like "accessing," receiving is a broad term. Receiving information may include one or more of: for example, accessing information or retrieving information (e.g., from a memory). Additionally, in one way or another, during operations such as: storing information, processing information, sending information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information, "receiving" is generally involved.
[0394] It should be appreciated that use of any of the following “ / ”, “and / or”, and “at least one of”, for example, in the context of “A / B”, “A and / or B”, and “at least one of A and B” is intended to encompass selection of only the first listed option (A), or only the second listed option (B), or both options (A and B). As a further example, in the context of “A, B, and / or C” and “at least one of A, B, and C”, such wording is intended to encompass selection of only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A and B and C). This may be extended to multiple items listed, as will be apparent to one of ordinary skill in this and related arts.
[0395] As will be apparent to one of ordinary skill in the art, embodiments may generate various signals formatted to carry information that may be stored or transmitted, for example. The information may include, for example, instructions for executing a method or data generated by one of the described embodiments. For example, a signal may be formatted to carry a bit stream of the described embodiments. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor readable medium.
Claims
1. A method for video decoding, comprising: decoding at least one syntax element representing a value N indicating a number of first polynomial coefficients of a single fixed polynomial model, N being selected from a plurality of values; Obtain N polynomial coefficients; forming a polynomial function having N monomials, the polynomial function being a function of (x, y), wherein x and y indicate matrix coordinates, wherein the N monomials are ordered in increasing complexity, wherein each of the N polynomial coefficients is associated with a corresponding monomial of the N monomials to represent the polynomial function; obtaining a quantization matrix based on the polynomial function and the matrix size, wherein the quantization matrix coefficients at the matrix coordinates (x, y) are obtained from the polynomial function of (x, y); as well as Blocks of an image are decoded based on the quantization matrix.
2. The method of claim 1, wherein the N monomials represent a subset of an ordered sequence of monomials.
3. The method of claim 1, wherein a k-th polynomial coefficient of the N polynomial coefficients is associated with a k-th monomial of the N monomials. The method of claim 1 , wherein matrix elements of the quantization matrix are clipped to clipping values.
5. The method of claim 1, further comprising: A set of polynomial coefficients is decoded from the bitstream, wherein the N polynomial coefficients are determined in response to the decoded set of polynomial coefficients.
6. A device for video decoding, comprising: One or more processors, wherein the one or more processors are configured to: decoding at least one syntax element representing a value N indicating a number of first polynomial coefficients of a single fixed polynomial model, N being selected from a plurality of values; Obtain N polynomial coefficients; forming a polynomial function having N monomials, the polynomial function being a function of (x, y), wherein x and y indicate matrix coordinates, wherein the N monomials are ordered in increasing complexity, wherein each of the N polynomial coefficients is associated with a corresponding monomial of the N monomials to represent the polynomial function; obtaining a quantization matrix based on the polynomial function and the matrix size, wherein the quantization matrix coefficients at the matrix coordinates (x, y) are obtained from the polynomial function of (x, y); as well as Blocks of an image are decoded based on the quantization matrix.
7. The apparatus of claim 6, wherein the N monomials represent a subset of an ordered sequence of monomials.
8. The apparatus of claim 6, wherein a k-th polynomial coefficient of the N polynomial coefficients is associated with a k-th monomial of the N monomials.
9. The apparatus of claim 6, wherein matrix elements of the quantization matrix are clipped to a clipping value.
10. The apparatus of claim 6, wherein the one or more processors are further configured to: A set of polynomial coefficients is decoded from the bitstream, wherein the N polynomial coefficients are determined in response to the decoded set of polynomial coefficients.
11. A video encoding method, comprising: determining a value N indicating a number of first polynomial coefficients of the single fixed polynomial model, N being selected from a plurality of values; encoding at least one syntax element representing said value N, Obtain N polynomial coefficients; forming a polynomial function having N monomials, the polynomial function being a function of (x, y), wherein the N monomials are ordered in increasing complexity, wherein each of the N polynomial coefficients is associated with a corresponding monomial of the N monomials to represent the polynomial function; obtaining a quantization matrix based on the polynomial function and the matrix size, wherein the quantization matrix coefficients at the matrix coordinates (x, y) are obtained from the polynomial function of (x, y); quantizing transform coefficients of a block of an image based on the quantization matrix; as well as The quantized transform coefficients are entropy encoded.
12. The method of claim 11, wherein the N monomials represent a subset of an ordered sequence of monomials.
13. The method of claim 11, wherein a k-th polynomial coefficient of the N polynomial coefficients is associated with a k-th monomial of the N monomials.
14. The method of claim 11, wherein matrix elements of the quantization matrix are clipped to clipping values.
15. The method of claim 11, further comprising: A set of polynomial coefficients is encoded in a bitstream, wherein the N polynomial coefficients are determined responsive to the set of polynomial coefficients.
16. An apparatus for video encoding, comprising: One or more processors, wherein the one or more processors are configured to: determining a value N indicating a number of first polynomial coefficients of the single fixed polynomial model, N being selected from a plurality of values; encoding at least one syntax element representing said value N, Obtain N polynomial coefficients; forming a polynomial function having N monomials, the polynomial function being a function of (x, y), wherein the N monomials are ordered in increasing complexity, wherein each of the N polynomial coefficients is associated with a corresponding monomial of the N monomials to represent the polynomial function; obtaining a quantization matrix based on the polynomial function and the matrix size, wherein the quantization matrix coefficients at the matrix coordinates (x, y) are obtained from the polynomial function of (x, y); quantizing transform coefficients of a block of an image based on the quantization matrix; as well as The quantized transform coefficients are entropy encoded.
17. The apparatus of claim 16, wherein the N monomials represent a subset of an ordered sequence of monomials.
18. The apparatus of claim 16, wherein a k-th polynomial coefficient of the N polynomial coefficients is associated with a k-th monomial of the N monomials.
19. The apparatus of claim 16, wherein matrix elements of the quantization matrix are clipped to a clipping value.
20. The apparatus of claim 16, wherein the one or more processors are further configured to: A set of polynomial coefficients is encoded in a bitstream, wherein the N polynomial coefficients are determined responsive to the set of polynomial coefficients.
