Single-index quantization matrix design for video encoding and decoding

Abstract

Different quantization matrices may be transmitted corresponding to different block sizes, color components, and prediction modes. In order to more efficiently signal the coefficients of the quantization matrix, in one implementation, a unified matrix identifier matrixId is used based on the size identifier (sizeId) associated with the CU size of the larger size listed first and the matrix type (matrixTypeId) of the luma QM listed first. For example, the unified identifier is derived as: matrixId=N*sizeId+matrixTypeId, where N is the number of possible type identifiers, for example, N=6. This single identifier allows reference to any previously transmitted matrix when prediction (copying) is used, and transmitting larger matrices first avoids interpolation during the prediction process. When a block uses intra-block copy prediction mode, a QM identifier can be obtained, just as if the block uses inter-prediction mode.

Description

Technical Field

[0001] This embodiment mainly relates to a quantization matrix design method and device in video encoding or decoding. Background Art

[0002] To achieve high compression efficiency, image and video coding schemes typically employ prediction and transforms to exploit spatial and temporal redundancy in video content. Typically, intra-frame or inter-frame prediction is used to exploit intra-frame or inter-frame picture correlations. The difference between the original block and the predicted block, typically represented as a prediction error or prediction residual, is then transformed, quantized, and entropy decoded. To reconstruct the video, the compressed data is decoded using the inverse processes of entropy decoding, quantization, transform, and prediction. Summary of the Invention

[0003] According to one embodiment, a video decoding method is provided, comprising: obtaining a single identifier of a quantization matrix based on a block size, a color component, and a prediction mode of a block to be decoded in a picture; decoding a syntax element indicating a reference quantization matrix, wherein the syntax element specifies a difference between an identifier of the reference quantization matrix and the obtained identifier of the quantization matrix; obtaining the quantization matrix based on the reference quantization matrix; dequantizing transform coefficients of the block in response to the quantization matrix; and decoding the block in response to the dequantized transform coefficients.

[0004] According to another embodiment, a method for video encoding is provided, comprising: accessing a block to be encoded in a picture; accessing a quantization matrix of the block; obtaining a single identifier of the quantization matrix based on the block size, color components and prediction mode of the block; encoding a syntax element indicating a reference quantization matrix, wherein the syntax element specifies a difference between an identifier of the reference quantization matrix and the obtained identifier of the quantization matrix; quantizing transform coefficients of the block in response to the quantization matrix; and entropy encoding the quantized transform coefficients.

[0005] According to another embodiment, a device for video decoding is provided, comprising one or more processors, wherein the one or more processors are configured to: obtain a single identifier for a quantization matrix based on a block size, color component, and prediction mode of a block to be decoded in a picture; decode a syntax element indicating a reference quantization matrix, wherein the syntax element specifies a difference between an identifier of the reference quantization matrix and the obtained identifier of the quantization matrix; obtain the quantization matrix based on the reference quantization matrix; dequantize transform coefficients of the block in response to the quantization matrix; and decode the block in response to the dequantized transform coefficients.

[0006] According to another embodiment, a device for video encoding is provided, which includes one or more processors, wherein the one or more processors are configured to: access a block to be encoded in a picture; access a quantization matrix of the block; obtain a single identifier of the quantization matrix based on the block size, color components and prediction mode of the block; encode a syntax element indicating a reference quantization matrix, wherein the syntax element specifies a difference between an identifier of the reference quantization matrix and the obtained identifier of the quantization matrix; quantize transform coefficients of the block in response to the quantization matrix; and entropy encode the quantized transform coefficients.

[0007] According to another embodiment, a device for video decoding is provided, comprising: a device for obtaining a single identifier of a quantization matrix based on a block size, color components, and prediction mode of a block to be decoded in a picture; a device for decoding a syntax element indicating a reference quantization matrix, wherein the syntax element specifies a difference between an identifier of the reference quantization matrix and the obtained identifier of the quantization matrix; a device for obtaining the quantization matrix based on the reference quantization matrix; a device for dequantizing transform coefficients of the block in response to the quantization matrix; and a device for decoding the block in response to the dequantized transform coefficients.

[0008] According to another embodiment, a video encoding apparatus is provided, comprising: a device for accessing a block to be encoded in a picture; a device for accessing a quantization matrix of the block; a device for obtaining a single identifier of the quantization matrix based on the block size, color components and prediction mode of the block; a device for encoding a syntax element indicating a reference quantization matrix, wherein the syntax element specifies a difference between an identifier of the reference quantization matrix and the obtained identifier of the quantization matrix; a device for quantizing transform coefficients of the block in response to the quantization matrix; and a device for entropy encoding the quantized transform coefficients.

[0009] One or more embodiments further provide a computer program comprising instructions, which, when executed by one or more processors, cause the one or more processors to perform an encoding method or a decoding method according to any of the above-described embodiments. One or more embodiments of the present invention further provide a computer-readable storage medium on which instructions for encoding or decoding video data according to the method described above are stored. One or more embodiments further provide a computer-readable storage medium on which a bitstream generated according to the above-described method is stored. One or more embodiments further provide a method and apparatus for transmitting or receiving a bitstream generated according to the above-described method. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1A block diagram of a system is shown in which aspects of the present embodiments may be implemented.

[0011] Figure 2 A block diagram of an embodiment of a video encoder is shown.

[0012] Figure 3 A block diagram of an embodiment of a video decoder is shown.

[0013] Figure 4 It is shown that transform coefficients are extrapolated to zero for block sizes greater than 32 in VVC draft 5.

[0014] Figure 5 The fixed prediction tree described in JCTVC-H0314 is shown.

[0015] Figure 6 Prediction (decimation) from a larger size is shown according to an embodiment.

[0016] Figure 7 Combining prediction from a larger size and decimation of rectangular blocks according to an embodiment is shown.

[0017] Figure 8 The QM derivation process for rectangular blocks of chroma according to an embodiment is shown.

[0018] Figure 9 The QM derivation process for rectangular blocks of chroma (adapted to 4:2:2 format) according to an embodiment is shown.

[0019] Figure 10 The QM derivation process for rectangular blocks of chroma (adapted to the 4:4:4 format) is shown.

[0020] Figure 11 A flow chart for parsing a zoom list data syntax structure according to an embodiment is shown.

[0021] Figure 12 A flow chart for encoding zoom list data syntax structure according to an embodiment is shown.

[0022] Figure 13 A flow chart for a QM derivation process according to an embodiment is shown. DETAILED DESCRIPTION

[0023] Figure 1A block diagram of an example of a system in which various aspects and embodiments can be implemented is shown. System 100 can be implemented as a device including the various components described below, and is configured to perform one or more aspects described in this application. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances and servers. The elements of system 100 can be implemented individually or in combination in a single integrated circuit, multiple ICs and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 100 are distributed over multiple ICs and / or discrete components. In various embodiments, system 100 is communicatively coupled to other systems or other electronic devices via, for example, a communication bus or by dedicated input and / or output ports. In various embodiments, system 100 is configured to implement one or more aspects described in this application.

[0024] The system 100 includes at least one processor 110 configured to execute instructions loaded therein, for implementing various aspects described in the present application, for example. The processor 110 may include embedded memory, input / output interfaces, and various other circuits known in the art. The system 100 includes at least one memory 120 (e.g., a volatile memory device and / or a non-volatile memory device). The system 100 includes a storage device 140, which may include non-volatile memory and / or volatile memory, including but not limited to EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, magnetic disk drive, and / or optical disk drive. As non-limiting examples, the storage device 140 may include an internal storage device, an attached storage device, and / or a network-accessible storage device.

[0025] System 100 includes an encoder / decoder module 130, which is configured to process data to provide encoded video or decoded video, for example, and may include its own processor and memory. Encoder / decoder module 130 represents one or more modules that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both encoding and decoding modules. In addition, encoder / decoder module 130 may be implemented as a separate element of system 100 or may be incorporated into processor 110 as a combination of hardware and software as known to those skilled in the art.

[0026] Program code to be loaded onto the processor 110 or the encoder / decoder 130 to perform various aspects described herein may be stored in the storage device 140 and subsequently loaded onto the memory 120 for execution by the processor 110. According to various embodiments, one or more of the processor 110, the memory 120, the storage device 140, and the encoder / decoder module 130 may store one or more of various items during execution of the processes described herein. These stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from processing of equations, formulas, operations, and operational logic.

[0027] In several embodiments, memory internal to the processor 110 and / or encoder / decoder module 130 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 110 or the encoder / decoder module 130) is used for one or more of these functions. The external memory may be memory 120 and / or a storage device 140, such as dynamic volatile memory and / or non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store the operating system of the television. In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as working memory for video encoding and decoding operations such as MPEG-2, HEVC, or VVC.

[0028] Input to the elements of system 100 may be provided through various input devices, as shown in block 105. Such input devices include, but are not limited to, (i) an RF section that receives an RF signal transmitted over the air, for example, by a broadcaster, (ii) a composite input terminal, (iii) a USB input terminal, and / or (iv) an HDMI input terminal.

[0029] In various embodiments, the input device of block 105 has associated corresponding input processing elements known in the art. For example, the RF section may be associated with elements adapted to: (i) select a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a frequency band), (ii) down-convert the selected signal, (iii) again band-limit the signal to a narrower frequency band to select a signal frequency band, which in some embodiments may be referred to as a channel, (iv) demodulate the down-converted and band-limited signal, (v) perform error correction, and (vi) demultiplex to select a desired data packet stream. The RF section of various embodiments includes one or more elements to perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or near-baseband frequency) or to baseband. In a set-top box embodiment, the RF part and its relevant input processing element receive the RF signal that transmits by wired (for example, cable) medium, and by filtering, down-conversion and filtering to the frequency band of expectation again to perform frequency selection.Various embodiments rearrange the order of above-mentioned (and other) element, remove some in these elements, and / or add other element that performs similar or different functions.Adding element can be included in and inserts element between existing element, for example, inserts amplifier and analog to digital converter.In various embodiments, the RF part comprises antenna.

[0030] In addition, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 100 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, may be implemented, for example, in a separate input processing IC or in the processor 110, as desired. Similarly, various aspects of USB or HDMI interface processing may be implemented, for example, in a separate interface IC or in the processor 110, as desired. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 110 and the encoder / decoder 130, which operate in conjunction with memory and storage elements to process the data streams as needed for presentation on an output device.

[0031] The various components of the system 100 may be disposed within an integrated housing, and the various components may be interconnected and transmit data therebetween using a suitable connection arrangement 115 such as an internal bus as is known in the art, including an I2C bus, wiring, and printed circuit boards.

[0032] System 100 includes a communication interface 150 that enables communication with other devices via a communication channel 190. Communication interface 150 may include, but is not limited to, a transceiver configured to transmit and receive data over communication channel 190. Communication interface 150 may include, but is not limited to, a modem or a network card, and communication channel 190 may be implemented, for example, within a wired and / or wireless medium.

[0033] In various embodiments, data is streamed to system 100 using a Wi-Fi network such as IEEE 802.11. Wi-Fi signals for these embodiments are received via communication channel 190 and communication interface 150 suitable for Wi-Fi communication. Communication channel 190 for these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications. Other embodiments provide streaming data to system 100 using a set-top box that passes data through an HDMI connection to input box 105. Still other embodiments provide streaming data to system 100 using an RF connection to input box 105.

[0034] System 100 can provide output signals to various output devices, including a display 165, speakers 175, and other peripherals 185. In various embodiments, other peripherals 185 include one or more of a stand-alone DVR, a disc player, a stereo system, a lighting system, and other devices that provide functionality based on the output of system 100. In various embodiments, control signals are transmitted between system 100 and display 165, speakers 175, or other peripherals 185 using signaling such as AV Link, CEC, or other communication protocols that enable device-to-device control with or without user intervention. Output devices can be communicatively coupled to system 100 via dedicated connections through respective interfaces 160, 170, and 180. Alternatively, output devices can be connected to system 100 via communication interface 150 using communication channel 190. Display 165 and speakers 175 can be integrated into a single unit in an electronic device (e.g., a television) along with other components of system 100. In various embodiments, display interface 160 includes a display driver, such as a timing controller (TCon) chip.

[0035] For example, if the RF portion of input 105 is part of a separate set-top box, the display 165 and speaker 175 may alternatively be separate from one or more of the other components. In various embodiments where the display 165 and speaker 175 are external components, the output signals may be provided via dedicated output connections, including, for example, an HDMI port, a USB port, or a COMP output.

[0036] Figure 2An example video encoder 200 is shown, such as a High Efficiency Video Coding (HEVC) encoder. Figure 2 Also shown are encoders in which improvements are made to the HEVC standard or encoders that employ techniques similar to HEVC, such as the VVC (Versatile Video Coding) encoder under development by the JVCT (Joint Video Discovery Team).

[0037] In this application, the terms "reconstruction" and "decoding" are used interchangeably, the terms "encoding" and "decoding" are used interchangeably, and the terms "image," "picture," and "frame" are used interchangeably. Typically, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side.

[0038] Before being encoded, the video sequence may undergo pre-encoding processing (201), for example, applying a color transform to the input color picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of the input picture components in order to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and appended to the bitstream.

[0039] In encoder 200, a picture is encoded by encoder elements as described below. The picture to be encoded is divided (202) and processed in units such as CUs. Each unit is encoded using, for example, intra or inter mode. When the unit is encoded in intra mode, it performs intra prediction (260). In inter mode, motion estimation (275) and compensation (270) are performed. The encoder decides (205) whether to use intra mode or inter mode to encode the unit, and indicates the intra / inter decision by, for example, a prediction mode flag. For example, a prediction residual is calculated by subtracting (210) the predicted block from the original image block.

[0040] The prediction residual is then transformed (225) and quantized (230). The quantized transform coefficients, along with motion vectors and other syntax elements, are entropy coded (245) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass the transform and quantization, i.e., directly decode the residual without applying a transform or quantization process.

[0041] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual is combined with the prediction block (255) to reconstruct the image block. An in-loop filter (265) is applied to the reconstructed image to perform, for example, deblocking / SAO (sample adaptive offset) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (280).

[0042] Figure 3 A block diagram of an example video decoder 300 is shown. In the decoder 300, a bitstream is decoded by decoder elements as described below. The video decoder 300 generally performs the same Figure 2 The encoder 200 typically also performs video decoding as part of encoding the video data, as well as a decoding pass that is the inverse of the encoding pass described in

[0045] .

[0043] In particular, the input to the decoder includes a video bitstream, which may be generated by the video encoder 200. The bitstream is first entropy decoded (330) to obtain transform coefficients, motion vectors, and other decoding information. Picture segmentation information indicates how the picture is segmented. The decoder can therefore divide (335) the picture according to the decoded picture segmentation information. The transform coefficients are dequantized (340) and inverse transformed (350) to decode the prediction residual. The decoded prediction residual is combined (355) with the prediction block to reconstruct the image block. The prediction block can be obtained (370) from intra-frame prediction (360) or motion compensated prediction (i.e., inter-frame prediction) (375). An in-loop filter (365) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (380).

[0044] The decoded image may be further subjected to post-decoding processing (385), such as an inverse color transform (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or performing an inverse remapping of the remapping process performed in the pre-encoding process (201). The post-decoding process may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0045] The HEVC specification allows the use of a quantization matrix in the dequantization process, where the transform coefficients are scaled by the current quantization step size and further scaled by the quantization matrix (QM), as follows:

[0046] d[x][y]=Clip3(coeffMin,coeffMax,((TransCoeffLevel[xTbY][yTbY][cIdx][x][y]*m[x][y]*levelScale[qP%6]<<(qP / 6))+(1<<(bdShift-1)))>>bdShift)

[0047] in:

[0048] TransCoeffflevel[…] is the absolute value of the transform coefficient of the current block identified by the spatial coordinates xTbY, yTbY and its component index cIdx of the current block.

[0049] x and y are horizontal / vertical frequency indices.

[0050] qP is the current quantization parameter.

[0051] Multiplying by levelScale[qP%6] and left shifting by (qP / 6) is equivalent to multiplying by the quantization step size qStep=(levelScale[qP%6]<<(qP / 6)).

[0052] m[…][…] is a two-dimensional quantization matrix. Here, since the quantization matrix is ​​used for scaling, it can also be called a scaling matrix.

[0053] bdShift is an additional scaling factor that takes into account the image sampling bit depth. The term 1<<(bdShift-1)) is used for rounding to the nearest integer purpose.

[0054] ·d[…] is the resulting absolute value of the dequantized transform coefficient.

[0055] The syntax used by HEVC to transmit the quantization matrix is ​​described as follows:

[0056]

[0057] It can be noticed

[0058] • Specify a different matrix for each transform size (sizeId). In the scaling list data syntax structure, the scaling matrix is ​​scanned into a 1-D scaling list (eg, ScalingList).

[0059] • For a given transform size, six matrices are specified for intra / inter decoding and Y / Cb / Cr components.

[0060] The matrix can be any of the following

[0061] o If scaling_list_pred_mode_flag is zero (reference matrixId is obtained as matrixId-scaling_list_pred_matrix_id_delta), the matrix is ​​copied from the previously transmitted matrix of the same size

[0062] o Copy the matrix according to the default values ​​specified in the standard (if both scaling_list_pred_mode_flag and scaling_list_pred_matrix_id_delta are zero)

[0063] o Use exp-Golomb entropy coding, fully specified in DPCM coding mode with top-right diagonal scanning order.

[0064] • For block sizes larger than 8x8, only the 8x8 coefficients are transmitted for signaling the quantization matrix in order to save decoding bits. The coefficients are then interpolated using zero-holding (ie, repetition), except for the DC coefficient which is explicitly transmitted.

[0065] In VVC draft 5, the use of quantization matrices similar to HEVC has been adopted based on the contributions of JVET-N0847 (see O. Chubach et al., "CE7-related: Support of quantization matrices for VVC", JVET-N0847, Geneva, CH, March 2019). The scaling_list_data syntax has been adapted to the VVC codec as shown below.

[0066]

[0067] In the design of VVC draft 5 adopting JVET-N0847, as in HEVC, QM is identified by two parameters: matrixId and sizeId. This is shown in the following two tables.

[0068] Table 1: Block size identifiers (JVET-N0847)

[0069] brightness Chroma sizeId - - 0 - 2x2 1 4x4 4x4 2 8x8 8x8 3 16x16 16x16 4 32x32 32x32 5 64x64 - 6

[0070] Table 2: QM Type Identifiers (JVET-N0847)

[0071] CuPredMode cIdx (color component) matrixId MODE_INTRA 0(Y) 0 MODE_INTRA 1(Cb) 1 MODE_INTRA 2(Cr) 2 MODE_INTER 0(Y) 3 MODE_INTER 1(Cb) 4 MODE_INTER 2(Cr) 5

[0072] Note: MODE_INTRA QM is also used for MODE_IBC (Intra Block Copy).

[0073] The combinations of the two identifiers are shown in the following table:

[0074] Table 3: (MatrixId, SizeId) Combinations (JVET-N0847)

[0075]

[0076] As in HEVC, for block sizes larger than 8×8, only the 8×8 coefficients and the DC coefficient are transmitted. Zero-preserving interpolation is used to reconstruct the QM of the correct size. For example, for a 16×16 block, each coefficient is repeated twice in both directions, and then the DC coefficient is replaced by the transmitted coefficient.

[0077] For rectangular blocks, the reserved size (sizeId) selected for the QM is the larger size, that is, the maximum of the width and height. For example, for a 4×16 block, a QM with a block size of 16×16 is selected. The reconstructed 16×16 matrix is ​​then vertically decimated by a factor of 4 to obtain the final 4×16 quantization matrix (i.e., 3 of the 4 rows are skipped).

[0078] For the following, we refer to the QM for a given set of block sizes (square or rectangular) as size-N, which is related to the sizeId and the square block size it is used for. For example, for block sizes 16x16 or 16x4, the QM is identified as size-16 (sizeId 4 in VVC draft 5). The sign of size-N is used to distinguish the exact block shape, as well as the number of QM coefficients signaled (restricted to 8×8, as shown in Table 3).

[0079] Furthermore, in VVC draft 5, for the case of size -64, the QM coefficients of the lower right quadrant are not transmitted (they are inferred to be 0, hereinafter referred to as "zero output"). This is achieved by the "x>=4&y>=4" condition in the scaling_list_data syntax. This avoids transmitting QM coefficients that are never used by the transform / quantization process. In fact, in VVC, for transform block sizes greater than 32 in any dimension (64xN, Nx64, where N<=64), any transform coefficient with an x / y frequency coordinate greater than or equal to 32 is not transmitted and is inferred to be zero, so no quantization matrix coefficient is required to quantize it. This is in Figure 4 , where the shaded areas correspond to transform coefficients that are inferred to be zero.

[0080] Compared to HEVC, VVC requires more quantization matrices due to the larger number of block sizes. However, in VVC draft 5, QM prediction is still limited to duplication of matrices of the same block size, which can lead to bit waste. In addition, since VVC only uses block size 2x2 for chroma and block size 64x64 for luma, the syntax associated with QMs is more complex. In addition, as in HEVC, JVET-N0847 describes a specific matrix derivation process for each block size.

[0081] During HEVC standardization, some QM prediction techniques have been explored, for example, in JCTVC-E073 (see J. Tanaka et al., “Quantization Matrix for HEVC”, JCTVC-E073, Geneva, CH, March 2011) and JCTVC-H0314 (see Y. Wang et al., “Layered quantization matrices representation and compression”, JCTVC-H0314, San José, CA, USA, February 2012).

[0082] JCTVC-E073: QMs are transmitted in a specific parameter set (QMPS). Within the QMPS, QMs are transmitted in increasing size order (sizeId / matrixId similar to HEVC). Prediction (= copying) of QM coefficients from any previously decoded QMs, including the previous QMPS, was proposed. Upconversion with linear interpolation was used to accommodate smaller reference QMs, while simple downsampling was used to accommodate larger reference QMs. This was ultimately rejected during HEVC standardization.

[0083] JCTVC-H0314: QM is transmitted in descending order. Figure 5 The fixed prediction tree shown (without explicit reference index) copies the previously sent QM instead of transmitting a new QM, and uses simple downsampling if the reference QM is larger. This was ultimately rejected during HEVC standardization.

[0084] Both proposals involve HEVC and cannot cope with the complexity introduced by VVC.

[0085] This application proposes to simplify the quantization matrix signaling and prediction process of VVC draft 5 (following the adoption of JVET-N0847) by incorporating one or more of the following, while enhancing them so that any QM can be predicted from any previously signaled QM:

[0086] - Unified QM indexes to include size and type so that reference index discrepancies can be resolved for any prior

[0087] The QM index of the previous transmission;

[0088] -Transmit quantization matrices in descending order of block size;

[0089] -Specify the prediction process as a replication or extraction process as needed;

[0090] - All QM coefficients of size-64 are transmitted so that size-64 QM can be used as a predictor.

[0091] Furthermore, a QM derivation process involving upsampling of blocks larger than 8x8 and downsampling of rectangular blocks is described to select the QM index according to the block parameters and to adapt the QM signaled size to the actual block size.

[0092] For ease of presentation, we refer to the process of predicting the quantization matrix based on default values ​​or previously transmitted values ​​as the QM prediction process, and the process of adapting the transmitted or predicted QM to the size and chroma format of the transform block as the QM derivation process. The QM prediction process can be part of the process of parsing the scaling list data, for example, at the picture level. The derivation process is typically at a lower level, such as at the transform block level. Each aspect is presented in further detail below, followed by draft text examples and performance results.

[0093] • Derivation and use of a single matrix index for identifying a QM. A single identifier allows referencing any previously transmitted matrix when using prediction (copying), and transmitting larger matrices first to avoid interpolation during prediction.

[0094] • The QM prediction process consists in copying or extracting the previously signaled QM (reference QM), which is the transmitted, predicted or default reference QM.

[0095] The QM derivation process, which for a given transform block consists in selecting a QM index based on the block size, color components, and prediction mode, and then adapting the size of the selected QM to the size of the block. The resizing process is based on bit shifting of the x and y coordinates within the transform block to index the coefficients of the selected QM.

[0096] • Transmission of all coefficients for size -64QM.

[0097] These aspects simplify the specification compared to VVC draft 5 (text changes are halved compared to JVET-N0847) and bring significant bit savings (the bit cost of scaling_list_data can be halved).

[0098] Unified QM Index

[0099] The QM used for quantization / dequantization of a transform block is identified by a single parameter matrixId. In one embodiment, the unified matrixId (QM index) is a combination of the following items:

[0100] - A size identifier related to the CU size (i.e., the CU encompasses a square shape, since only square-sized matrices are transmitted), not the block size. Note that here, for luma or chroma, the size identifier is controlled by the luma block size (e.g., max(luma block width, luma block height)). When the luma and chroma trees are separated, for chroma, "CU size" will refer to the size of the block projected on the luma plane.

[0101] - Matrix types for luma QMs are listed first, as they can be larger than chroma (e.g. in case of 4:2:0 chroma format)

[0102] According to this embodiment, the QM index derivation is shown in Tables 4 and 5 and Equation (1).

[0103] Table 4: Size identifiers (proposed)

[0104] brightness Chroma sizeId 64x64 32x32 0 32x32 16x16 1 16x16 8x8 2 8x8 4x4 3 4x4 2x2 4

[0105] Table 5: Matrix type identifiers (proposed)

[0106] CuPredMode cIdx (color component) matrixTypeId MODE_INTRA 0(Y) 0 MODE_INTER 0(Y) 1 MODE_INTRA 1(Cb) 2 MODE_INTER 1(Cb) 3 MODE_INTRA 2(Cr) 4 MODE_INTER 2(Cr) 5

[0107] The unified matrixId is derived as follows:

[0108] MatrixId=N*sizeId+MatrixTypeId (1)

[0109] Where N is the number of possible type identifiers, for example N=6.

[0110] In another embodiment, if more than six QM types are defined, sizeId should be multiplied by the correct number, which is the number of quantization matrix types. In other embodiments, other parameters may also be different, such as a specific block size, a signaled matrix size (here limited to 8×8), or the presence of a DC coefficient. Note that the QMs are listed by decreasing block size and are identified by a single index, as shown in Table 6.

[0111] Table 6: Unified matrixId (proposed)

[0112]

[0113] QM prediction process

[0114] Instead of transmitting the QM coefficients, the QM can be predicted based on default values ​​or based on any previously transmitted values. In one embodiment, when the reference QM is of the same size, the QM is copied, otherwise it is decimated by the relevant ratio, such as Figure 6 This is shown in the example in , where a size-4 luma QM is predicted from a size-8.

[0115] The decimation is described by the following equation:

[0116] ScalingMatrix[matrixId][x][y]=refScalingMatrix[i][j] (2)

[0117] where matrixSize = (matrixId < 20) × 8: (matrixId < 26) × 4: 2)

[0118] x=0…matrixSize–1, y=0…matrixSize–1,

[0119] i=x<<(log2(refMatrixSize)-log2(matrixSize)), and

[0120] j=y<<(log2(refMatrixSize)-log2(matrixSize)).

[0121] where refMatrixSize matches the size of refScalingMatrix (and therefore the range of the i and j variables).

[0122] exist Figure 6 In the example shown, luma size-4QM (which is a 4×4 array: matrixSize is 4) is predicted based on luma size-8QM, which is an 8×8 array (refMatrixSize is 8); one of the two rows and one of the two columns are discarded to generate a 4×4 array (i.e., the element (2x, 2y) in the reference QM is copied to the element (x, y) in the current QM).

[0123] Equation (3) takes the following form:

[0124] ScalingMatrix[matrix Id][x][y]=refScalingMatrix[i][j] (3)

[0125] x=0...3, y=0...3, i=x<<1, j=y<<1.

[0126] When the reference QM has a DC value, it is copied as the DC value if the current QM requires a DC value; otherwise, it is copied to the upper-left QM coefficient.

[0127] In a preferred embodiment, this QM prediction process is part of the QM decoding process, but in another embodiment it may be deferred to the QM derivation process, where decimation for prediction purposes is merged with the QM resizing sub-process.

[0128] QM derivation process

[0129] The proposed derivation process for the quantization matrix first selects the correct QM index based on the block parameters (unified QM index) as described above. It then unifies the processes for decimation of rectangular blocks, repetition for blocks larger than, for example, 8×8, and chroma format adaptation into a single process. The proposed process is based on bit shifting of the x and y output coordinates. To select the right row / column of the selected QM, it is only necessary to left-shift and then right-shift the x / y output coordinates, as shown in the following equations.

[0130] m[x][y]=ScalingMatrix[matrixId][i][j] (4)

[0131] where i=(x<<log2MatrixSize)> >log2(blkwidth), and

[0132] j=(y<<log2MatrixSize)> >log2(blkHeight).

[0133] where log2MatrixSize is the log2 of the size of ScalingMatrix[matrixId] (which is a square 2D array), blkWidth and blkHeight are the width and height of the current transform block, respectively, where x ranges from 0 to blkWidth-1 and y ranges from 0 to blkHeight-1.

[0134] The following are some examples to illustrate the QM derivation process. Figure 7 In the example shown in , the QM for the luma 16×8 block is derived from the luma size -16QM, which is actually an 8×8 array plus the DC coefficient. For this example, blkWidth is equal to 16, blkHeight is equal to 8, and log2MatrixSize is equal to 3, so equation (5) takes the following form:

[0135] m[x][y]=ScalingMatrix[matrixId][i][j] (5)

[0136] Where i=(x<<3)>>4, and j=(y<<3)>>3, where x=0...15 and y=0...7.

[0137] Here, x is shifted right by 1, and y remains unchanged (i.e., column i in the selected QM is copied to columns 2*i and 2*i+1 in the current QM). In addition, since the selected QM has a DC coefficient, it is copied to m[0][0].

[0138] In such Figure 8 In another example illustrated in , a QM for the chroma 4x2 block of an 8x4 CU (4:2:0 format) is generated. This matches the 8x4 CU size, and its enclosing square is 8x8. Therefore, the selected QM is a size-8 QM, where the chroma QM is decoded as a 4x4 array. Here, blkWidth is equal to 4, blkHeight is equal to 2, and log2MatrixSize is equal to 2, so equation (6) takes the following form:

[0139] m[x][y]=ScalingMatrix[matrixId][i][j] (6)

[0140] Where i=(x<<2)>>2, and j=(y<<2)>>1, where x=0...3 and y=0...1.

[0141] Here, x is unchanged and y is left-shifted by 1 (ie, row 2y in the reference QM is copied to row y in the current QM).

[0142] In the following example, the proposed adaptation to 4:2:2 and 4:4:4 formats differs from VVC draft 5. Instead of finding a QM that matches the chroma block size (except for 64x64 where there is no chroma matrix), the size matching is based on the same (luminance) CU size (i.e., the size of the block projected on the luminance plane), and coefficients are repeated if necessary. This makes the QM design independent of the chroma format.

[0143] exist Figure 9 In the example shown in , a QM is generated for the chroma 8x2 block of an 8x4 CU (4:2:2 format). The selected QM is the same as Figure 8 The above example shown in is the same, but the 4:2:2 chroma format requires twice as many columns. Here, the columns are repeated, so x is shifted right by 1 and y is still shifted left by 1. In particular, blkWidth is equal to 8, blkHeight is equal to 2, and log2MatrixSize is equal to 2, so equation (7) takes the following form:

[0144] m[x][y]=ScalingMatrix[matrix Id][i][j] (7)

[0145] Where i=(x<<2)>>3, and j=(y<<2)>>1, where x=0...7 and y=0...1.

[0146] exist Figure 10 In the example shown in , a QM is generated for the chroma 8x4 block of an 8x4 CU (4:4:4 format). The selected QM is still the same as Figure 8 and 9 The example shown in is the same as in , but the 4:4:4 chroma format requires twice as many rows and columns as the 4:2:0 chroma format. Here, the columns must be repeated, so x is right-shifted by 1, but the decimation of the rows can be skipped (due to the rectangular shape), so y is not shifted. In particular, blkWidth is equal to 8, blkHeight is equal to 4, and log2MatrixSize is equal to 2, so Equation (8) takes the following form:

[0147] m[x][y]=ScalingMatrix[matrixId][i][j] (8)

[0148] Where i=(x<<2)>>3, and j=(y<<2)>>2, where x=0...7 and y=0...3.

[0149] Number of coefficients transmitted for size-64

[0150] In one embodiment, to enable prediction from smaller QMs of size-64, all coefficients of size-64 are transmitted in the scaling list syntax, even though the lower right quadrant is never used by the VVC transform and quantization process. In general, we can send all coefficients of the maximum QM.

[0151] However, it is worth noting that compared to previous work (JVET-N0847), the overhead associated with this increased number of transmitted coefficients can be limited to 2×16 bits in the worst case, since the syntax element scaling_list_delta_coef can be set to zero for the lower right quadrant of size-64 QM when it is not used as a predictor: for the lower right quadrant, 4×4=16 delta coefficients are signaled, each costing 1 bit if forced to zero (using exp-Golomb decoding), and there are two size-64 QMs (luminance intra / inter).

[0152] The tests described in Table 8 show that this overhead is negligible compared to the gains from improved prediction.

[0153] In another embodiment, the coefficients of the lower right quadrant of the size-64 QMs are not transmitted as part of the signaling of the size-64 QM, but are transmitted as supplementary parameters when a smaller QM is first predicted from a given size-64 QM.

[0154] Table 7 provides some comparisons between the method described in JVET-N0847 and the proposed method:

[0155] Table 7

[0156]

[0157] Below, some syntax and semantics according to an embodiment are described.

[0158] PPS syntax and semantics (minor adjustments)

[0159]

[0160] pps_scaling_list_data_present_flag equal to 1 specifies that scaling list data for pictures referencing the PPS are derived based on the scaling list specified by the active SPS and the scaling list specified by the PPS. pps_scaling_list_data_present_flag equal to 0 specifies that scaling list data for pictures referencing the PPS are inferred to be equal to those specified by the active SPS. When scaling_list_enabled_flag is equal to 0, the value of pps_scaling_list_data_present_flag shall be equal to 0. When scaling_list_enabled_flag is equal to 1, sps_scaling_list_data_present_flag is equal to 0, and pps_scaling_list_data_present_flag is equal to 0, the default scaling matrix is ​​used to derive the array ScalingMatrix as described in the Scaling List Data Semantics specified in clause 7.4.5.

[0161] Please note that this syntax / semantics is an example close to the HEVC standard or VVC draft, and is not restrictive. For example, scaling_list_data transport is not limited to SPS or PPS and can be sent in other ways.

[0162] Scaling List Data Syntax / Semantics (Simplified)

[0163]

[0164] scaling_list_pred_mode_flag[matrixId] equal to 0 specifies that the scaling matrix is ​​derived from the value of the reference scaling matrix. The reference scaling matrix is ​​specified by scaling_list_pred_matrix_id_delta[matrixId]. scaling_list_pred_mode_flag[matrixId] equal to 1 specifies that the value of the scaling list is explicitly signaled.

[0165] scaling_list_pred_matrix_id_delta[matrixId] specifies the reference scaling matrix used to derive the scaling matrix, as described below. The value of scaling_list_pred_matrix_id_delta[matrixId] should be in the range of 0 to matrixId, inclusive.

[0166] When scaling_list_pred_mode_flag[matrixId] is equal to zero:

[0167] -The variable refMatrixSize and the array refScalingMatrix are first derived as follows:

[0168] ο If scaling_list_pred_matrix_id_delta[matrixId] is equal to zero, the following applies to set the default value:

[0169] ■ refMatrixSize is set equal to 8,

[0170] ■If matrixId is an even number,

[0171] refScalingMatrix= (9)

[0172] {

[0173] {16, 16, 16, 16, 16, 16, 16} / / Placeholder for the default value within the frame

[0174] {16, 16, 16, 16, 16, 16, 16}

[0175] {16, 16, 16, 16, 16, 16, 16}

[0176] {16, 16, 16, 16, 16, 16, 16}

[0177] {16, 16, 16, 16, 16, 16, 16}

[0178] {16, 16, 16, 16, 16, 16, 16}

[0179] {16, 16, 16, 16, 16, 16, 16}

[0180] {16, 16, 16, 16, 16, 16, 16}

[0181] },

[0182] Otherwise

[0183] refScalingMatrix= (10)

[0184] {

[0185] {16, 16, 16, 16, 16, 16, 16} / / Placeholder for inter-frame default value

[0186] {16, 16, 16, 16, 16, 16, 16}

[0187] {16, 16, 16, 16, 16, 16, 16}

[0188] {16, 16, 16, 16, 16, 16, 16}

[0189] {16, 16, 16, 16, 16, 16, 16}

[0190] {16, 16, 16, 16, 16, 16, 16}

[0191] {16, 16, 16, 16, 16, 16, 16}

[0192] {16, 16, 16, 16, 16, 16, 16}

[0193] },

[0194] o Otherwise (if scaling_list_pred_matrix_id_delta[matrixId] is greater than zero), the following applies:

[0195] refMatrixId=matrixId-scaling_list_pred_matrix_id_delta[matrixId](11)

[0196] refMatrixSize=(refMatrixId<20)? 8:(refMatrixId<26)? 4:2) (12)

[0197] refScalingMatrix=ScalingMatrix[refMatrixId] (13)

[0198] -Then derive the array ScalingMatrix[matrixId] as follows:

[0199] ScalingMatrix[matrixId][x][y]=refScalingMatrix[i][j] (14)

[0200] Where matrixSize = (matrixId<20) × 8: (matrixId<26) × 4: 2)

[0201] x=0…matrixSize–1, y=0…matrixSize–1,

[0202] i=x<<(log2(refMatrixSize)-log2(matrixSize)), and

[0203] j=y<<(log2(refMatrixSize)-log2(matrixSize))

[0204] scaling_list_dc_coef_minus8[matrixId] plus 8 specifies the first value of the scaling matrix when relevant, as described in clause xxx. The value of scaling_list_dc_coef_minus8[matrixId] shall be in the range -7 to 247, inclusive.

[0205] When scaling_list_pred_mode_flag[matrixId] is equal to zero, scaling_list_pred_matrix_id_delta[matrixId] is greater than zero, and refMatrixId < 14, the following applies:

[0206] - if matrixId < 14, scaling_list_dc_coef_minus8[matrixId] is inferred to be equal to scaling_list_dc_coef_minus8[refMatrixId],

[0207] Otherwise, ScalingMatrix[matrixId][0][0] is set equal to scaling_list_dc_coef_minus8[refMatrixId]+8

[0208] When scaling_list_pred_mode_flag[matrixId] is equal to zero, scaling_list_pred_matrix_id_delta[matrixId] is equal to zero (indicating the default value), and matrixId < 14, then scaling_list_dc_coef_minus8[matrixId] is inferred to be equal to 8

[0209] scaling_list_delta_coef specifies the difference between the current matrix coefficient ScalingList[matrixId][i] and the previous matrix coefficient ScalingList[matrixId][i-1] when scaling_list_pred_mode_flag[matrixId] is equal to 1. The value of scaling_list_delta_coef should be in the range of -128 to 127, inclusive. The value of ScalingList[matrixId][i] should be greater than 0. When present (i.e., scaling_list_pred_mode_flag[matrixId] is equal to 1), the array ScalingMatrix[matrix_Id] is derived as follows:

[0210] ScalingMatrix[matrixId][i][j]=ScalingList[matrixId][k] (15)

[0211] Where k = 0…coefNum–1,

[0212] i = diagScanOrder[log2(coefNum) / 2][log2(coefNum) / 2][k][0], and

[0213] j=diagScanOrder[log2(coefNum) / 2][log2(coefNum) / 2][k][1]

[0214] The main simplifications compared to the JVET-N0847 syntax are the removal of a for() loop and the simplification of indexing from [sizeId][matrixId] to [matrixId].

[0215] “Clause xxx” refers to an undetermined section number to be introduced in the VVC specification, which matches the scaling matrix derivation process in this paper.

[0216] Note that this syntax / semantics is an example close to the HEVC standard or VVC draft 5 and is not restrictive. For example, the coefficient range is not limited to 1…255, it can be 1…127 (7 bits) or -64…63, for example. Moreover, it is not limited to 30 QMs, organized as 6 types × 5 sizes (there can be 8 types, and fewer or more sizes. See Table 5, which can be modified. A simple modification of the coefNum and the conditions for the presence of the DC coefficient would then be required). The type of QM prediction (here just copying) is also not restrictive. For example, scaling factors or offsets can be added, and explicit decoding can be added as a residual on top of the prediction. The same situation represents the method used for coefficient transmission (here DPCM), the presence of a DC coefficient, and a fixed number of coefficients (only a subset can be transmitted).

[0217] Regarding default values, it is not limited to the two default value QMs associated with MODE_INTRA and MODE_INTER, and here it is similarly filled with all 16 values ​​until the relevant default value is reached (for example, the same default value QM as HEVC can be selected).

[0218] Likewise, instead of comparing sequences, one can mathematically express the number of coefficients coefNum to be signaled, with the same result: coefNum = Min(64,4096>>((matrixId+4) / 6)*2), which is closer to the HEVC or current VVC draft approach, but introduces a potentially undesirable split.

[0219] Note here that the larger matrix is ​​transmitted first, and a single index is used so that the prediction reference (indicated by scaling_list_pred_matrix_id_delta) can be any previously transmitted matrix, regardless of its intended block size or type, or the default value (e.g. if scaling_list_pred_matrix_id_delta is zero).

[0220] Figure 11A process (1100) for parsing a scaling list data syntax structure according to an embodiment is shown. For this embodiment, the input is a decoded bitstream and the output is a ScalingMatrix array. For clarity, details about DC values ​​are omitted. Specifically, at step 1110, a QM prediction mode is decoded from the bitstream. If the QM is predicted (1120), the decoder further determines whether to infer (predict) the QM in the bitstream or to signal the QM based on the aforementioned flag. At step 1130, the decoder decodes QM prediction data from the bitstream, which is required to infer the QM when there is no signaling, such as the QM index difference scaling_list_pred_matrix_id_delta. The decoder then determines (1140) whether to predict the QM based on a default value (e.g., if scaling_list_pred_matrix_id_delta is zero) or based on a previously decoded QM. If the reference QM is a default QM, the decoder selects (1150) the default QM as the reference QM. There may be several default QMs to choose from, for example depending on the parity of matrixId. Otherwise, the decoder selects (1155) a previously decoded QM as the reference QM. The index of the reference QM is derived from the matrix and the above index difference. In step 1160, the decoder predicts the QM based on the reference QM. If the reference QM is the same size as the current QM, the prediction consists of a simple copy, or if it is larger than expected, it consists of decimation. The result is stored in ScalingMatrix[matrixId].

[0221] If QM is not predicted (1120), the decoder determines (1170) the number of QM coefficients to be decoded from the bitstream based on matrixId. For example: 64 if matrixId is less than 20, 16 if matrixId is between 20 and 25, otherwise 4. In step 1175, the decoder decodes the relevant number of QM coefficients from the bitstream. In step 1180, the decoder organizes the decoded QM coefficients in a 2D matrix based on a scan order, such as diagonal scanning. The result is stored in ScalingMatrix[matrixId]. Using ScalingMatrix[matrixId], the decoder can use the QM derivation process to obtain the quantization matrix m[][] for dequantizing the transform blocks that may be in a non-square shape and / or in a different chroma format.

[0222] At step 1190, the decoder checks whether the current QM is the last QM to be parsed. If not, control returns to step 1110; otherwise, the QM parsing process stops when all QMs have been parsed from the bitstream.

[0223] Figure 12 A process (1200) for encoding a scaling list data syntax structure on the encoder side according to an embodiment is shown. On the encoder side, the QMs are scanned from larger to smaller block sizes (e.g., matrixId from 0 to 30) in the order described. In step 1210, the encoder searches for prediction preferences to decide whether the current QM is a copy (or extraction) of a previously encoded QM. The QMs can be designed in a way that optimizes QM prediction efficiency, for example, if a QM is initially close enough, or close to a default value QM, they can be forced to be equal (or extracted if the sizes are different). In addition, the coefficients in the lower right quadrant of the size-64 QM can be optimized to better predict subsequent QMs, or to reduce the QM bit cost if they are never reused for prediction. Once decided, the QM prediction mode is encoded in the bitstream.

[0224] Specifically, if the encoder decides (1220) to use prediction, then at step 1230, the prediction mode is encoded (e.g., scaling_list_pred_mode_flag = 0). At step 1240, the prediction parameters (e.g., QM index difference scaling_list_pred_matrix_id_delta) are encoded: a zero index difference for the default QM value, or a relative index difference if the previous QM is selected as the prediction reference. On the other hand, if explicit signaling is decided, then at step 1250, the prediction mode is encoded (e.g., scaling_list_pred_mode_flag = 0). Then, a diagonal scan is performed (1260), followed by QM coefficient encoding (1270).

[0225] At step 1280, the encoder checks whether the current QM is the last QM to be encoded. If not, control returns to step 1210; otherwise, the QM encoding process stops when all QMs are encoded into the bitstream.

[0226] Figure 13A process 1300 for QM derivation according to an embodiment is shown. The input may include a ScalingMatrix array and transform block parameters, such as size (width / height), prediction mode (intra / inter / IBC, ...) and color components (Y / U / V). The output is a QM with the same size as the transform block. For clarity, details about the DC value are omitted. Specifically, in step 1310, the decoder determines the QM index matrixId based on the current transform block size (width / height), prediction mode (intra / inter / IBC, ...), color components (Y / U / V) (unified QM index) as described above. In step 1320, the decoder adjusts the size of the selected QM (ScalingMatrix[matrixId]) to match the transform block size, as described above. In a variant, step 1320 may include the decimation required for prediction.

[0227] The QM derivation process is similar on the encoder side. Quantization divides the transform coefficients by the QM value, while dequantization multiplies them by the QM value. However, the QMs are the same. In particular, the QMs used for reconstruction in the encoder will match those signaled in the bitstream.

[0228] Conceptually, the transform coefficients d[x][y] can be quantized as follows, where qStep is the quantization step size and m[][] is the quantization matrix:

[0229] TransCoeffLevel[xTbY][yTbY][cIdx][x][y]=d[x][y] / qStep / m[x][y]

[0230] But for integer calculations and to avoid division, it usually looks like:

[0231] TransCoeffLevel[xTbY][yTbY][cIdx][x][y]=((d[x][y]*im[x][y]*

[0232] ilevelScale[qP%6]>>(qP / 6))+(1<<(bdShift-1)))>>bdShift)

[0233] Where, for example, im[x][y]~=65536 / m[x][y] and ilevelScale[0...5]=65536 / levelScale[0...5], and bdShift is an appropriate value. In practice, for software encoders, im*ilevelScale is usually pre-calculated and stored in a table.

[0234] In the above description, the QM prediction process and the QM derivation process are performed separately. In another embodiment, QM prediction can be deferred to the QM derivation process. This embodiment does not change the QM signaling syntax. This embodiment, which differs functionally due to continuous resizing, defers the prediction portion (obtaining the reference QM + copying / reducing) and diagonal scanning to the "QM derivation process" described later.

[0235] In one embodiment, the output of the scaling list data parsing process 1100 is an array of ScalingList() rather than an array of ScalingMatrix, along with prediction flags and valid prediction parameters: the ScalingMatrixPredId array always contains the index of the defined ScalingList (default or signaled). During QM decoding, this array is recursively constructed by interpreting scaling_list_pred_matrix_id_delta so that the QM derivation process can directly use the index to obtain the actual value to construct the QM for dequantizing the current transform block.

[0236] In the following, an example is provided to illustrate the zoom list semantics according to an embodiment.

[0237] Scaling matrix derivation procedure (new: partially replaces the description in Scaling list semantics; this is section XXX)

[0238] The inputs to this process are the prediction mode predMode, the color component variable cIdx, the block width blkWidth and the block height blkHeight.

[0239] The output of this process is a (blkWidth) x (blkHeight) array m[x][y] (scaling matrix), where x and y are the horizontal and vertical coefficient positions. Note that SubWidthC and SubHeightC depend on the chroma format and indicate the ratio of the number of samples in the luma component and the chroma components.

[0240] The variable matrixId is derived as follows:

[0241] matrixId=6*sizeId+matrixTypeId (xxx-1)

[0242] Where, subWidth=(cIdx>0)?SubWidthC:1,

[0243] subHeight=(cIdx>0)? SubHeightC:1,

[0244] sizeId=6–max(log2(blkWidth*subWidth),log2(blkHeight*subHeight)), and

[0245] matrixTypeId=(2*cIdx+(predMode==MODE_INTER?1:0))

[0246] The variable log2MatrixSize is derived as follows:

[0247] log2MatrixSize=(matrixId<20)? 3:(matrixId<26)? 2:1 (xxx-2)

[0248] For an x ​​range from 0 to blkWidth-1, inclusive, and a y range from 0 to blkHeight-1, inclusive, the output array m[x][y] is derived by applying the following formula:

[0249] m[x][y]=ScalingMatrix[matrixId][i][j] (xxx-3)

[0250] where i=(x<<log2MatrixSize)> >log2(blkWidth), and

[0251] j=(y<<log2MatrixSize)> >log2(blkHeight)

[0252] If matrixId is lower than 14, then m[0][0] is further modified as follows:

[0253] m[0][0]=scaling_list_dc_coef_minus8[matrixId]+8 (xxx-4)

[0254] Note that this is an example, not a limitation, for the scaling_list_data syntax and semantics. For example, it is not limited to the two default QMs associated with MODE_INTRA and MODE_INTER, and similarly filled here with all 16 values ​​until the relevant default value is reached (for example, the same default QM as HEVC can be selected). There can be a single default QM or more than 2 default QMs. For example, if there are more or less than six types of matrices per block size, the MatrixId calculation can be different. It is important that horizontal and vertical reduction and enlargement to accommodate block sizes different from the selected QM are completed in a single process, preferably simple (left shift followed by right shift here).

[0255] For rectangular blocks, we are not restricted to choosing a QM identifier for the current block that encloses a square: the derivation of sizeId in Eq. xxx-1 may follow different rules.

[0256] Likewise, the chosen QM size log2MatrixSize can be expressed mathematically instead of a series of comparisons, where the result is the same: log2MatrixSize = min(3, 6 - (MatrixId + 4) / 6), but this introduces a division that may be undesirable.

[0257] In the following, examples are provided describing semantics for a zoom process according to an embodiment.

[0258] Scaling of transform coefficients (adaptation)

[0259] […]

[0260] For the derived scaled transform coefficients d[x][y], where x=0...nTbW-1, y=0...nTbH-1, the following applies:

[0261] The intermediate scaling factor array m of -(nTbW)x(nTbH) is derived as follows:

[0262] - m[x][y] is set equal to 16 if one or more of the following conditions are true:

[0263] -scaling_list_enabled_flag is equal to 0.

[0264] -transform_skip_flag[xTbY][yTbY] is equal to 1.

[0265] Otherwise, m is the output of the scaling matrix derivation process as specified in clause xxx, which is called with the prediction mode CuPredMode[xTbY][yTbY], the color component variables cIdx, the block width nTbW and the block height nTbH as input.

[0266] -The scaling factor ls[x][y] is derived as follows:

[0267] […]

[0268] The main change compared to VVC draft 5 is the invocation of clause xxx instead of copying part of the array described in the scaling_list_data semantics.

[0269] Note that, as mentioned above, this is an example intended to minimize changes to the current VVC draft, and is not intended to be limiting. For example, the color component inputs used for the scaling matrix derivation process can be different from cIdx. Furthermore, QM is not limited to being used as a scaling factor, but can be used as a QP offset, for example.

[0270] The QM decoding performance was tested using the same test set used during HEVC standardization, augmented with a set of QMs derived from recommended or default QMs for public standards (JPEG, MPEG2, AVC, HEVC) and QMs found in actual broadcasts. In all tests, some QMs were copied from one type to another (e.g., luma to chroma, or intra to inter), and / or from one size to another.

[0271] The following table reports the number of bits required to encode scaling_list_data using the following three different methods: HEVC, JVET-N0847, and the proposed. Specifically, HEVC uses the HEVC test set (24 QMs per test), and the other two use the derived test set (30 QMs per test, with additional sizes: size-2 for chroma and size-64 for luma; size-2 QM is downsampled from size-4, size-64 is copied from size-32, size-32 is copied from size-16, and sizes 16, 8, 4 remain as they are).

[0272] Table 8

[0273]

[0274] For this test, it can be seen that the proposed technique saves a large amount of bits, even compared to HEVC, while the proposed method encodes more QM.

[0275] Returning to the approach in JCTVC-E073, the reference index in JCTVC-E073 (triplet: QMPS, size, type) is more complex than the one proposed here because the previous QMPS index requires storing the previous QMPS. Linear interpolation introduces complexity. Downsampling is similar to the one proposed here.

[0276] Returning to the approach in JCTVC-H0314, the large-to-small transfer is close to what is proposed here, however the fixed prediction tree in JCTVC-H0314 is less flexible than the unified index and explicit reference proposed here.

[0277] QM for intra block copy mode

[0278] In the above, different QMs are specified for the two block prediction modes (i.e., intra and inter). However, in addition to intra and inter, there is a new prediction mode in VVC: IBC (Intra Block Copy), in which blocks can be predicted from reconstructed samples of the same picture using appropriate displacement vectors. For QM selection in IBC prediction mode, JVET-N0847 and the above embodiments use the same QMs as for intra mode.

[0279] Since IBC mode is closer to inter than intra prediction, in one embodiment, it is proposed to reuse the signaled QM for inter mode (instead of intra). However, although close to inter prediction, IBC is different: the displacement vectors do not match the object or camera motion, but are used for texture replication. This may lead to specific artifacts, where a specific QM can help optimize the decoding of IBC blocks in different embodiments. Below, we propose changing the QM selection for the IBC prediction mode:

[0280] • The preferred embodiment is to choose the same QM as inter mode (instead of intra), because IBC is closer to inter prediction than intra prediction.

[0281] • Another option is to have a specific QM for IBC mode.

[0282] o These can be explicitly signaled in the syntax, or inferred (e.g. the average of intra and inter QMs).

[0283] In a preferred embodiment, the QM selection or derivation process for a particular transform block selects the inter QM if the block has IBC prediction mode. Figure 12 , the QM derivation process of step 1210 needs to be adjusted as described below.

[0284] In the draft text presented above, the QM choice is described in equation (xxx-1), which can be changed as follows:

[0285] matrixId=6*sizeId+matrixTypeId (xxx-1)

[0286] Where subWidth=(cIdx>0)? SubWidthC:1,

[0287] subHeight=(cIdx>0)? SubHeightC:1,

[0288] sizeId=6–max(log2(blkWidth*subWidth),log2(blkHeight*subHeight)), and

[0289]

[0290] matrixTypeId=(2*cIdx+(predMode==MODE_INTRA?0:1))

[0291] Note that a block can be coded in intra mode (MODE_INTRA), inter mode (MODE_INTER), or intra block copy mode (MODE_IBC). When matrixTypeId is set as in Table 6, or (matrixTypeId = (2*cIdx + (predMode == MODE_INTER? 1:0))), a MODE_IBC block selects matrixTypeId as if it were a MODE_INTRA block. Where (xxx-1) changes: matrixTypeId = (2*cIdx + (predMode == MODE_INTRA? 0:1)), a MODE_IBC block selects matrixTypeId as if it were a MODE_INTER block.

[0292] In the draft text proposed by JVET-N0847, the QM selection is described in Table 7-14, which can be changed as follows: In particular, the matrixId of MODE_IBC is assigned in the same way as MODE_INTER, rather than the same as MODE_INTRA as in JVET-N0847.

[0293] Table 9: JVET-N0847 specification changes to matrixId according to sizeId, prediction mode and color components

[0294] sizeId CuPredMode cIdx (color component) matrixId 2,3,4,5,6 MODE_INTRA 0(Y) 0 1,2,3,4,5 MODE_INTRA 1(Cb) 1 1,2,3,4,5 MODE_INTRA 2(Cr) 2 2,3,4,5,6 MODE_INTER 0(Y) 3 1,2,3,4,5 MODE_INTER 1(Cb) 4 1,2,3,4,5 MODE_INTER 2(Cr) 5 2,3,4,5,6 MODE_IBC 0(Y) 03 1,2,3,4,5 MODE_IBC 1(Cb) 14 1,2,3,4,5 MODE_IBC 2(Cr) 25

[0295] Variant 1: Explicit signaling of QM for IBC

[0296] In this variant, specific QMs are used for IBC blocks (different from intra and inter QMs), and these QMs are explicitly signaled in the bitstream. This results in more QMs, which requires adaptation of the scaling_list_data syntax and matrixId mapping, and has a bit cost impact. According to this variant, the QM selection described in equation (xxx-1) can be changed as follows

[0297] matrixId=6*sizeId+matrixTypeId (xxx-1)

[0298] Where subWidth=(cIdx>0)? SubWidthC:1,

[0299] subHeight=(cIdx>0)? SubHeightC:1,

[0300] sizeId=6–max(log2(blkWidth*subWidth),log2(blkHeight*subHeight)), and

[0301]

[0302] matrixTypeId=(3*cIdx+(predMode==MODE_INTRA?0:(predMode==MODE_ INTER? 1:2)))

[0303] In JVET-N0847, the QM selection table can be changed as follows:

[0304] Table 10: Changes to JVET-N0847 to add more QMs for IBC

[0305]

[0306] Variant 2: Inferred QM for IBC Mode

[0307] In this variant, specific QMs are used for IBC blocks (different from intra and inter QMs). However, these QMs are not signaled in the bitstream but are inferred: for example, the average of the intra and inter QMs, or a specific default value, or a specific variation of the inter QM, such as scaling and offset.

[0308] Variant 3: Explicit IBC QM for Luminance Only

[0309] In this variant, the additional QM for IBC is limited to luma; the chroma QM for IBC can reuse the inter QM as in variant 1, or infer a new chroma QM as in variant 2.

[0310] Various methods are described herein, and each method includes one or more steps or actions for implementing the described method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or use of specific steps and / or actions may be modified or combined. In addition, terms such as "first", "second" and the like may be used to modify elements, components, steps, operations, etc., for example, "first decoding" and "second decoding" in various embodiments. Unless otherwise specifically required, the use of these terms does not mean the sequencing of the modified operation. Therefore, in this example, the first decoding does not need to be performed before the second decoding, and may, for example, occur before, during, or in a time period overlapping with the second decoding.

[0311] Various methods and other aspects described in this application can be used to modify modules, e.g. Figure 2 and Figure 3The quantization and inverse quantization modules (230, 240, 340) of the video encoder 200 and decoder 300 are shown. In addition, aspects of the present invention are not limited to VVC or HEVC and can be applied to, for example, other standards and recommendations, as well as extensions of any such standards and recommendations. Unless otherwise specified or technically excluded, the aspects described in this application can be used alone or in combination.

[0312] Various numerical values ​​are used in this application. The specific values ​​are for illustrative purposes, and the described aspects are not limited to these specific values.

[0313] Various implementations involve decoding. As used herein, "decoding" may include, for example, all or part of a process performed on a received coded sequence to produce a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. Whether the phrase "decoding process" is intended to specifically refer to a subset of operations or generally refer to a broader decoding process will be clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0314] Various implementations involve encoding.In a manner similar to the discussion above regarding "decoding," "encoding" as used in this application may include all or part of a process performed on an input video sequence to produce an encoded bitstream, for example.

[0315] Note that the syntax elements used here are descriptive terms. Therefore, they do not exclude the use of other syntax element names. In the above, the syntax elements for PPS and scaling lists are mainly used to illustrate various embodiments. It should be noted that these syntax elements can be placed in other syntax structures.

[0316] The implementation and aspects described herein can be implemented in, for example, a method or process, a device, a software program, a data stream or a signal. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method), the implementation of the features discussed can also be implemented in other forms (e.g., a device or program). For example, a device can be implemented with appropriate hardware, software, and firmware. The method can be implemented in, for example, a device such as a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes a communication device, such as a computer, a cellular phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate information communication between end users.

[0317] Reference to "one embodiment" or "an embodiment" or "an implementation" or "an implementation" and other variations thereof means that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in an implementation" or "in an implementation" and any other variations thereof in various places throughout this application are not necessarily all referring to the same embodiment.

[0318] Additionally, the present application may refer to “determining” various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.

[0319] Furthermore, the present application may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0320] Additionally, this application may refer to "receiving" various information. Like "accessing," receiving is intended to be a broad term. Receiving information can include, for example, accessing information or retrieving information (e.g., from a memory device). Furthermore, during operations such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information, "receiving" is often involved in one way or another.

[0321] It should be understood that, for example, in the case of "A / B," "A and / or B," and "at least one of A and B," the use of any of the following " / ," "and / or," and "at least one of" is intended to encompass selection of only the first listed option (A), or only the second listed option (B), or both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such wording is intended to include selection of only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A, B, and C). This can be extended to multiple items listed, as will be apparent to one of ordinary skill in this and related arts.

[0322] Furthermore, as used herein, the term "signal" specifically refers to indicating something to a corresponding decoder. For example, in certain embodiments, an encoder signals a quantization matrix for dequantization. Thus, in embodiments, the same parameters are used on both the encoder and decoder sides. Thus, for example, an encoder can send specific parameters to a decoder (explicit signaling) so that the decoder can use the same specific parameters. Conversely, if the decoder already has specific parameters along with other parameters, signaling can be used instead of sending them (implicit signaling), simply allowing the decoder to know and select the specific parameters. By avoiding the transmission of any actual functionality, bit savings are achieved in various embodiments. It should be understood that signaling can be implemented in various ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal information to a corresponding decoder. While the foregoing refers to the verb form of the term "signal," the term "signal" may also be used herein as a noun.

[0323] As will be apparent to one of ordinary skill in the art, implementations can generate various signals formatted to carry information that can be stored or transmitted, for example. The information can include, for example, instructions for performing a method, or data generated by one of the described implementations. For example, a signal can be formatted to carry a bitstream of the described embodiments. Such a signal can be formatted as, for example, an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal can be, for example, analog or digital information. As is known, the signal can be transmitted over a variety of different wired or wireless links. The signal can be stored on a processor-readable medium.

Claims

1. A method for video decoding, comprising: Obtaining a single identifier for identifying a quantization matrix from a plurality of quantization matrices based on a size identifier, a color component type, and a prediction mode parameter of a block to be decoded in a picture, wherein the single identifier is derived based on N*sizeId+matrixTypeId, wherein matrixTypeId indicates the prediction mode parameter and the color component type of the block, N is the number of possible type identifiers indicated by matrixTypeId, and sizeId indicates the size identifier; decoding a syntax element indicating a reference quantization matrix, wherein the syntax element specifies a difference between an identifier of the reference quantization matrix and the obtained identifier of the quantization matrix; Based on the reference quantization matrix, obtaining the quantization matrix; Dequantizing transform coefficients of the block based on the quantization matrix; as well as The block is decoded based on the dequantized transform coefficients.

2. The method according to claim 1, wherein: The size identifier of the block is different from a size identifier of another block corresponding to the reference quantization matrix.

3. The method according to claim 1, wherein: The elements of the quantization matrix are used as scaling factors when dequantizing the corresponding transform coefficients of the block.

4. The method according to claim 1, wherein: The elements of the quantization matrix are used as offsets when dequantizing the corresponding transform coefficients of the block.

5. A method for video encoding, comprising: Access the block to be encoded in the picture; accessing a quantization matrix for the block; Based on a size identifier, a color component type, and a prediction mode parameter of the block, obtaining a single identifier for identifying the quantization matrix from a plurality of quantization matrices, wherein the single identifier is derived based on N*sizeId+matrixTypeId, wherein matrixTypeId indicates the prediction mode parameter and the color component type of the block, N is the number of possible type identifiers indicated by matrixTypeId, and sizeId indicates the size identifier; encoding a syntax element indicating a reference quantization matrix, wherein the syntax element specifies a difference between an identifier of the reference quantization matrix and the obtained identifier of the quantization matrix; quantizing transform coefficients of the block based on the quantization matrix; as well as The quantized transform coefficients are entropy encoded.

6. The method according to claim 5, wherein: The size identifier of the block is different from a size identifier of another block corresponding to the reference quantization matrix.

7. The method according to claim 5, wherein: The elements of the quantization matrix are used as scaling factors when quantizing the corresponding transform coefficients of the block.

8. The method according to claim 5, wherein: The elements of the quantization matrix are used as offsets when quantizing the corresponding transform coefficients of the block.

9. The method according to claim 5, wherein: The reference quantization matrix is ​​previously signaled.

10. The method according to claim 5, wherein: The quantization matrix is ​​obtained from the reference quantization matrix by copying or decimating.

11. The method according to claim 10, wherein: In response to the quantization matrix having the same size as the reference quantization matrix, the quantization matrix is ​​obtained from the reference quantization matrix by copying.

12. The method according to claim 10, wherein: In response to the quantization matrix having a size different from that of the reference quantization matrix, the quantization matrix is ​​obtained from the reference quantization matrix by decimating at a corresponding ratio.

13. The method according to claim 5, wherein: The block size is MxN, and where M is the width and N is the height, the identifier of the block is based on the size of max(M,N), which is defined as the larger of M and N.

14. The method according to claim 5, wherein: The set of quantization matrices is signaled in order of increasing identifiers, with the quantization matrix for the largest block size being signaled first.

15. An apparatus for video decoding, comprising: at least one memory; as well as One or more processors, wherein the one or more processors are configured to: Obtaining a single identifier for identifying a quantization matrix from a plurality of quantization matrices based on a size identifier, a color component type, and a prediction mode parameter of a block to be decoded in a picture, wherein the single identifier is derived based on N*sizeId+matrixTypeId, wherein matrixTypeId indicates the prediction mode parameter and the color component type of the block, N is the number of possible type identifiers indicated by matrixTypeId, and sizeId indicates the size identifier; decoding a syntax element indicating a reference quantization matrix, wherein the syntax element specifies a difference between an identifier of the reference quantization matrix and the obtained identifier of the quantization matrix; Based on the reference quantization matrix, obtaining the quantization matrix; Dequantizing transform coefficients of the block based on the quantization matrix; as well as The block is decoded based on the dequantized transform coefficients.

16. The device according to claim 15, wherein: The size identifier of the block is different from a size identifier of another block corresponding to the reference quantization matrix.

17. The device according to claim 15, wherein: The elements of the quantization matrix are used as scaling factors when dequantizing the corresponding transform coefficients of the block.

18. The device according to claim 15, wherein: The elements of the quantization matrix are used as offsets when dequantizing the corresponding transform coefficients of the block.

19. An apparatus for video encoding, comprising: at least one memory; as well as One or more processors, wherein the one or more processors are configured to: Access the block to be encoded in the picture; accessing a quantization matrix for the block; Based on a size identifier, a color component type, and a prediction mode parameter of the block, obtaining a single identifier for identifying the quantization matrix from a plurality of quantization matrices, wherein the single identifier is derived based on N*sizeId+matrixTypeId, wherein matrixTypeId indicates the prediction mode parameter and the color component type of the block, N is the number of possible type identifiers indicated by matrixTypeId, and sizeId indicates the size identifier; encoding a syntax element indicating a reference quantization matrix, wherein the syntax element specifies a difference between an identifier of the reference quantization matrix and the obtained identifier of the quantization matrix; quantizing transform coefficients of the block based on the quantization matrix; as well as The quantized transform coefficients are entropy encoded.

Images