Image processing apparatus and image processing method

Abstract

[Problem] To provide a structure that efficiently generates a quantization matrix or exhibits improved signal transmission. [Solution] Provided is an image processing apparatus equipped with a decoding unit for decoding scaling list data and generating a first quantization matrix of a first size, and a generating unit for generating a second quantization matrix for a second size of a transform block to which zeroing of a high-frequency component is applied, by referring only to a partial matrix of the first quantization matrix generated by the decoding unit, and an inverse quantization unit for inversely quantizing quantized transform coefficients of the second size of the transform block, by using the second quantization matrix generated by the generating unit.

Description

[0001] This application is a divisional application of national application number 201980020797.2, international application date March 7, 2019, national entry date September 21, 2020, and invention title "Image Processing Apparatus and Image Processing Method". Technical Field

[0002] This disclosure relates to image processing apparatus and image processing methods. Background Technology

[0003] Conventionally, many video coding methods, such as H.264 / AVC and H.265 / HEVC, have been standardized for the purpose of efficiently transmitting, storing, or recording digital images. H.265 / HEVC is considered to achieve approximately twice the coding (compression) efficiency of H.264 / AVC. The Joint Video Experts Group (JVET), a standardization organization jointly established by ITU-T and ISO / IEC, has begun discussing the standardization of Future Video Coding (FVC), a next-generation video coding method, to further improve coding efficiency beyond that of H.265 / HEVC. The FVC reference software being developed based on the HEVC model is called the Joint Exploratory Model (JEM), and various technical elements incorporated into the JEM are described in Non-Patent Document 1.

[0004] Existing video coding methods involve various techniques, such as prediction (intra-frame prediction / inter-frame prediction), orthogonal transform, quantization, and entropy coding. Quantization, as one of these techniques, coarser quantizes the high-frequency components of the transform coefficients in the frequency domain after the orthogonal transform than quantizing the low-frequency components of the transform coefficients. This achieves the desired data rate while suppressing subjective image quality degradation. According to H.265 / HEVC (hereinafter referred to as "HEVC"), orthogonal transform and quantization are performed for each block called a transform unit (TU). Candidate TU sizes include 4×4, 8×8, 16×16, and 32×32, and quantization matrices corresponding to some TU sizes can be transmitted from the encoder to the decoder via a signal. The quantization matrix affects the quantization steps of the individual frequency components of the transform coefficients for each block. FVC allows a maximum TU size of 128×128 and also allows non-square TUs.

[0005] The larger the TU size, the greater the diversity of quantization matrices used. However, transmitting all these quantization matrices via signaling increases the overhead of the encoded bitstream, thereby reducing coding efficiency. To address this problem, Patent Documents 1 and 2 propose a technique in which, instead of transmitting all the quantization matrices used via signaling, only some of the quantization matrices are transmitted via signaling, and the remaining quantization matrices are generated from the quantization matrices transmitted via signaling, thus avoiding the increase in overhead.

[0006] Reference List

[0007] Non-patent literature

[0008] Non-Patent Document 1: J. Chen, E. Alshina, G.S. Sullivan, J.R.R. Hm, and J. Boyce, “Algorithm Description of Joint Exploration Test Model (JEM7)”, 7th Meeting of the Joint Video Exploration Group (JVET) of JVET-G1001, ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11: Turin, Italy, July 13-21, 2017

[0009] Patent documents

[0010] Patent Document 1: WO 2012 / 077408 A

[0011] Patent Document 2: WO 2012 / 160890 A Summary of the Invention

[0012] Technical issues

[0013] The decrease in coding efficiency caused by signal transmission of the quantization matrix and the impact of generating different quantization matrices from a given quantization matrix on device performance are mutually exclusive. In particular, in the case of FVC with significant variations in the combination of block size and shape, not only coding efficiency but also the processing cost required to generate the quantization matrix (e.g., hardware resource consumption, processing latency, increased power consumption, etc.) are factors that cannot be ignored.

[0014] Therefore, it is desirable to provide an improved system for efficiently generating or transmitting quantization matrices via signals.

[0015] Solution to the problem

[0016] According to this disclosure, an image processing apparatus is provided. The image processing apparatus includes: a decoding unit that decodes scaled list data to generate a first quantization matrix of a first size; a generation unit that generates a second quantization matrix for a zero-reduction transform block of a second size with applied high-frequency components by referring only to a portion of the first quantization matrix generated by the decoding unit; and an inverse quantization unit that inverse quantizes the quantization transform coefficients of the second-size transform block using the second quantization matrix generated by the generation unit.

[0017] Furthermore, according to this disclosure, an image processing method performed by an image processing apparatus is provided. The image processing method includes: decoding scaling list data to generate a first quantization matrix of a first size; generating a second quantization matrix for a transform block of a second size with zero-reduction applied to it, by referring only to a portion of the generated first quantization matrix; and inverse quantization of the quantization transform coefficients of the transform block of the second size using the generated second quantization matrix.

[0018] Furthermore, according to this disclosure, an image processing apparatus is provided. The image processing apparatus includes: a generation unit that generates a second quantization matrix for a transform block of a second size to which high-frequency components are applied, by referring only to a portion of a first quantization matrix of a first size; a quantization unit that uses the second quantization matrix generated by the generation unit to quantize the transform coefficients of the transform block of the second size in an image to be encoded, to generate quantized transform coefficients; and an encoding unit that encodes the quantized transform coefficients and a scaling list representing the first quantization matrix to generate an encoded stream.

[0019] Furthermore, according to this disclosure, an image processing method performed by an image processing apparatus is provided. The image processing method includes: generating a second quantization matrix for a transform block of a second size to which high-frequency components are applied, by referring only to a portion of a first quantization matrix of a first size; quantizing transform coefficients of the transform block of the second size in an image to be encoded using the generated second quantization matrix to generate quantized transform coefficients; and encoding the quantized transform coefficients and a scaling list representing the first quantization matrix to generate an encoded stream.

[0020] Beneficial effects of the invention

[0021] According to the technology disclosed herein, quantization matrices can be generated or transmitted via signals efficiently.

[0022] Note that the aforementioned beneficial effects are not necessarily limiting, and any beneficial effects disclosed in this specification or other beneficial effects that can be understood from this specification may be provided in addition to or in place of the aforementioned beneficial effects. Attached Figure Description

[0023] Figure 1 This is an illustration diagram used to explain the types of quantization matrices that can be used in HEVC.

[0024] Figure 2 This is an illustrative diagram showing an example of QTBT block partitioning in FVC.

[0025] Figure 3A This is an explanatory diagram used to illustrate the zeroing of the transform coefficients of the square transform block in FVC.

[0026] Figure 3B This is an explanatory diagram used to illustrate the zeroing of the transform coefficients of a non-square transform block in an FVC.

[0027] Figure 4 This is an illustrative diagram illustrating an example of a basic implementation of the technology according to this disclosure on the decoder side.

[0028] Figure 5 This is an illustrative diagram used to explain the generation of the quantization matrix for a transform block that has not been zeroed out.

[0029] Figure 6A This is a first explanatory diagram used to illustrate the generation of the quantization matrix for a transform block with zeroing applied.

[0030] Figure 6B This is a second explanatory diagram used to illustrate the generation of the quantization matrix for a transform block with zeroing applied.

[0031] Figure 6C This is the third explanatory diagram used to illustrate the generation of the quantization matrix for a transform block that has been zeroed out.

[0032] Figure 7 This is an illustrative diagram illustrating an example of the basic implementation of the technology according to this disclosure on the encoder side.

[0033] Figure 8 This is a block diagram illustrating an example configuration of the encoder according to the first embodiment.

[0034] Figure 9 It is shown by Figure 8 A flowchart illustrating an example of the quantization-related processing performed by the encoder.

[0035] Figure 10 This is a flowchart illustrating an example of the process for generating a quantization matrix.

[0036] Figure 11 This is a flowchart illustrating an example of the process of scaling list encoding.

[0037] Figure 12This is a block diagram illustrating an example configuration of the decoder according to the first embodiment.

[0038] Figure 13 It is shown by Figure 12 The flowchart is an example of the process of inverse quantization related processing performed by the decoder.

[0039] Figure 14 This is a flowchart illustrating an example of the process of decoding scaling list data.

[0040] Figure 15 This is a block diagram illustrating an example configuration of the encoder according to the second embodiment.

[0041] Figure 16 It is shown by Figure 15 A flowchart illustrating an example of the quantization-related processing performed by the encoder.

[0042] Figure 17 This is a block diagram illustrating an example configuration of the decoder according to the second embodiment.

[0043] Figure 18 It is shown by Figure 17 The flowchart is an example of the process of inverse quantization related processing performed by the decoder.

[0044] Figure 19 This is a block diagram illustrating an example of a hardware configuration. Detailed Implementation

[0045] In the following, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that in this specification and the drawings, components having substantially the same functional configuration will be indicated by the same reference numerals, and redundant descriptions will be omitted.

[0046] The scope of this disclosure is not limited to the content described in the following detailed description, and the content of references REF1 to REF3, which are publicly known at the time of filing of this disclosure, is also included within the scope of reference. In other words, the content of these references also provides a basis for determining whether the support requirements are met. For example, unless explicitly stated separately, the quadtree block structure described in reference REF2 and the quadtree plus binary tree (QTBT) block structure described in reference REF3 are outside the scope of this disclosure, these block structures are included within the scope of this disclosure. Similarly, technical terms such as “parse,” “syntax,” and “semantics” are also included within the scope of this disclosure, regardless of whether these terms are directly mentioned in the following detailed description, and the support requirements regarding the use of these terms in the claims are met.

[0047] • REF1: ITU-T H.264 Recommendation (04 / 2017) "Advanced video coding for generic audiovisual services", April 2017

[0048] REF2: ITU-T H.265 Recommendation (12 / 2016) "High efficiency video coding", December 2016

[0049] • REF3: J. Chen, E. Alshina, G. J. Sullivan, J. R. Ohm, and J. Boyce, “Algorithm Description of Joint Exploration Test Model (JEM7)”, 7th Meeting of the Joint Video Exploration Group (JVET) of JVET-G1001, ITU-T SG 16WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11: Turin, Italy, July 13-21, 2017

[0050] The following descriptions will be presented in the following order.

[0051] 1. Overview

[0052] 1-1. Definition of Terms

[0053] 1-2. Existing Technology and Its Problems

[0054] 1-3. Based on the technical principles of this disclosure

[0055] 1-4. Control Reset

[0056] 1-5. Controlling the size of the basic quantization matrix

[0057] 2. First Implementation Method

[0058] 2-1. Encoder Configuration

[0059] 2-2. Examples of Syntax and Semantics

[0060] 2-3. Select a reference quantization matrix

[0061] 2-4. Resampling Methods

[0062] 2-5. Flowchart of quantization-related processing performed during encoding

[0063] 2-6. Decoder Configuration

[0064] 2-7. The process of inverse quantization related processing performed during decoding

[0065] 3. Second Implementation Method

[0066] 3-1. Encoder Configuration

[0067] 3-2. Flow of quantization-related processing performed during encoding

[0068] 3-3. Decoder Configuration

[0069] 3-4. The process of inverse quantization related processing performed during decoding

[0070] 4. Hardware Configuration Example

[0071] 5. Summary

[0072] <1. Overview>

[0073] [1-1. Definition of Terms]

[0074] Some terms used in this specification are defined as follows.

[0075] A transform block is a block set up in an image and considered a processing unit in orthogonal and inverse orthogonal transformations. The size of a transform block is M×N, where M represents the number of components in the horizontal direction and N represents the number of components in the vertical direction (M and N are each integers equal to or greater than 2). When M=N, the transform block is square; when M≠N, the transform block is non-square.

[0076] Transform coefficients are the coefficients for each frequency component derived by transforming signal samples in the spatial domain to signal samples in the frequency domain. For example, a transformation from the spatial domain to the frequency domain can be performed as an orthogonal transform (or inverse orthogonal transform) such as the Discrete Cosine Transform (DCT) and the Discrete Sine Transform (DST). Typically, orthogonal transforming signal samples from a two-dimensional M×N transform block produces a two-dimensional array of the same size, i.e., a two-dimensional array of M×N transform coefficients. Typically, the component in the upper left corner of the resulting two-dimensional array is the direct current (DC) component, while components located further from the upper left corner correspond to higher frequencies.

[0077] A quantization matrix is ​​a matrix that expresses the quantization steps used to quantize a two-dimensional array of transform coefficients in the frequency domain by applying different quantization steps to different components. The elements of the quantization matrix do not always need to be quantization steps; instead, they can be, for example, parameters input to the formula used to derive the quantization steps.

[0078] A scaling list is a list of values ​​derived by transforming the elements of a quantization matrix into one-dimensional elements using a specific scan sequence. This means the scaling list is a form of information indicating the quantization matrix. Typically, to explicitly transmit a two-dimensional quantization matrix as a signal via an encoded stream, the quantization matrix is ​​transformed into a scaling list and then encoded.

[0079] [1-2. Existing Technology and Its Problems]

[0080] (1) Signal transmission of quantization matrix in HEVC

[0081] In HEVC, coding units (CUs) are set up in a quadtree pattern in the image as processing units in the coding process. CUs that select inter-frame prediction as the prediction type are directly partitioned to set up one or more transform units (TUs). On the other hand, when CUs select intra-frame prediction, each of the prediction units (PUs) that make up the CU is partitioned to set up one or more TUs. In HEVC, such a TU (transform unit) is equivalent to a transform block. The minimum size of a TU is 4×4, and the maximum size is 32×32. The encoder and decoder use such TUs as processing units to perform orthogonal transform / quantization and inverse orthogonal transform / inverse quantization, respectively. In HEVC, the quantization step can be uniform within the transform block or can vary depending on its location within the transform block (i.e., depending on the frequency components of the transform coefficients). For example, when the coded stream runs at the same bit rate, quantizing the high-frequency components of the transform coefficients more coarsely than quantizing the low-frequency components makes it possible to relatively suppress subjective image quality degradation.

[0082] The quantization steps, which differ depending on the position within the transform block, are expressed by elements of a quantization matrix whose size is equal to the size of the transform block. In HEVC, for each of the 4×4, 8×8, and 16×16 TU size candidates, six (=2×3) types of quantization matrices can be used, differing from each other in the combination of prediction type (Intra or Inter) and color components (Y, Cb, or Cr). For a 32×32 TU size, two types of quantization matrices can be used, differing from each other in the prediction type (Intra or Inter). The size of the quantization matrix is ​​identified by a size ID, and the combination of prediction type and color components of the quantization matrix is ​​identified by a matrix ID. Figure 1 The diagram illustrates the types of quantization matrices that can be used in HEVC.

[0083] Furthermore, in HEVC, the three existing types of quantization matrices are defined as follows.

[0084] • An existing flat quantization matrix of size 4×4 that does not vary depending on the prediction type and color components.

[0085] • An existing non-flat quantization matrix of size 8×8 used for intra-frame prediction that does not vary depending on the color components.

[0086] • An existing non-flat quantization matrix of size 8×8 used for inter-frame prediction that does not vary depending on the color components.

[0087] A 16×16 quantization matrix is ​​generated by upsampling the elements of an 8×8 quantization matrix using a nearest neighbor algorithm. A 32×32 quantization matrix is ​​generated by upsampling the elements of a 16×16 quantization matrix using the same nearest neighbor algorithm. In HEVC, when a different quantization matrix than the existing quantization matrix is ​​desired, a user-defined specific quantization matrix can be explicitly signaled. However, it should be noted that while quantization matrices of size 4×4 and 8×8 can be signaled globally by scanning all elements, signaling of 16×16 and 32×32 quantization matrices is achieved by signaling and upsampling the 8×8 quantization matrix. Note that the element values ​​of the DC components that make up specific quantization matrices of size 16×16 and 32×32 can be signaled individually.

[0088] When a quantization matrix is ​​explicitly transmitted via signaling, as described above, the elements of the two-dimensional quantization matrix are mapped to a scaling list, which is a one-dimensional array of elements, according to a specific scan sequence. Each element of the scaling list is then encoded as a difference value created from each element of the quantization matrix. The syntax for the scaling list data, which indicates the scaling list created in HEVC in the manner described above, is described in Section 7.3.4 of reference REF2.

[0089] (2) QTBT block structure in FVC

[0090] In FVC, the CUs are set up in QTBT mode in the image, and the TUs and CUs are treated together. More specifically, first, each of the coding tree units (CTUs) arranged in a grid pattern in the image is divided in a quadtree pattern, and square quadtree leaves of various sizes are formed as needed. Then, if necessary, each quadtree leaf is further divided in a binary tree pattern along either the horizontal or vertical boundary. Figure 2 This shows an example of QTBT block partitioning in FVC. Figure 2Image Im0 shown includes four CTUs, each with a size of 128×128. The upper left CTU comprises 13 CUs formed by four recursive QT partitionings. Of the 13 CUs, the smallest CU has a size of 8×8, and the largest CU has a size of 64×64. The lower left CTU comprises five CUs formed by four recursive BT partitionings. The upper right CTU comprises nine CUs formed through multiple recursive QT and BT partitionings. The lower right CTU is unpartitioned and therefore includes one CU. Figure 2 In the example, the minimum CU size is 8×8. However, square CUs or non-square CUs with side lengths of 2 or 4 are also allowed. In FVC, each of these CUs is also used as a transform block.

[0091] (3) Zeroing of high-frequency components in FVC

[0092] As understood from the above description, while the upper limit of the permissible transform block size (i.e., TU size) in HEVC is 32×32, the upper limit of the permissible transform block size in FVC is 128×128, a significant increase compared to 32×32. Such large transform blocks can be used, for example, in applications that efficiently encode high-definition video images known as "4K". However, it should be noted that despite the large data size of the transform blocks, the high-frequency components of the transform coefficients obtained by performing orthogonal transforms using transform blocks larger than 32×32 contribute almost nothing to the subjective image quality. Therefore, a zeroing method is employed in FVC. According to this method, in transform blocks with a length greater than 32 on one side, the 32nd high-frequency component and all other high-frequency components following it on the high-frequency side are set to zero (zeroed out). Figure 3A and Figure 3B The following are some examples of such zeroing performed in FVC.

[0093] Figure 3A Three square transform blocks B01, B02, and B03 are shown. Transform block B01 has a size of 32×32, and therefore zeroing is not applied to transform block B01. Transform block B02 has a size of 64×64. In this case, the transform coefficients of transform block B02 are set to zero except for the 32×32 transform coefficients in the upper left corner. Transform block B03 has a size of 128×128. In this case, the transform coefficients of transform block B03 are set to zero except for the 32×32 transform coefficients in the upper left corner. Besides the 32×32 square transform block, Figure 3B Nine non-square transform blocks of various sizes are also shown. (See example from...) Figure 3B As understood, according to FVC, along a side of length 64 or greater, 32 frequency components are maintained on the low-frequency side, while the remaining frequency components (belonging to the high-frequency side) are zero.

[0094] (4) Description of the problem

[0095] As mentioned above, in HEVC, signal transmission of 16×16 and 32×32 quantization matrices is achieved by signal transmission and upsampling of 8×8 quantization matrices to avoid the degradation in coding efficiency caused by signal transmission of quantization matrices. However, there is a trade-off between the degradation in coding efficiency caused by signal transmission of quantization matrices and the impact on device performance of generating different quantization matrices from a specific quantization matrix (e.g., through upsampling). In the case of FVC with significantly varying combinations of block size and shape, in particular, attempting to simply generate quantization matrices corresponding to these combinations from other quantization matrices leads to a significant increase in the processing costs required for quantization matrix generation (e.g., hardware resource consumption, processing latency, increased power consumption, etc.). This can potentially degrade the performance of devices such as encoders and decoders.

[0096] [1-3. Technical Principles Based on this Disclosure]

[0097] To address or at least mitigate the problems of the prior art described above, the present disclosure provides an improved system for efficiently generating or transmitting quantization matrices via signals.

[0098] (1) An example of how this technology is implemented on the decoder side.

[0099] Figure 4 This is an illustrative diagram illustrating an example of a basic implementation of the technology according to this disclosure on the decoder side. Figure 4 The processing steps S11 to S16 related to inverse quantization that can be performed by the decoder are shown.

[0100] • Step S11: Decode the encoded stream to parse some control data affecting inverse quantization and inverse quadrature transform, and generate quantization transform coefficients. Control parameters include, for example, block partitioning data defining block partitioning and scaling list (SL) data defining quantization matrices of one or more sizes.

[0101] • Step S12: Based on the block partitioning data, set up multiple transform blocks in the image in QTBT mode. The block partitioning data includes, for example, a set of parameters indicating the recursive QT partitioning or BT partitioning of each CTU. The transform blocks set at this step can have various shapes and sizes.

[0102] Step S13: Decode the scaling list data to generate a scaling list representing quantization matrices of one or more sizes, and transform the generated scaling list into those quantization matrices. For each size, multiple types of quantization matrices that differ from each other in terms of the combination of prediction type and color components can be generated. In this specification, the quantization matrix generated based on the scaling list data in the manner described above (i.e., the quantization matrix explicitly transmitted with a signal) is referred to as the basic quantization matrix (basic QM). The basic quantization matrix generated in step S13 is stored in the QM memory M11.

[0103] • Step S14: Generate each of one or more quantization matrices of a size other than the basic quantization matrix by referencing one of the other quantization matrices that have already been generated. In this manner, additional quantization matrices are generated by referencing another quantization matrix of a different size. In this specification, such a quantization matrix is ​​referred to as an additional quantization matrix (additional QM). The quantization matrix referenced when generating the additional quantization matrix is ​​referred to as a reference quantization matrix (reference QM). Typically, the additional quantization matrix can be generated by resampling the reference quantization matrix. When the size of one side of the additional quantization matrix is ​​larger than the size of the corresponding side of the reference quantization matrix, upsampling is performed along that side. Conversely, when the size of one side of the additional quantization matrix is ​​smaller than the size of the corresponding side of the reference quantization matrix, downsampling is performed along that side. The reference memory M12 stores at least some of the reference quantization matrices referenced in such upsampling and downsampling.

[0104] As described above, in FVC, the high-frequency components of transform blocks whose length on one side is greater than a certain threshold are forced to zero. When the size of the quantization matrix is ​​equal to the size of such a transform block with zeroing applied, it is not necessary to compute the elements of the quantization matrix corresponding to the zeroed transform coefficients. Therefore, allocating a large amount of memory resources to the computation of these elements would be wasteful. Considering this conclusion, the technology according to this disclosure introduces a method in which, when generating an additional quantization matrix with zeroing applied, the entire reference quantization matrix is ​​not referenced, but only a portion of the reference quantization matrix is ​​referenced, which covers the range that substantially contributes to the quantization of non-zero coefficients. This means that the operation of computing the elements corresponding to the positions of the zeroed coefficients is skipped, and the reference memory M12 does not need to store the elements of the reference quantization matrix outside the range of the aforementioned partial quantization matrix. The additional quantization matrix generated in step S14 is stored in the QM memory M11.

[0105] • Step S15: For each of the multiple transform blocks set in step S12, the quantized transform coefficients are inverse quantized to recover the transform coefficients. The quantization matrix used in step S15 is a quantization matrix selected from multiple quantization matrices stored in the QM memory M11, which corresponds to the object transform block in terms of the combination of prediction type and color components, and has a size equal to the size of the object transform block. Some transform coefficients of the transform blocks to which zeroing has been applied already have zero frequency components. Therefore, the inverse quantization of these frequency components can be skipped.

[0106] In one implementation, step S14 is included in step S15. In this implementation, the quantization matrix required for each transform block can be generated in a so-called "on-the-fly" mode during inverse quantization (if the quantization matrix has not yet been generated). In another implementation, step S14 can be performed before processing multiple transform blocks, such that the quantization matrices of all size candidates are pre-stored in the QM memory M11.

[0107] • Step S16: For each of the multiple transform blocks set in step S12, perform an inverse orthogonal transform on the transform coefficients in the frequency domain to recover the prediction error in the spatial domain. The prediction error recovered at this step can be combined with the predicted image through additional processing (not shown) to reconstruct the image.

[0108] (2) Various sampling

[0109] exist Figure 4 The process of generating an additional quantization matrix from the reference quantization matrix, performed at step S14, typically involves a combination of downsampling or upsampling of elements in the horizontal direction and downsampling or upsampling of elements in the vertical direction of the quantization matrix. Upsampling includes, for example, interpolating matrix elements using interpolation methods such as nearest neighbor algorithms, bilinear interpolation, and bicubic interpolation. Downsampling includes, for example, sparse matrix elements.

[0110] Figure 5 This is an illustrative diagram used to explain the generation of the quantization matrix for a transform block that has not been zeroed out. Figure 5 The upper left portion shows the reference quantization matrix B10. In this example, for simplicity, it is assumed that the reference quantization matrix B10 has a square shape and a size of N1×N1. It is also assumed that zeroing of high-frequency components is not applied to the transform block of size N1×N1.

[0111] exist Figure 5The quantization matrices B11, B12, B13, and B14 shown at the bottom are the additional quantization matrices to be generated. Additional quantization matrix B11 has a horizontal dimension smaller than N1, and its vertical dimension is also smaller than N1. Additional quantization matrix B11 corresponds to a zero-return transform block without applied high-frequency components. Therefore, additional quantization matrix B11 is generated by downsampling the entire reference quantization matrix B10 in both the horizontal and vertical directions. Additional quantization matrix B12 has a horizontal dimension smaller than N1, while its vertical dimension is larger than N1. Additional quantization matrix B12 corresponds to a zero-return transform block without applied high-frequency components. Therefore, additional quantization matrix B12 is generated by downsampling the entire reference quantization matrix B10 in the horizontal direction and upsampling it in the vertical direction. Additional quantization matrix B13 has a horizontal dimension larger than N1, while its vertical dimension is smaller than N1. The additional quantization matrix B13 corresponds to the transform block without zeroing out high-frequency components. Therefore, the additional quantization matrix B13 is generated by upsampling the entire reference quantization matrix B10 in the horizontal direction and downsampling it in the vertical direction. The additional quantization matrix B14 has a dimension greater than N1 in both the horizontal and vertical directions. The additional quantization matrix B14 corresponds to the transform block without zeroing out high-frequency components. Therefore, the additional quantization matrix B14 is generated by upsampling the entire reference quantization matrix B10 in both the horizontal and vertical directions. Figure 5 In this context, horizontal resampling (downsampling or upsampling) and vertical resampling are described as different operations. However, these resampling operations can be performed as a single operation.

[0112] Figures 6A to 6C This is an explanatory diagram illustrating the generation of the quantization matrix for a transform block with zeroing applied. As described above, an additional quantization matrix for the transform block with zeroing applied is generated by referencing only a portion of the reference quantization matrix. Figure 5 In the same way, for the sake of simplicity, in Figures 6A to 6C In this context, it is assumed that a square quantization matrix B10 with size N1×N1 is selected as the reference quantization matrix.

[0113] exist Figure 6A The lower part shows quantization matrices B21 and B22, which are the additional quantization matrices to be generated. The additional quantization matrix B21 has a horizontal dimension greater than N. TH Furthermore, the additional quantization matrix B21 has a dimension smaller than N1 in the vertical direction, N THThis represents the upper limit of the size of the zero-reduction matrix without the application of high-frequency components. The additional quantization matrix B21 corresponds to the transform block with the application of high-frequency components. Therefore, the additional quantization matrix B21 is generated by upsampling a portion of the reference quantization matrix B10 in the horizontal direction and downsampling a portion of the reference quantization matrix B10 in the vertical direction. In the horizontal direction, the size N of the partial matrix... PAR The ratio of the size N1 of the reference quantization matrix B10 to the size N of the non-zero part is equal to the size N of the non-zero part. TH The ratio of the size N2 of the additional quantization matrix B21. The size of the additional quantization matrix B21 in the horizontal direction is also greater than N. TH Furthermore, the additional quantization matrix B21 has a vertical dimension greater than N1 but less than N. TH N TH This represents the upper limit of the zero-reduction size without the application of high-frequency components. The additional quantization matrix B22 corresponds to the transform block with the application of high-frequency components. Therefore, the additional quantization matrix B22 is generated by upsampling a portion of the reference quantization matrix B10 in both the horizontal and vertical directions.

[0114] exist Figure 6B The lower part shows quantization matrices B31 and B32, which are the additional quantization matrices to be generated. Additional quantization matrix B31 has a horizontal dimension less than N1 and a vertical dimension greater than N. TH N TH This represents the upper limit of the size of the zero-reduction matrix without the application of high-frequency components. The additional quantization matrix B31 corresponds to the transform block with the application of high-frequency components. Therefore, the additional quantization matrix B31 is generated by downsampling a portion of the reference quantization matrix B10 in the horizontal direction and upsampling a portion of the reference quantization matrix B10 in the vertical direction. In the vertical direction, the size N of the partial matrix... PAR The ratio of the size N1 of the reference quantization matrix B10 to the size N of the non-zero part is equal to the size N of the non-zero part. TH The ratio of the size N2 of the additional quantization matrix B31. The size of the additional quantization matrix B32 in the horizontal direction is greater than N1 but less than N. TH Furthermore, the additional quantization matrix B32 has a dimension greater than N in the vertical direction. TH N TH This represents the upper limit of the zero-reduction size without the application of high-frequency components. The additional quantization matrix B32 corresponds to the transform block with the application of high-frequency components. Therefore, the additional quantization matrix B32 is generated by upsampling portions of the reference quantization matrix B10 in both the horizontal and vertical directions.

[0115] exist Figure 6CThe quantization matrix B41 shown at the bottom is the additional quantization matrix to be generated. The additional quantization matrix B41 has a horizontal dimension greater than N. TH Furthermore, the additional quantization matrix B41 also has a dimension greater than N in the vertical direction. TH N TH This represents the upper limit of the size of the zero-reduction matrix without the application of high-frequency components. The additional quantization matrix B41 corresponds to the transform block with the application of high-frequency components. Therefore, the additional quantization matrix B41 is generated by upsampling a portion of the reference quantization matrix B10 in both the horizontal and vertical directions. In the horizontal direction, the size N of the portion matrix... PAR_H The ratio of the size N1 of the reference quantization matrix B10 to the size N of the non-zero part is equal to the size N of the non-zero part. TH With the size N of the additional quantization matrix B41 2_H The ratio. In the vertical direction, the size N of the partial matrix. PAR_V The ratio of the size N1 of the reference quantization matrix B10 to the size N of the non-zero part is equal to the size N of the non-zero part. TH With the size N of the additional quantization matrix B41 2_V The ratio.

[0116] As in Figures 6A to 6C As shown in the example, when a quantization matrix that is not explicitly transmitted via signal is generated separately, only a portion of the reference quantization matrix is ​​referenced instead of the entire reference quantization matrix. This process reduces the cost of operations used to resample matrix elements and avoids wasting memory resources.

[0117] (3) An example of how this technology is implemented on the encoder side.

[0118] Figure 7 This is an illustrative diagram illustrating an example of the basic implementation of the technology according to this disclosure on the encoder side. Figure 7 The processing steps S21 to S27 related to quantization that can be performed by the encoder are shown.

[0119] • Step S21: Set up multiple transform blocks in the image using QTBT mode. For example, the block structure can be determined as a result of preliminary image analysis or a search for the optimal block structure. Additionally, a set of parameters representing the block structure set at this step (e.g., parameters indicating the recursive QT or BT partitioning of each CTU) is generated as block partitioning data.

[0120] • Step S22: Set a basic quantization matrix of one or more user-defined sizes (i.e., a quantization matrix different from existing quantization matrices). For each size, multiple types of quantization matrices, different from each other in terms of the combination of prediction type and color components, can be set. A particular type of quantization matrix can be the same as another type of quantization matrix. The basic quantization matrix set in step S22 is stored in the QM memory M21.

[0121] Step S23: Generate each of one or more quantization matrices (additional quantization matrices) of a size other than the basic quantization matrix by referencing one of the basic quantization matrices. (See above reference.) Figures 5 to 6C As described, an additional quantization matrix can be generated by resampling a reference quantization matrix. The reference memory M22 stores at least some of the reference quantization matrices referenced when generating the additional quantization matrix. In the encoder-side implementation of this technology, in the same manner as described above in the decoder-side implementation, when generating an additional quantization matrix with a zero-return size applied, instead of referencing the entire reference quantization matrix, only a portion of the reference quantization matrix is ​​referenced, covering the range that substantially contributes to the quantization of non-zero coefficients.

[0122] • Step S24: For each of the multiple transform blocks set in step S21, perform an orthogonal transform on the spatial domain prediction error to generate transform coefficients in the frequency domain.

[0123] • Step S25: For each of the multiple transform blocks set in step S21, the transform coefficients are quantized to generate quantized transform coefficients. The quantization matrix used in step S25 is a quantization matrix selected from multiple quantization matrices stored in the QM memory M21, which corresponds to the object transform block in terms of the combination of prediction type and color components, and has a size equal to the size of the object transform block. Some transform coefficients of the transform blocks to which zeroing has been applied already have zero frequency components. Therefore, the quantization of these frequency components can be skipped.

[0124] In one implementation, step S23 is included in step S25. In this implementation, the quantization matrix required for each transform block can be generated in a so-called "on-the-fly" mode during quantization (if the quantization matrix has not yet been generated). In another implementation, step S23 can be performed before processing multiple transform blocks, such that the quantization matrices of all size candidates are pre-stored in the QM memory M21.

[0125] As will be described later, encoders typically include a local decoder that performs inverse quantization. Although Figure 7The inverse quantization performed by the local decoder is not shown, but the same quantization matrix used in step S25 can also be used in that inverse quantization.

[0126] • Step S26: Transform the basic quantization matrix of one or more sizes set in step S22 into a scaling list, and then encode each one in the scaling list to generate scaling list data.

[0127] • Step S27: Encode the control parameters, including block partitioning data and scaling list data, as well as the quantization transform coefficients, to generate an encoded stream.

[0128] [1-4. Control Reset]

[0129] In the previous section, we have mainly described applying the zeroing of high-frequency components to a size greater than the upper limit N on at least one side. TH An example of a transform block. According to reference REF3, the upper bound N used in FVC is defined as a specification-based static definition of the upper bound. TH The value is 32. In this case, it is not necessary to encode the control parameters indicating which transform block to apply zeroing to. However, for more flexible control over zeroing, for example, the following control parameters can be encoded separately.

[0130] • Zeroing flag: When this flag is true, zeroing of high-frequency components is applied to the transform block associated with this flag. However, when this flag is false, zeroing of high-frequency components is not applied to the transform block associated with this flag.

[0131] • Zero-point size information: This information indicates the size of the transform block for applying zero-pointing with high-frequency components. Zero-point size information may include, for example, boundary values ​​(thresholds) equivalent to at least one of an upper and lower limit of the size for which zero-pointing is not applied (or applied). Zero-point size information may also include an index indicating the size for which zero-pointing is not applied (or applied). Zero-point size information may be encoded regardless of the presence or absence of a zero-pointing flag, or only when the zero-pointing flag is true.

[0132] The aforementioned control parameters for controlling zeroing can be encoded, for example, for each sequence, image, slice, tile, CTU, or transform block. In this way, by dynamically determining the application / non-application of zeroing or the size of the applied zeroing, images expressing even minute high-frequency components can be flexibly reproduced according to user needs or system requirements / constraints.

[0133] [1-5. Controlling the size of the basic quantization matrix]

[0134] As mentioned above, in HEVC, specific quantization matrices of sizes 16×16 and 32×32 are not transmitted directly with the signal, but are generated by upsampling the smaller quantization matrix. Specifically, HEVC has predefined, specification-based rules that stipulate that quantization matrices of a maximum size of 8×8 can be transmitted directly with the signal. Meanwhile, it is known that models based on human luminance sensitivity characteristics (e.g., see reference REF4 below) show that calculating the elements of the most efficient quantization matrix of size M×N (where M and N each represent powers of 2) results in a subset of smaller quantization matrices forming a larger quantization matrix. Therefore, as a method to obtain an optimal set of quantization matrices, downsampling the larger basic quantization matrix to generate additional quantization matrices, rather than upsampling the smaller basic quantization matrix to generate additional quantization matrices, is also considered advantageous.

[0135] • REF4: Long-Wen Chang, Ching-Yang Wang, and Shiuh-Ming Lee, “Designing JPEG quantization tables based on human visual system,” International Conference on Image Processing, ICIP 99, October 24-28, 1999

[0136] In some embodiments described later, size specification information is encoded, indicating the size of the quantization matrix generated from the scaling list data as the basic quantization matrix. Therefore, the size specification information specifies various sizes of the basic quantization matrix, thereby enabling the flexible use of various types of quantization matrices. Clearly, the technology according to this disclosure can also be applied to systems where the size of the quantization matrix is ​​predetermined based on specifications and directly transmitted as a signal, regardless of the size of the matrix.

[0137] The technical principles described so far according to this disclosure can be implemented by an image processing apparatus (encoder) that encodes a series of images constituting a video, or by an image processing apparatus (decoder) that decodes an encoded bitstream to reconstruct the video. An image processing apparatus can have the corresponding functions of both an encoder and a decoder. The image processing apparatus can be any type of device, such as an image capture apparatus, a video recorder, a playback apparatus, a display apparatus, and an information processor. Two specific embodiments of such an image processing apparatus will be described in detail below.

[0138] <2. First Implementation Method>

[0139] [2-1. Encoder Configuration]

[0140] (1) Overall configuration

[0141] Figure 8 This is a block diagram illustrating an example configuration of an image processing apparatus 10a according to a first embodiment, which has the function of an encoder. (Refer to...) Figure 8 The image processing apparatus 10a is shown to include an encoding control unit 11, a rearrangement buffer 12, a subtraction unit 13, an orthogonal transformation unit 14, a quantization unit 15a, a reversible encoding unit 16, an accumulation buffer 17, a rate control unit 18, an inverse quantization unit 21, an inverse orthogonal transformation unit 22, an addition unit 23, an in-loop filter 24, a frame memory 25, a switch 26, an intra-frame prediction unit 30, an inter-frame prediction unit 35, a mode setting unit 40, and a QM storage unit 115a.

[0142] The encoding control unit 11 controls the overall encoder function of the image processing apparatus 10a, which will be described in detail below. According to this embodiment, the encoding control unit 11 includes a block setting unit 111 and a basic QM setting unit 113. The block setting unit 111 performs the functions already described above. Figure 7 The module described is the block setup processing step S21. The basic QM setup unit 113 executes the steps already referred to above. Figure 7 The modules described are the basic QM setup processing steps S22. These modules will be described further later.

[0143] The rearrangement buffer 12 rearranges a series of images that make up the video to be encoded according to a given group of pictures (GOP) structure. The rearrangement buffer 12 outputs the rearranged images to the subtraction unit 13, the intra-frame prediction unit 30, and the inter-frame prediction unit 35.

[0144] Subtraction unit 13 calculates the prediction error, which represents the difference between the input image (original image) from rearrangement buffer 12 and the prediction image, and subtraction unit 13 outputs the calculated prediction error to orthogonal transformation unit 14.

[0145] The quadrature transform unit 14 performs an orthogonal transform on each of one or more transform blocks set in the image to be encoded. This orthogonal transform can be performed, for example, as a discrete cosine transform (DCT) or a discrete sine transform (DST). More specifically, the quadrature transform unit 14 performs an orthogonal transform on signal samples in the spatial domain for each transform block to generate transform coefficients in the frequency domain, where the signal samples represent the incoming prediction error from the subtraction unit 13. Additionally, under the control of the encoding control unit 11, the quadrature transform unit 14 applies zeroing to the high-frequency components of transform blocks of a specific size to make the high-frequency components zero. For example, in a transform block with a side length greater than 32, the 32nd high-frequency component and other high-frequency components following it on the high-frequency side can be made zero. The quadrature transform unit 14 outputs the generated transform coefficients to the quantization unit 15a.

[0146] Quantization unit 15a is provided with input transform coefficients from orthogonal transform unit 14 and a rate control signal from rate control unit 18, which will be described later. For each of one or more transform blocks in the image to be encoded, quantization unit 15a quantizes the transform coefficients using a quantization matrix of size equal to the size of the transform block to generate quantized transform coefficients (which will also be referred to as "quantized data" below). Under the control of encoding control unit 11, quantization unit 15a skips the quantization of frequency components that are zero in the transform coefficients. Then, quantization unit 15a outputs the generated quantized data to reversible encoding unit 16 and inverse quantization unit 21. In addition, quantization unit 15a switches the quantization steps based on the rate control signal from rate control unit 18, thereby changing the bit rate of the quantized data. According to this embodiment, quantization unit 15a includes QM generation unit 117a. QM generation unit 117a performs the functions described above. Figure 7 The module described in QM generation process step S23. QM generation unit 117a includes... Figure 8 The reference memory M22 is not shown in the diagram. This module will be described further later.

[0147] The reversible encoding unit 16 encodes the incoming quantized data from the quantization unit 15a to generate an encoded stream. The reversible encoding unit 16 also encodes various control parameters referenced by the decoder and inserts these encoding parameters into the encoded stream. The control parameters at this time include, for example, the block partitioning data and scaling list (or scaling list data) mentioned above. The reversible encoding unit 16 outputs the generated encoded stream to the accumulation buffer 17. According to this embodiment, the reversible encoding unit 16 includes an SL encoding unit 119. The SL encoding unit 119 performs the operations already referenced above. Figure 7 The module described is for the QM transform / SL data generation and processing step S26. This module will be described further later.

[0148] The accumulation buffer 17 temporarily stores the incoming encoded stream from the reversible encoding unit 16 using a storage medium. Then, the accumulation buffer 17 outputs the accumulated encoded stream to a transmission unit (not shown), which is, for example, a communication interface or an interface connected to a peripheral device, at a bit rate corresponding to the bandwidth in the transmission path.

[0149] The rate control unit 18 monitors the free space of the accumulation buffer 17. The rate control unit 18 then generates a rate control signal based on the free space of the accumulation buffer 17 and outputs the generated rate control signal to the quantization unit 15a. For example, when the free space of the accumulation buffer 17 is found to be small, the rate control unit 18 generates a rate control signal to reduce the bit rate of the quantized data. When the free space of the accumulation buffer 17 is found to be large enough, the rate control unit 18 generates a rate control signal to increase the bit rate of the quantized data.

[0150] The inverse quantization unit 21, the inverse quadrature transform unit 22, and the addition unit 23 constitute the local decoder. The local decoder is responsible for decoding the encoded data to reconstruct the image.

[0151] For each transform block, the inverse quantization unit 21 performs inverse quantization on the quantized transform coefficients using the same quantization matrix as that used by the quantization unit 15a to recover the transform coefficients. The inverse quantization unit 21 skips the inverse quantization of frequency components that are forced to zero included in the quantized transform coefficients. Then, the inverse quantization unit 21 outputs the recovered transform coefficients to the inverse quadrature transform unit 22.

[0152] For each transform block, the inverse quadrature transform unit 22 performs an inverse quadrature transform. More specifically, for each transform block, the inverse quadrature transform unit 22 performs an inverse quadrature transform on the transform coefficients in the frequency domain from the inverse quantization unit 21, thereby recovering the prediction error in the form of signal samples in the spatial domain. The inverse quadrature transform unit 22 then outputs the recovered prediction error to the adder unit 23.

[0153] Adding unit 23 adds the recovered prediction error from inverse quadrature transform unit 22 to the prediction image from intra-frame prediction unit 30 or inter-frame prediction unit 35 to reconstruct the decoded image. Then, adding unit 23 outputs the reconstructed decoded image to in-loop filter 24 and frame memory 25.

[0154] The in-loop filter 24 consists of a series of filters applied to the decoded image to improve image quality. The in-loop filter 24 may include one or more of, for example, bilateral filters, deblocking filters, adaptive offset filters, and adaptive loop filters described in reference REF3. The in-loop filter 24 outputs the decoded image, which has been filtered through the series of filters, to the frame memory 25.

[0155] The frame memory 25 stores the decoded image before filtering from the addition unit 23 and the decoded image after filtering from the in-loop filter 24.

[0156] Switch 26 reads the decoded image before filtering for intra-frame prediction from frame memory 25 and provides the read decoded image as a reference image to intra-frame prediction unit 30. Switch 26 also reads the decoded image after filtering for inter-frame prediction from frame memory 25 and provides the read decoded image as a reference image to inter-frame prediction unit 35.

[0157] Intra-prediction unit 30 performs intra-prediction processing based on the original image and the decoded image. For example, intra-prediction unit 30 evaluates the cost based on the prediction error and the amount of generated code for each of the prediction mode candidates included in the search range. Then, intra-prediction unit 30 selects the prediction mode that minimizes the cost as the optimal prediction mode. In addition, intra-prediction unit 30 generates a prediction image based on the selected optimal prediction mode. Then, intra-prediction unit 30 outputs the prediction image and its corresponding cost, along with some control parameters containing prediction mode information, to mode setting unit 40.

[0158] Inter-frame prediction unit 35 performs inter-frame prediction processing (motion compensation) based on the original image and the decoded image. For example, inter-frame prediction unit 35 evaluates the cost based on the prediction error and the amount of generated code for each of the prediction mode candidates included in the search range. Then, inter-frame prediction unit 35 selects the prediction mode that minimizes the cost as the optimal prediction mode. In addition, inter-frame prediction unit 35 generates a prediction image based on the selected optimal prediction mode. Then, inter-frame prediction unit 35 outputs the prediction image and its corresponding cost, along with some control parameters containing prediction mode information, to mode setting unit 40.

[0159] Based on a comparison between the input cost from the intra-prediction unit 30 and the input cost from the inter-prediction unit 35, the mode setting unit 40 sets the prediction type for each block. For blocks whose prediction type is set to intra-prediction, the mode setting unit 40 outputs the prediction image generated by the intra-prediction unit 30 to the subtraction unit 13 and the addition unit 23. For blocks whose prediction type is set to inter-prediction, the mode setting unit 40 outputs the prediction image generated by the inter-prediction unit 35 to the subtraction unit 13 and the addition unit 23. Additionally, the mode setting unit 40 outputs the control parameters to be encoded to the reversible coding unit 16.

[0160] (2) Functions related to quantization matrix generation

[0161] exist Figure 8Among the various components shown, the block setting unit 111, the basic QM setting unit 113, the QM storage unit 115a, the QM generation unit 117a, and the SL encoding unit 119 are mainly involved in the generation of the quantization matrix performed by the encoder.

[0162] The block setting unit 111 divides each image into multiple transform blocks using QTBT block partitioning, thereby setting multiple transform blocks in each of a series of images. Additionally, the block setting unit 111 generates block partitioning data defining the block structure of the set transform blocks and outputs the generated block partitioning data to the reversible encoding unit 16. The size of the transform blocks set by the block setting unit 111 can be, for example, in the range of 2×2 to 128×128. The shape of the transform blocks can be square or non-square. Figure 2 The image shows some examples of the shape and size of the transform block.

[0163] The basic QM setting unit 113 sets one or more basic quantization matrices of different sizes as quantization matrices used by the image processing device 10a. Typically, the basic quantization matrix has at least one element whose value differs from the value of an element of an existing quantization matrix as defined by the FVC specification. For example, the values ​​of elements of the basic quantization matrix can be determined as a result of preliminary image analysis or parameter tuning. For each size, the basic QM setting unit 113 can set multiple types of quantization matrices that differ from each other in terms of the combination of prediction type and color components. A particular type of quantization matrix can be the same as another type of quantization matrix. In one example, the basic quantization matrix includes only square quantization matrices. In another example, the basic quantization matrix includes both square and non-square quantization matrices. In examples of the syntax described later, the size of the quantization matrix is ​​identified by a size ID, and the type of the quantization matrix is ​​identified by a matrix ID.

[0164] QM storage unit 115a is a storage module that stores various types of quantization matrices of various sizes, which are used by image processing device 10a. The quantization matrices stored in QM storage unit 115a include a basic quantization matrix set by basic QM setting unit 113 and additional quantization matrices generated by QM generation unit 117a, which will be described later. According to this embodiment, the basic quantization matrix is ​​set before performing orthogonal transformation and quantization across multiple transform blocks, and the basic quantization matrix is ​​stored in QM storage unit 115a through these processes of orthogonal transformation and quantization. On the other hand, when the transform coefficients of each transform block are quantized, additional quantization matrices are generated as needed and stored in QM storage unit 115a. QM storage unit 115a can manage matrix management information, which is internal control information indicating the size of existing quantization matrices. The matrix management information consists of, for example, a set of tags indicating whether (i.e., for example, already generated) a quantization matrix identified by two size IDs corresponding to the horizontal and vertical dimensions, respectively.

[0165] When quantization unit 15a quantizes the transform coefficients of each transform block, QM generation unit 117a determines whether a quantization matrix with a size equal to the transform block size has been generated by referring to the matrix management information provided by QM storage unit 115a. If a quantization matrix with a size equal to the transform block size has been generated, QM generation unit 117a reads the generated quantization matrix from QM storage unit 115a. Conversely, if a quantization matrix with a size equal to the transform block size has not yet been generated, QM generation unit 117a selects one of the generated basic quantization matrices as a reference quantization matrix and resamples the selected reference quantization matrix to generate an additional quantization matrix. QM generation unit 117a includes a memory in which the reference quantization matrix to be resampled, or a portion thereof, is temporarily stored.

[0166] For example, when zeroing of high-frequency components is not applied to a transform block of the object size, QM generation unit 117a generates a quantization matrix for the transform block by referencing the entire reference quantization matrix. On the other hand, when zeroing of high-frequency components is applied to a transform block of another object size, QM generation unit 117a generates a quantization matrix for the transform block by referencing only a portion of the reference quantization matrix. In the latter case, the ratio of the size of the referenced portion (i.e., the portion temporarily stored in the memory of QM generation unit 117a) to the size of the reference quantization matrix is ​​equal to the ratio of the size of the non-zero portion to the size of the generated additional quantization matrix. In this manner, quantization unit 15a uses one of various quantization matrices generated in a resource-efficient manner to quantize the transform coefficients of each transform block.

[0167] In one example, QM generation unit 117a can determine whether to apply zeroing to a transform block based on a specification-based rule that predefines the size of the transform block to which zeroing is to be applied. In another example, QM generation unit 117a can determine, under the control of encoding control unit 11, whether to apply zeroing to one transform block but not another. In the latter example, one or both of the aforementioned zeroing flag and zeroing size information can be encoded as control parameters and inserted into the encoded stream, wherein the zeroing flag indicates whether zeroing is to be applied to a transform block, and the zeroing size information indicates the size of the transform block to which zeroing is to be applied.

[0168] The SL encoding unit 119 encodes the scaling list representing the basic quantization matrix set by the basic QM setting unit 113 to generate scaling list data. The scaling list data is inserted into the encoded stream generated by the reversible encoding unit 16.

[0169] In the example, the SL encoding unit 119 includes size specification information in the scaling list data, which indicates the size of the quantization matrix explicitly transmitted via the scaling list data as the basic quantization matrix. When the number of sizes of the quantization matrix transmitted via the scaling list data changes, the SL encoding unit 119 may also include size count information in the scaling list data, which indicates the number of sizes to be transmitted via signaling. In this case, a bitstream constraint can be imposed to stipulate that when two or more sizes are transmitted via signaling, these sizes should be different from each other (i.e., they should be identified by different size IDs). Such a bitstream constraint prevents the encoder from redundantly encoding the size specification information, thereby reducing coding overhead and avoiding resource waste.

[0170] In another example, the size of the basic quantization matrix, explicitly transmitted via signaling through the scaling list data, is predefined according to a specification-based rule. In this case, the SL encoding unit 119 does not include the aforementioned size specification information and size count information in the scaling list data.

[0171] [2-2. Examples of Syntax and Semantics]

[0172] Table 1 below shows the syntax for HEVC scaling list data as described in reference REF2.

[0173] Table 1: Syntax for scaling list data in HEVC (excerpt from REF2, Section 7.3.4)

[0174]

[0175] HEVC scaling list data includes one or more of the following control parameters for each of the four sizes and six types, identified by size ID (“sizeId”) and matrix ID (“matrixId”).

[0176] ·scaling_list_pred_mode_flag[sizeId][matrixId]

[0177] ·scaling_list_pred_matrix_id_delta[sizeId][matrixId]

[0178] ·scaling_list_dc_coef_minus8[sizeId-2][matrixId]

[0179] ·scaling_list_delta_coef

[0180] "scaling_list_pred_mode_flag[sizeId][matrixId]" is a control flag used to switch the encoding method used for scaling the list. By referencing a quantization matrix of another type specified by "scaling_list_pred_matrix_id_delta[sizeId][matrixId]", quantization matrices of the type for which this control flag is set to false are simply encoded. On the other hand, differential encoding is performed on quantization matrices for which the aforementioned control flag is set to true, using "scaling_list_dc_coef_minus8[sizeId-2][matrixId]" and multiple "scaling_list_delta_coef". The number of "scaling_list_delta_coef" is indicated by the intermediate variable "coefNum".

[0181] Table 2 below shows examples of the syntax for the scaling list data that can be modified in this implementation. The examples in Table 2 include size specification information indicating the size of the quantization matrix generated as the basic quantization matrix, which has already been described in [1-5. Controlling the Size of the Basic Quantization Matrix].

[0182] Table 2: Examples of syntax for modifying scaling list data

[0183]

[0184] Specifically, the parameter "size_id_minusX" in the second row of Table 2 represents the size specification information. The value of the parameter "size_id_minusX" is given by subtracting the preset offset value X from the actual size ID.

[0185] Table 3 below shows examples of dimension ID definitions that can be modified from the dimension ID definitions in HEVC.

[0186] Table 3: Examples of Modified Size ID Definitions

[0187] Size of the square quantization matrix Size ID 2×2 0 4×4 1 8×8 2 16×16 3 32×32 4 64×64 5 128×128 6

[0188] In the example in Table 3, indices 0 to 6, used as size IDs, are assigned to candidate values ​​(2 to 128) for the size of one side of the square quantization matrix in descending order, with the smallest index at the top. In this example, the relationship between the candidate value N for the size of one side and the value “sizeId” of the size ID is given by the following formula.

[0189] N = 1 << (sizeId + 1)

[0190] However, it should be noted that the relationship between the candidate size value and the size ID is not limited to the relationship defined by the above formula.

[0191] The size of a non-square quantization matrix can be determined by specifying a dimension ID in both the horizontal and vertical directions. In the example syntax in Table 2, the size specification includes only one parameter, "size_id_minusX". This means that only square quantization matrices are explicitly signaled as the basic quantization matrix. It also means that quantization matrices whose size is identified by a dimension ID smaller than the offset value X are not signaled. Not limited to the above example, the size specification can include two parameters, each indicating a dimension ID used to identify the non-square basic quantization matrix. Furthermore, the offset subtraction can be omitted, in which case the dimension ID without subtracting the offset value is directly encoded.

[0192] Table 4 below shows examples of matrix ID definitions that can be modified from the matrix ID definition in HEVC.

[0193] Table 4: Examples of Modified Matrix ID Definitions

[0194] Prediction type Component ID (Color Component) Matrix ID Intra-frame 0(Y) 0 Intra-frame 1(Cb) 1 Intra-frame 2(Cr) 2 Inter-frame 0(Y) 3 Inter-frame 1(Cb) 4 Inter-frame 2(Cr) 5

[0195] According to already referenced Figure 1The definition of matrix ID in the described HEVC becomes an exception when size ID = 3 (32×32). However, in the examples in Table 4, the definition of matrix ID remains the general definition, regardless of the value of size ID. This fact causes a difference between the matrix ID loop in the third row of the syntax in Table 1 and the matrix ID loop in the fourth row of the syntax in Table 2. The maximum number of element values ​​to be differentially encoded, indicated by the intermediate variable "coefNum", is 64 in both the syntax in Table 1 and the syntax in Table 2. However, this maximum number can be changed or varied to allow for more flexible quantization matrix design.

[0196] [2-3. Selecting a reference quantization matrix]

[0197] The quantization matrix to be referenced when generating an additional quantization matrix can be determined according to any given rule. In a simple example, the largest quantization matrix available from the square quantization matrices can be selected as the reference quantization matrix. In this example, the size ID of the largest quantization matrix to be explicitly signaled is defined as "maxSignaledSizeId", and the matrix management information "QMAvailFlag[sizeId][matrixId]" indicates whether a quantization matrix identified by a specific combination of size ID and matrix ID exists (in this case, when "QMAvailFlag[sizeId][matrixId]" is true, a quantization matrix identified by this combination exists, and when "QMAvailFlag[sizeId][matrixId]" is false, a quantization matrix identified by this combination does not exist). Therefore, the reference size ID "refSizeId" indicating the size of the reference quantization matrix can be determined by the following pseudocode.

[0198] Table 5: Examples of pseudocode used to determine the size of the reference quantization matrix

[0199]

[0200] In another example, a reference quantization matrix can be selected from the available quantization matrices, based on the size that differs least from the size of the additional quantization matrix to be generated. Furthermore, reference quantization matrix information indicating which quantization matrix should be referenced can be encoded separately.

[0201] [2-4. Resampling Methods]

[0202] Then, an example describing a specific method for resampling to generate an additional quantization matrix will be given. In this example, for simplicity, it is assumed that the size of at least one side of the object transformation block is greater than the upper bound N. THUnder the given conditions, the zeroing of high-frequency components is applied to the object transformation block. The following description can also be applied to situations where the zeroing conditions are different, but with slight modifications that will be apparent to those skilled in the art.

[0203] First, assume that a first quantization matrix of a first size is chosen as the reference quantization matrix for generating a second quantization matrix of a second size. In this assumed case, the first size is W1×H1, and the second size is W2×H2. A flag “zoFlag” can be set using one of the following two formulas, indicating whether zeroing of high-frequency components is applied to the transform block of the second size.

[0204] zoFlag = max(W2, H2) > N TH

[0205] zoFlag = W2 > N TH ||H2>N TH

[0206] According to these formulas, when the zeroing of high-frequency components is applied to the transform block of the second size (W2×H2), the flag "zoFlag" is set to 1, while when the zeroing is not applied to the transform block of the second size (W2×H2), the flag "zoFlag" is set to 0.

[0207] When the range of elements actually generated by resampling of the second quantization matrix is ​​limited such that its range in the horizontal direction is R... WIDTH2 And the range in the vertical direction is R. HEIGHT2 When these ranges are given by the following formula.

[0208] W R2 =min(W2, N) TH )

[0209] H R2 =min(H2, N) TH )

[0210] R WIDTH2 =[O, W R2 -1]

[0211] R HEIGHT2 =[O, H R2 -1]

[0212] In these formulas, W R2 and H R2 These represent the number of elements included in the horizontal range and the number of elements included in the vertical range, respectively. The ratio r of the dimension of the non-zero portion (the portion not subject to zeroing) in the horizontal direction to the second dimension (W² × H²) is also mentioned. WIDTH2And the ratio r of the dimension of the non-zero portion (the portion not subject to zeroing) in the vertical direction to the second dimension (W2×H2). HEIGHT2 It can be given by the following formula.

[0213] r WIDTH2 =W R2 / W2

[0214] r HEIGHT2 =H R2 / H2

[0215] Using these ratios r WIDTH2 and r HEIGHT2 The horizontal range R of a portion of the first quantization matrix referenced when generating the non-zero portion of the second quantization matrix. WIDTH1 and the range R in the vertical direction of a portion of the first quantization matrix referenced when generating the non-zero portion of the second quantization matrix. HEIGHT1 It can be exported as follows.

[0216] W R1 =W1·r WIDTH2

[0217] H R1 =H1·r HEIGHT2

[0218] R WIDTH1 =[0, W R1 -1]

[0219] R HEIGHT1 =[0, H R1 -1]

[0220] In these formulas, W R1 and H R1 These represent the number of elements included in the horizontal range and the number of elements included in the vertical range, respectively.

[0221] As understood from the above description, when zeroing is applied to the object transform block, a second quantization matrix for quantizing the transform coefficients of the object transform block (or inverse quantizing the quantized transform coefficients of the object transform block) can be generated by referring only to a portion of the first quantization matrix of the first size W1×H1. On the other hand, when zeroing is not applied to the object transform block, the result is W R2 =W2 and H R2 =H2, thus obtaining r WIDTH2 =r HEIGHT2 =1, thus giving W R1 =W1 and H R1=H1. In this case, in order to generate the second quantization matrix, the entire first quantization matrix is ​​referenced. Therefore, when the zeroing of high-frequency components is applied to the object transformation block, the QM generation unit 117a reads only all elements of the reference quantization matrix from the QM storage unit 115a, including those in the reference range R. WITTH1 and R HEIGHHT1 The elements in the QM generation unit 117a are resampled, and the read elements are buffered. Without applying zeroing to the object transform block, the QM generation unit 117a reads all elements of the reference quantization matrix from the QM storage unit 115a, and buffers the read elements.

[0222] When the ratio of the first dimension to the second dimension in the horizontal direction is expressed as s WIDTH The ratio of the first dimension to the second dimension in the vertical direction is expressed as S. HEIGHT When, the size ratio S is given below. WIDTH and S HEIGHT .

[0223] s WIDTH =W1 / W2

[0224] s HEIGHT =H1 / H2

[0225] For example, a size ratio of s can be used WIDTH and s HEIGHT The following formula expresses the results of the nearest neighbor algorithm on the first quantization matrix Q. REF Upsampling is performed to derive the second quantization matrix Q. ADD Element Q ADD The processing of [j][i]. Note that j and i represent row and column indices, respectively. Assume that all elements of the second quantization matrix are reset to 0 before resampling.

[0226] j′=Floor(j·s WIDTH )

[0227] i′=Floor(i·s HEIGHT )

[0228] Q ADD [j][i]=Q REF [j,][i′](for i∈R) WIDTH2 , j∈R HEIGHT2 )

[0229] In these formulas, Floor(x) represents a function that returns the largest integer that is equal to or less than the parameter x.

[0230] The first quantization matrix Q can be expressed by the following formula. REF Perform downsampling to derive the second quantization matrix QADD Element Q ADD The processing of [j][i]. It is assumed that all elements of the second quantization matrix are reset to 0 before resampling is performed.

[0231] Q ADD [j][i]=Q REF [j·S WIDTH ][i·s HEIGHT (For i∈R) WIDTH2 , j∈R HEIGHT2 )

[0232] The resampling methods described in this section are merely examples. For instance, upsampling could be performed in one direction of the matrix, while downsampling could be performed in the other. Obviously, resampling could also be omitted in one direction of the matrix. Furthermore, in upsampling, other interpolation methods such as bilinear interpolation and bicubic interpolation could be used instead of the nearest neighbor algorithms mentioned above.

[0233] [2-5. The flow of quantization-related processing performed during encoding]

[0234] (1) Overall process

[0235] Figure 9 It is shown by Figure 8 A flowchart illustrating an example of the quantization-related processing flow performed by the image processing apparatus 10a. For the purpose of simplicity, quantization-independent processing steps among the various processing steps included in the encoding process are not described in this flowchart.

[0236] First, the basic QM setting unit 113 sets one or more basic quantization matrices, including a first quantization matrix of a first size (step S111). These basic quantization matrices are stored in the QM storage unit 115a.

[0237] For each of the multiple transformation blocks set in the image by the block setting unit 111 through QTBT block division, the following series of processing steps S113 to S116 (step S112) are repeated. Each transformation block for which these processing steps are repeated is called an object transformation block.

[0238] QM generation unit 117a first determines, for example, by referring to matrix management information provided by QM storage unit 115a whether a quantization matrix of the size corresponding to the block size of the object transform block exists (step S113). When no such quantization matrix exists, QM generation unit 117a performs a quantization matrix generation process, which will be described later, to generate an additional quantization matrix from the reference quantization matrix (step S114). The additional quantization matrix generated at this step is stored in QM storage unit 115a. When the expected quantization matrix already exists, QM generation unit 117a reads the quantization matrix of the size corresponding to the block size of the object transform block from QM storage unit 115a (step S115). Subsequently, quantization unit 15a quantizes the transform coefficients of the object transform block using the additionally generated quantization matrix or the quantization matrix read from QM storage unit 115a (step S116).

[0239] The SL encoding unit 119 converts each of the one or more basic quantization matrices set in step S111 into a one-dimensional code, thereby transforming each of the basic quantization matrices into a scaling list (step S117). The SL encoding unit 119 then performs scaling list encoding processing, which will be described later, to generate scaling list data (step S118).

[0240] The reversible encoding unit 16 encodes the quantized transform coefficients generated at step S116 for each transform block, so that the encoded quantized transform coefficients, along with the scaling list data, become part of the encoded stream. (Not in...) Figure 9 The process is illustrated below. Scaling list data can be updated in any kind of update unit, such as sequences, images, slices, and tiles.

[0241] (2) Quantization matrix generation process

[0242] Figure 10 It shows that it can be shown Figure 9 The flowchart is an example of the process of performing the quantization matrix generation process at step S114.

[0243] First, QM generation unit 117a selects a reference quantization matrix that should be referenced when generating the quantization matrix for the object transform block (step S121). In one example, the reference quantization matrix can be selected according to predefined specification-based rules (e.g., rules for selecting the quantization matrix with the largest size or the size closest to the object transform block size from the available quantization matrices). In another example, the reference quantization matrix can be selected dynamically.

[0244] Subsequently, QM generation unit 117a determines whether to apply zeroing of high-frequency components to the object transform block (step S122). In one example, QM generation unit 117a may determine whether to apply zeroing of high-frequency components to the object transform block based on predefined norm-based rules (e.g., rules based on whether the length of at least one side of the object transform block is greater than a specific threshold). In another example, the determination regarding whether to apply zeroing of high-frequency components to the object transform block may be dynamically changed.

[0245] Without applying zeroing to the object transform block, QM generation unit 117a reads the entire reference quantization matrix selected in step S121 from QM storage unit 115a and buffers it by storing the read reference quantization matrix in internal memory (step S123). Then, QM generation unit 117a resamples the read reference quantization matrix to generate an additional quantization matrix (step S124).

[0246] On the other hand, when zeroing is applied to the object transformation block, the QM generation unit 117a calculates the ratio of the size of the non-zero portion to the size of the object transformation block (e.g., the ratio r mentioned above). WIDTH2 and r HEIGHT2 (Step S125). Subsequently, based on the calculated ratio, the QM generation unit 117a reads a portion of the selected reference quantization matrix from the QM storage unit 115a and buffers it by storing the read portion in internal memory (step S126). The QM generation unit 117a then resamples the read portion to generate the non-zero portion of the additional quantization matrix (step S127).

[0247] QM generation unit 117a then stores the generated additional quantization matrix in QM storage unit 115a (step S128).

[0248] (3) Encoding processing of scaled lists

[0249] Figure 11 It shows that it can be shown Figure 9 The flowchart is an example of the process of scaling list encoding processing performed at step S118.

[0250] First, the SL encoding unit 119 encodes the dimension ID used to identify the size of the basic quantization matrix to generate dimension specification information (step S131). When the basic quantization matrix is ​​always a square matrix, one dimension ID is encoded for each size, as shown in Table 3. However, when the basic quantization matrix includes non-square matrices, two dimension IDs corresponding to the two directions can be encoded for each size. Dimension specification information can be generated by subtracting a preset offset value from the dimension ID.

[0251] For each combination of prediction type and color component, i.e., for each quantization matrix type identified by matrix ID, the following series of processing steps S133 to S136 (step S132) are repeated. The matrix ID for which the processing steps are repeated is called the object matrix ID.

[0252] SL encoding unit 119 determines whether to explicitly encode a series of element values ​​of the scaling list associated with the object matrix ID (step S133). In other words, SL encoding unit 119 determines the encoding method for the scaling list. For example, if the scaling list associated with the object matrix ID is the same as the scaling list associated with different matrix IDs (for matrices with the same size ID), then SL encoding unit 119 can choose a simpler method that encodes only the reference scaling information instead of the element values.

[0253] When such a simpler encoding method is selected, the SL encoding unit 119 determines the reference scaling list (step S134) and encodes the reference scaling list information indicating the determined reference scaling list (step S135).

[0254] On the other hand, when choosing a method to encode a series of element values, the SL coding unit 119 uses, for example, differential pulse code modulation (DPCM) to encode the values ​​in... Figure 9 The series of element values ​​of the scaling list obtained in step S117 are encoded to generate scaling list data (step S136).

[0255] Figure 11 The flowchart shown depicts an example where the size ID of the basic quantization matrix is ​​explicitly encoded. In another example, the encoding of the size of the basic quantization matrix can be omitted. In yet another example, the number of dimensions to be transmitted by signaling can also be encoded as size count information.

[0256] [2-6. Decoder Configuration]

[0257] (1) Overall configuration

[0258] Figure 12 This is a block diagram illustrating an example configuration of an image processing apparatus 60a according to a first embodiment, which has the function of a decoder. (Refer to...) Figure 12 The image processing apparatus 60a is shown to include a decoding control unit 61, a reversible decoding unit 62, an inverse quantization unit 63a, an inverse quadrature transform unit 64, an addition unit 65, an in-loop filter 66, a rearrangement buffer 67, a frame memory 68, selectors 70 and 71, an intra-frame prediction unit 80, an inter-frame prediction unit 85, and a QM storage unit 165a.

[0259] The decoding control unit 61 controls the overall decoder function of the image processing apparatus 60a, which will be described in detail below. According to this embodiment, the decoding control unit 61 includes a block setting unit 161. The block setting unit 161 performs functions already described above. Figure 4 The module described is the block setting processing step S12. This module will be described further later.

[0260] The reversible decoding unit 62 parses control parameters included in the incoming encoded stream from, for example, a communication interface and a transmission unit (not shown) connected to a peripheral device. The control parameters parsed by the reversible decoding unit 62 include, for example, the block partitioning data and scaling list data mentioned above. The block partitioning data is output to the decoding control unit 61. According to this embodiment, the reversible decoding unit 62 includes an SL decoding unit 163. The SL decoding unit 163 performs the functions already referred to above. Figure 4 The module described is the scaling list decoding process step S13. This module will be further described later. The reversible decoding unit 62 decodes the encoded stream to generate quantized data for each of one or more transform blocks. The reversible decoding unit 62 outputs the generated quantized data to the inverse quantization unit 63a.

[0261] For each of one or more transform blocks set in the image, the inverse quantization unit 63a inversely quantizes the incoming quantized data, i.e., the quantized transform coefficients from the reversible decoding unit 62, to recover the transform coefficients. The inverse quantization unit 63a selects a quantization matrix with a size equal to the size of the transform block from a plurality of quantization matrices stored in the QM storage unit 165a, and uses the selected quantization matrix to inversely quantize the quantized transform coefficients of the transform block. Under the control of the decoding control unit 61, the inverse quantization unit 63a skips the inverse quantization of frequency components that are forced to zero. Then, the inverse quantization unit 63a outputs the recovered transform coefficients to the inverse quadrature transform unit 64. According to this embodiment, the inverse quantization unit 63a includes a QM generation unit 167a. The QM generation unit 167a performs the functions already referred to above. Figure 4 The module described is for QM generation processing step S14. This module will be described further later.

[0262] For each of one or more transform blocks, the inverse quadrature transform unit 64 performs an inverse quadrature transform. This inverse quadrature transform can be performed, for example, as an inverse discrete cosine transform or an inverse discrete sine transform. More specifically, for each transform block, the inverse quadrature transform unit 64 performs an inverse quadrature transform on the transform coefficients in the frequency domain from the inverse quantization unit 63a, thereby generating a prediction error representing a signal sample in the spatial domain. The inverse quadrature transform unit 64 then outputs the generated prediction error to the adder unit 65.

[0263] Adding unit 65 adds the incoming prediction error from inverse quadrature transform unit 64 to the incoming prediction image from selector 71 to generate a decoded image. Then, adding unit 65 outputs the generated decoded image to in-loop filter 66 and frame memory 68.

[0264] The in-loop filter 66 consists of a series of filters applied to the decoded image to improve image quality. The in-loop filter 66 may include one or more of, for example, bilateral filters, deblocking filters, adaptive offset filters, and adaptive loop filters described in reference REF3. The in-loop filter 66 outputs the decoded image, which has been filtered through the series of filters, to the rearrangement buffer 67 and the frame memory 68.

[0265] The rearrangement buffer 67 rearranges the incoming image from the in-loop filter 66 to generate a time-based image sequence that makes up the video. The rearrangement buffer 67 then outputs the generated image sequence to an external device (e.g., a display connected to the image processing device 60a).

[0266] The frame memory 68 stores the decoded image before filtering from the addition unit 65 and the decoded image after filtering from the in-loop filter 66.

[0267] Depending on the prediction type for each block, selector 70 switches between intra-prediction unit 80 and inter-prediction unit 85 to determine the destination to which the image from frame memory 68 is sent. For example, when intra-prediction is specified as the prediction type, selector 70 outputs the unfiltered decoded image provided by frame memory 68 as a reference image to intra-prediction unit 80. When inter-prediction is specified as the prediction type, selector 70 outputs the filtered decoded image as a reference image to inter-prediction unit 85.

[0268] Depending on the prediction type of each block, selector 71 switches between intra-prediction unit 80 and inter-prediction unit 85 to determine which unit the predicted image to be provided to adder unit 65 originates from. For example, when intra-prediction is specified as the prediction type, selector 71 provides the predicted image from intra-prediction unit 80 to adder unit 65. When inter-prediction is specified as the prediction type, selector 71 provides the predicted image from inter-prediction unit 85 to adder unit 65.

[0269] Intra-prediction unit 80 performs intra-prediction based on information about intra-prediction obtained by parsing the coded stream and a reference image from frame memory 68 to generate a predicted image. Then, intra-prediction unit 80 outputs the generated predicted image to selector 71.

[0270] Inter-frame prediction unit 85 performs inter-frame prediction based on information about inter-frame prediction obtained by parsing the coded stream and a reference image from frame memory 68 to generate a predicted image. Then, inter-frame prediction unit 85 outputs the generated predicted image to selector 71.

[0271] (2) Functions related to quantization matrix generation

[0272] exist Figure 12 Among the various components shown, the block setting unit 161, SL decoding unit 163, QM storage unit 165a, and QM generation unit 167a are mainly involved in the generation of quantization matrix performed by the decoder.

[0273] The block setting unit 161 sets multiple transform blocks in each image by performing QTBT block partitioning based on the block partitioning data. The size of the transform blocks set by the block setting unit 161 can be in the range of, for example, from 2×2 to 128×128. The shape of the transform blocks can be square or non-square. Figure 2 The image shows some examples of the shape and size of the transform block.

[0274] The SL decoding unit 163 decodes the scaling list data to generate a basic quantization matrix of one or more sizes. In this example, the SL decoding unit 163 decodes size specification information indicating the size of the quantization matrix generated from the scaling list data. In this example, by referring to the size specification information, the SL decoding unit 163 identifies the size of the quantization matrix that is explicitly transmitted as a basic quantization matrix via the scaling list data. The SL decoding unit 163 can also decode size count information indicating the number of sizes to be transmitted via signaling. In this case, bitstream constraints can be imposed to stipulate that when two or more sizes are transmitted via signaling, these sizes should be different from each other (i.e., they should be identified by different size IDs). Such bitstream constraints prevent the encoder from redundantly encoding the size specification information, thereby reducing coding overhead and avoiding resource waste. In another example, the size of the basic quantization matrix explicitly transmitted via the scaling list data is predefined according to a specification-based rule. In this case, the scaling list data does not include the size specification information and size count information mentioned above, and the SL decoding unit 163 decodes the scaling list data for each of the predefined one or more sizes to generate a quantization matrix for one or more sizes.

[0275] The SL decoding unit 163 stores the basic quantization matrix generated based on the scaled list data in the QM storage unit 165a. As mentioned above, for each size, there can be multiple types of basic quantization matrices that differ from each other in terms of the combination of prediction type and color components. The basic quantization matrix can be generated by decoding a series of differentially encoded element values ​​or by referencing different types of basic quantization matrices. In one example, the basic quantization matrix consists only of square quantization matrices. In another example, the basic quantization matrix includes both square quantization matrices and non-square quantization matrices. An example of the syntax for scaled list data has already been described in [2-2. Examples of Syntax and Semantics] above.

[0276] QM storage unit 165a is a storage module that stores various types of quantization matrices of various sizes, which are used by the image processing apparatus 60a. The quantization matrices stored in QM storage unit 165a include a basic quantization matrix generated by SL decoding unit 163 and additional quantization matrices generated by QM generation unit 167a, which will be described later. According to this embodiment, the basic quantization matrix is ​​generated before performing inverse quantization and inverse quadrature transformation across multiple transform blocks, and is stored in QM storage unit 165a through these processes of inverse quadrature transformation and inverse quantization. On the other hand, when inverse quantization is performed on the quantization transform coefficients of each transform block, additional quantization matrices are generated as needed and stored in QM storage unit 165a. Similar to QM storage unit 115a on the encoder side, QM storage unit 165a can manage matrix management information indicating the size of existing quantization matrices. The matrix management information consists, for example, a set of markers indicating the existence of quantization matrices identified by two size IDs corresponding to the horizontal and vertical dimensions, respectively.

[0277] When the inverse quantization unit 63a inverse-quantizes the quantized transform coefficients of each transform block, the QM generation unit 167a determines, by referring to matrix management information provided by the QM storage unit 165a, whether a quantization matrix with a size equal to the transform block size has been generated based on the scaling list data. If a quantization matrix with a size equal to the transform block size has been generated, the QM generation unit 167a reads the generated quantization matrix from the QM storage unit 165a. If a quantization matrix with a size equal to the transform block size has not yet been generated, the QM generation unit 167a resamples one of the generated basic quantization matrices or a portion thereof to generate an additional quantization matrix. The QM generation unit 167a includes a memory for temporarily storing the reference quantization matrix or a portion thereof to be resampled.

[0278] For example, when the zeroing of high-frequency components is not applied to a transform block of the object size, the QM generation unit 167a generates the quantization matrix for the transform block by referencing the entire reference quantization matrix. On the other hand, when the zeroing of high-frequency components is applied to a transform block of another object size, the QM generation unit 167a generates the quantization matrix for the transform block by referencing only a portion of the reference quantization matrix. As described above, the ratio of the size of the referenced portion matrix to the size of the reference quantization matrix is ​​equal to the ratio of the size of the non-zero portion to the size of the generated additional quantization matrix. Examples of formulas for calculating these ratios have been described above in [2-4. Resampling Methods]. In this manner, the inverse quantization unit 63a uses one of the various quantization matrices generated in a resource-efficient manner to inverse quantize the quantization transform coefficients of each transform block.

[0279] In the example, QM generation unit 167a can determine whether to apply zeroing to a transform block based on a specification-based rule that predefines the size of the transform block to which zeroing is to be applied. In another example, QM generation unit 167a can determine whether to apply zeroing to a transform block based on one or more control parameters that can be obtained by parsing the encoded stream (e.g., a zeroing flag indicating whether zeroing is to be applied to the transform block, and zeroing size information indicating the size of the transform block to which zeroing is to be applied). Examples of these control parameters have been described above in [1-4. Zeroing Control].

[0280] [2.7 The flow of inverse quantization related processing performed during decoding]

[0281] (1) Overall process

[0282] Figure 13 It is shown by Figure 12 A flowchart illustrating an example of the inverse quantization-related processing performed by the image processing apparatus 60a. For the purpose of simplicity, processing steps unrelated to inverse quantization, which are included in the various processing steps of the decoding process, are not described in this flowchart.

[0283] First, the SL decoding unit 163 performs scaling list data decoding processing, which will be described later, to generate scaling lists representing one or more basic quantization matrices of different sizes (step S161). Then, the SL decoding unit 163 maps each scaling list, which is an array of one-dimensional element values, to a two-dimensional array of element values ​​using a specific scan sequence, thereby transforming the scaling lists into basic quantization matrices (step S162). The QM storage unit 165a stores the basic quantization matrices generated in this manner (step S163).

[0284] For each of the multiple transform blocks set in the image by the block setting unit 161 through QTBT block division, the following series of processing steps S165 to S168 (step S164) are repeated. Each transform block for which these processing steps are repeated is called an object transform block.

[0285] QM generation unit 167a first determines, for example, by referring to matrix management information provided by QM storage unit 165a whether a quantization matrix of a size corresponding to the block size of the object transformation block exists (step S165). When no such quantization matrix exists, QM generation unit 167a executes the steps already referred to above. Figure 10 The described quantization matrix generation process generates an additional quantization matrix from a reference quantization matrix (step S166). The additional quantization matrix generated in this step is stored in QM storage unit 165a. On the other hand, when the expected quantization matrix already exists, QM generation unit 167a reads a quantization matrix of a size corresponding to the block size of the object transform block from QM storage unit 165a (step S167). Subsequently, inverse quantization unit 63a performs inverse quantization on the quantization transform coefficients of the object transform block using either the additionally generated quantization matrix or the quantization matrix read from QM storage unit 165a (step S168).

[0286] The frequency domain transform coefficients generated as a result of the inverse quantization at step S168 are transformed by the inverse orthogonal transform unit 64 into prediction errors representing signal samples in the spatial domain. (Not in...) Figure 13 The process is described in the text. Corresponding to the updating of the scaled list data, the quantization matrix can be updated in any kind of update unit, such as sequences, images, slices, and tiles.

[0287] (2) Scaling list data decoding processing

[0288] Figure 14 It shows that it can be shown Figure 13 The flowchart is an example of the process of performing the scaling list data decoding process at step S161.

[0289] First, the SL decoding unit 163 decodes the size specification information to set a size ID (step S171) to identify the size of the quantization matrix corresponding to the scaling list to be generated. When the basic quantization matrix is ​​always a square matrix, one size ID is set for one size, as shown in Table 3. However, when the basic quantization matrix includes non-square matrices, two size IDs corresponding to the two directions can be set for one size based on the size specification information. The size ID can be obtained by adding a preset offset value to the value indicated by the size specification information.

[0290] For each combination of prediction type and color component, i.e., for each quantization matrix type identified by matrix ID, the following series of processing steps S173 to S177 (step S172) are repeated. The matrix ID for which the processing steps are repeated is called the object matrix ID.

[0291] SL decoding unit 163 determines whether a series of element values ​​of the scaling list associated with the object matrix ID are explicitly encoded (step S173). For example, SL decoding unit 163 may determine whether a series of element values ​​are encoded or only the scaling list information is encoded based on the size ID set in step S171 and the control flag associated with the object matrix ID (e.g., "scaling_list_pred_mode_flag[sizeId][matrixId]" in Table 1).

[0292] When only the scaling list information (e.g., “scaling_list_pred_matrix_id delta[sizeId][matrixId]” on Table 1) is encoded, the SL decoding unit 163 parses the reference scaling list information to derive the matrix ID of the basic quantization matrix to be referenced (step S174). Then, the SL decoding unit 163 generates a scaling list for the object matrix ID based on the reference scaling list that references the derived matrix ID as a key (step S175).

[0293] On the other hand, when a series of element values ​​in the scaling list are encoded, the SL decoding unit 163 parses the difference values ​​of the series of element values, which are differentially encoded in the scaling list data (step S176). Then, the SL decoding unit 163 decodes these difference values ​​using differential pulse code modulation (DPCM) to generate a scaling list for the object matrix ID (step S177).

[0294] Figure 14 The flowchart shown depicts an example of decoding size specification information that indicates the size ID of the basic quantization matrix. In another example, the size of the basic quantization matrix can be predefined based on a specification, thus skipping the decoding of the size specification information. Alternatively, size count information indicating the number of dimensions to be transmitted with the signal can also be decoded.

[0295] In the first embodiment described in this section, the quantization matrix is ​​generated in a so-called "on-demand" mode when the quantization matrix required for each transform block has not yet been generated during quantization or inverse quantization. With this configuration, quantization matrices defined by various combinations of shapes and sizes of the quantization matrix are not generated, thus avoiding the generation of quantization matrices defined by combinations of shapes and sizes not used in practical applications. Therefore, the processing cost required for quantization matrix generation can be reduced. Furthermore, only the additional quantization matrices actually used need to be stored in memory, in addition to the basic quantization matrix. Therefore, memory resource consumption is also reduced.

[0296] In the second embodiment, which will be described in the next section, an additional quantization matrix is ​​generated before processing the multiple transform blocks, compared to the first embodiment.

[0297] <3. Second Implementation Method>

[0298] [3-1. Encoder Configuration]

[0299] (1) Overall configuration

[0300] Figure 15 This is a block diagram illustrating an example configuration of an image processing apparatus 10b according to a second embodiment, which has the function of an encoder. (Refer to...) Figure 15 The image processing apparatus 10b is shown to include an encoding control unit 11, a rearrangement buffer 12, a subtraction unit 13, an orthogonal transform unit 14, a quantization unit 15b, a reversible encoding unit 16, an accumulation buffer 17, a rate control unit 18, an inverse quantization unit 21, an inverse orthogonal transform unit 22, an addition unit 23, an in-loop filter 24, a frame memory 25, a switch 26, an intra-frame prediction unit 30, an inter-frame prediction unit 35, a mode setting unit 40, a QM storage unit 115b, and a QM generation unit 117b. Similar to the first embodiment, the encoding control unit 11 includes a block setting unit 111 and a basic QM setting unit 113. Similarly to the first embodiment, the reversible encoding unit 16 includes an SL encoding unit 119.

[0301] Quantization unit 15b receives input transform coefficients from quadrature transform unit 14 and rate control signals from rate control unit 18. For each of one or more transform blocks in the image to be encoded, quantization unit 15b quantizes the transform coefficients using a quantization matrix with a size equal to the size of the transform block to generate quantized transform coefficients (quantized data). Under the control of encoding control unit 11, quantization unit 15b skips the quantization of frequency components in the transform coefficients that are forced to zero. Then, quantization unit 15b outputs the generated quantized data to reversible encoding unit 16 and inverse quantization unit 21. Quantization unit 15b can change the bit rate of the quantized data by switching quantization steps based on the rate control signal.

[0302] (2) Functions related to quantization matrix generation

[0303] exist Figure 15 Of the various components shown, the block setting unit 111, the basic QM setting unit 113, the QM storage unit 115b, the QM generation unit 117b, and the SL encoding unit 119 are primarily involved in the generation of the quantization matrix performed by the encoder. These components have functions that differ from those of the corresponding components according to the first embodiment. These differences will be described below.

[0304] QM storage unit 115b is a storage module that stores various types of quantization matrices of various sizes, which are used by the image processing device 10b. The quantization matrices stored in QM storage unit 115b include a basic quantization matrix set by basic QM setting unit 113 and additional quantization matrices generated by QM generation unit 117b, which will be described later. According to this embodiment, both the basic quantization matrix and the additional quantization matrix are generated before performing orthogonal transformation and quantization across multiple transform blocks, and are stored in QM storage unit 115b through these processes of orthogonal transformation and quantization. QM storage unit 115b can manage matrix management information, which is internal control information indicating the size of the existing quantization matrices.

[0305] Before quantizing the transform coefficients of multiple transform blocks, QM generation unit 117b generates an additional quantization matrix corresponding to each of the multiple size candidates for the transform block. For example, for each of the size candidates for the quantization matrix that is determined to be an ungenerated quantization matrix based on matrix management information provided by QM storage unit 115b, QM generation unit 117b selects one of the already generated basic quantization matrices as a reference quantization matrix and resamples the selected reference quantization matrix to generate the additional quantization matrix corresponding to the size candidate. QM generation unit 117b then stores the generated additional quantization matrix in QM storage unit 115b. QM generation unit 117b includes a memory for temporarily storing the reference quantization matrix to be resampled, or a portion thereof.

[0306] Similar to the QM generation unit 117a according to the first embodiment, when generating an additional quantization matrix of zero size without the application of high-frequency components, the QM generation unit 117b references the entire reference quantization matrix. On the other hand, when generating an additional quantization matrix of zero size with the application of high-frequency components, the QM generation unit 117b references only a portion of the reference quantization matrix. In the latter case, as described above, the ratio of the size of the referenced portion to the size of the reference quantization matrix is ​​equal to the ratio of the size of the non-zero portion to the size of the generated additional quantization matrix.

[0307] The descriptive content already described above regarding the syntax and semantics examples, the methods for selecting a reference quantization matrix, and the methods for generating an additional quantization matrix through resampling (upsampling or downsampling) of the first embodiment also applies to the second embodiment.

[0308] [3-2. The flow of quantization-related processing performed during encoding]

[0309] Figure 16 It is shown by Figure 15 A flowchart illustrating an example of the quantization-related processing flow performed by the image processing apparatus 10b. For the purpose of simplicity, quantization-independent processing steps among the various processing steps included in the encoding process are not described in this flowchart.

[0310] First, the basic QM setting unit 113 sets one or more basic quantization matrices, including a first quantization matrix of a first size (step S211). These basic quantization matrices are stored in the QM storage unit 115b.

[0311] For each of one or more size candidates, i.e., for each combination of size IDs in the horizontal direction and size IDs in the vertical direction, the following processing steps S213 and S214 (step S212a) are repeated. In this processing step loop, steps S213 and S214 (step S212b) are repeated for each of the plurality of matrix IDs corresponding to the combination of prediction type and color components. The size candidate and matrix ID for which the processing steps are repeated are referred to as the object size candidate and object matrix ID, respectively.

[0312] QM generation unit 117b first determines, for example, whether a quantization matrix corresponding to the object size candidate and object matrix ID exists by referring to matrix management information provided by QM storage unit 115b (step S213). When no such quantization matrix exists, QM generation unit 117b performs the steps already referred to above. Figure 10 The described quantization matrix generation process generates an additional quantization matrix from a reference quantization matrix (step S214). The additional quantization matrix generated at this step is stored in QM storage unit 115b.

[0313] When the loop of the processing step used to generate the additional quantization matrix ends, processing steps S216 and S217 (step S215) are repeated for each of the plurality of transform blocks set in the image by the block setting unit 111. Each transform block for which these processing steps are repeated is called an object transform block.

[0314] The quantization unit 15b reads the quantization matrix corresponding to the block size of the object transformation block from the QM storage unit 115b (step S216). Then, the quantization unit 15b uses the read quantization matrix to quantize the transformation coefficients of the object transformation block (step S217).

[0315] SL encoding unit 119 converts each of the one or more basic quantization matrices set in step S211 into a one-dimensional code, thereby transforming each of the basic quantization matrices into a scaling list (step S218). Then, SL encoding unit 119 performs the steps already referred to above. Figure 11 The scaling list encoding process described herein is used to generate scaling list data (step S219).

[0316] The reversible encoding unit 16 encodes the quantized transform coefficients generated at step S217 for each transform block, so that the encoded quantized transform coefficients, along with the scaling list data, become part of the encoded stream. (Not in...) Figure 16 The process is illustrated below. Scaling list data can be updated in any kind of update unit, such as sequences, images, slices, and tiles.

[0317] [3-3. Decoder Configuration]

[0318] (1) Overall configuration

[0319] Figure 17 This is a block diagram illustrating an example configuration of an image processing apparatus 60b according to a second embodiment, which has a decoder function. (Refer to...) Figure 17 The image processing apparatus 60b is shown to include a decoding control unit 61, a reversible decoding unit 62, an inverse quantization unit 63b, an inverse quadrature transform unit 64, an addition unit 65, an in-loop filter 66, a rearrangement buffer 67, a frame memory 68, selectors 70 and 71, an intra-frame prediction unit 80, an inter-frame prediction unit 85, a QM storage unit 165b, and a QM generation unit 167b. Similar to the first embodiment, the decoding control unit 61 includes a block setting unit 161. Similarly to the first embodiment, the reversible decoding unit 62 includes an SL decoding unit 163.

[0320] For each of the one or more transform blocks set in the image, the inverse quantization unit 63b inversely quantizes the incoming quantized data, i.e., the quantized transform coefficients from the reversible decoding unit 62, to recover the transform coefficients. The inverse quantization unit 63b selects a quantization matrix with a size equal to the transform block size from a plurality of quantization matrices stored in the QM storage unit 165b, and uses the selected quantization matrix to inversely quantize the quantized transform coefficients of the transform block. Under the control of the decoding control unit 61, the inverse quantization unit 63b skips the inverse quantization of frequency components that are forced to zero. Then, the inverse quantization unit 63b outputs the recovered transform coefficients to the inverse quadrature transform unit 64.

[0321] (2) Functions related to quantization matrix generation

[0322] exist Figure 17 Of the various components shown, the block setting unit 161, SL decoding unit 163, QM storage unit 165b, and QM generation unit 167b are primarily involved in the generation of the quantization matrix performed by the decoder. These components have functions that differ from those of the corresponding components according to the first embodiment. These differences will be described below.

[0323] QM storage unit 165b is a storage module that stores various types of quantization matrices of various sizes, which are used by the image processing device 60b. The quantization matrices stored in QM storage unit 165b include a basic quantization matrix generated by SL decoding unit 163 and additional quantization matrices generated by QM generation unit 167b, which will be described later. According to this embodiment, both the basic quantization matrix and the additional quantization matrix are generated before performing inverse quantization and inverse orthogonal transformation across multiple transform blocks, and are stored in QM storage unit 165b through these processes of inverse quantization and inverse orthogonal transformation. QM storage unit 165b can manage matrix management information, which is internal control information indicating the size of the existing quantization matrices.

[0324] Before inverse quantization of the transform coefficients of multiple transform blocks, QM generation unit 167b generates an additional quantization matrix corresponding to each of the multiple size candidates of the transform block. For example, for each of the size candidates of the quantization matrix that is determined to be an ungenerated quantization matrix based on matrix management information provided by QM storage unit 165b, QM generation unit 167b selects one of the already generated basic quantization matrices as a reference quantization matrix and resamples the selected reference quantization matrix to generate the additional quantization matrix corresponding to the size candidate. QM generation unit 167b then stores the generated additional quantization matrix in QM storage unit 165b. QM generation unit 167b includes a memory for temporarily storing the reference quantization matrix to be resampled or a portion thereof.

[0325] Similar to the QM generation unit 167a according to the first embodiment, when generating an additional quantization matrix of zero size without the application of high-frequency components, the QM generation unit 167b references the entire reference quantization matrix. On the other hand, when generating an additional quantization matrix of zero size with the application of high-frequency components, the QM generation unit 167b references only a portion of the reference quantization matrix. In the latter case, as described above, the ratio of the size of the referenced portion to the size of the reference quantization matrix is ​​equal to the ratio of the size of the non-zero portion to the size of the generated additional quantization matrix.

[0326] [3-4. The flow of inverse quantization related processing performed during decoding]

[0327] Figure 18 It is shown by Figure 17 A flowchart illustrating an example of the inverse quantization-related processing performed by the image processing apparatus 60b. For the purpose of simplicity, processing steps unrelated to inverse quantization among the various processing steps included in the decoding process are not described in this flowchart.

[0328] First, the SL decoding unit 163 has already been executed as referenced above. Figure 14 The described scaling list data decoding process generates scaling lists that express one or more basic quantization matrices of different sizes (step S261). Subsequently, the SL decoding unit 163 maps each scaling list, which is an array of one-dimensional element values, to a two-dimensional array of element values ​​using a specific scan sequence, thereby transforming the scaling lists into basic quantization matrices (step S262). The QM storage unit 165b stores the basic quantization matrices generated in this manner (step S263).

[0329] For each of one or more size candidates, i.e., for each combination of size IDs in the horizontal direction and size IDs in the vertical direction, the following processing steps S265 and S266 (step S264a) are repeated. In this processing step loop, steps S265 and S266 (step S264b) are repeated for each of the plurality of matrix IDs corresponding to the combination of prediction type and color components. The size candidate and matrix ID for which the processing steps are repeated are referred to as the object size candidate and object matrix ID, respectively.

[0330] QM generation unit 167b first determines, for example, whether a quantization matrix corresponding to the object size candidate and object matrix ID exists by referring to matrix management information provided by QM storage unit 165b (step S265). When no such quantization matrix exists, QM generation unit 167b performs the steps already referred to above. Figure 10 The described quantization matrix generation process generates an additional quantization matrix from a reference quantization matrix (step S266). The additional quantization matrix generated at this step is stored in QM storage unit 165b.

[0331] When the loop of the processing step used to generate the additional quantization matrix ends, processing steps S268 and S269 (step S267) are repeated for each of the plurality of transform blocks set in the image by the block setting unit 161. Each transform block for which these processing steps are repeated is called an object transform block.

[0332] The inverse quantization unit 63b reads the quantization matrix corresponding to the block size of the object transformation block from the QM storage unit 165b (step S268). Then, the inverse quantization unit 63b uses the read quantization matrix to inverse quantize the quantization transformation coefficients of the object transformation block (step S269).

[0333] The frequency domain transform coefficients generated as a result of the inverse quantization at step S269 are transformed by the inverse orthogonal transform unit 64 into prediction errors representing signal samples in the spatial domain. (Not in...) Figure 18The process is described in the text. Corresponding to the updating of the scaled list data, the quantization matrix can be updated in any kind of update unit, such as sequences, images, slices, and tiles.

[0334] According to the second embodiment described in this section, many quantization matrices that may be needed for quantization or inverse quantization processing are pre-generated before processing is performed on multiple transform blocks. This configuration eliminates the need for computation to generate additional insufficient quantization matrices after processing for each block in the image has begun, thereby improving processing performance.

[0335] <4. Hardware Configuration Example>

[0336] The above-described embodiments can be implemented using any of the following means: software, hardware, and a combination of software and hardware. When the image processing devices 10a, 10b, 60a, and 60b use software, the computer program constituting the software is pre-stored in a computer-readable medium (non-transitory medium), for example, contained within the device or disposed outside the device. For example, when each program is executed, the program is loaded into random access memory (RAM) and executed by a processor such as a central processing unit (CPU).

[0337] Figure 19 This is a block diagram illustrating an example of the hardware configuration to which the above-described embodiments can be applied. (Refer to...) Figure 19 The display device 800 includes a system bus 810, an image processing chip 820, and an off-chip memory 890. The image processing chip 820 includes n processing circuits 830-1, 830-2, ..., 830-n (n represents an integer of 1 or greater), a reference buffer 840, a system bus interface 850, and a local bus interface 860.

[0338] System bus 810 provides a communication path between image processing chip 820 and external modules (e.g., central control functions, application functions, communication interfaces, user interfaces, etc.). Processing circuits 830-1, 830-2, ... 830-n are connected to system bus 810 via system bus interface 850 and to off-chip memory 890 via local bus interface 860. Processing circuits 830-1, 830-2, ... 830-n can also be accessed by reference buffer 840, which is equivalent to on-chip memory (e.g., static random access memory (SRAM)). On-chip memory may include, for example... Figure 4 The internal memory M12 shown in the figure or Figure 7 The internal memory M22 is shown in the diagram. On the other hand, the external memory 890 may include, for example... Figure 4 The QM memory M11 shown in the figure or Figure 7The QM memory M21 is shown in the figure. The off-chip memory 890 may also include a frame memory that stores image data processed by the image processing chip 820.

[0339] <5. Summary>

[0340] So far, it has been referred to Figures 3A to 19 Implementations of the technology according to this disclosure are described in detail. According to the above implementation, when generating a second quantization matrix for a transform block of a second size applied to zero-return high-frequency components, the entire first quantization matrix of the first size generated by decoding the scaled list data is not referenced; instead, only a portion of the first quantization matrix is ​​referenced. Therefore, the impact of generating various quantization matrices on processing costs, such as hardware resource consumption, processing latency, and increased power consumption, can be reduced at least. Therefore, an improved system for efficiently generating or transmitting quantization matrices via signals is provided.

[0341] According to the above implementation, when the second dimension is a dimension that applies zero-level high-frequency components, the ratio of the size of the referenced partial matrix to the size of the first dimension when generating the second quantization matrix is ​​equal to the ratio of the size of the non-zero portion to the second dimension. Therefore, the resource waste caused by calculating the elements of the quantization matrix corresponding to the always zero transform coefficients can be eliminated.

[0342] In one implementation, size specification information indicating the dimensions of the base quantization matrix generated from the scaling list data is explicitly encoded or decoded as a control parameter. This means that the dimensions of the base quantization matrix, from which various quantization matrices are generated, can be specified as variable dimensions, thus allowing users to flexibly use the quantization matrix in various desired forms. In another implementation, the scaling list data is decoded for each of one or more predefined dimensions, and one or more dimensions include the dimensions of the base quantization matrix. In this case, the need to encode the size specification information is eliminated.

[0343] In one implementation, rules are pre-set based on specifications to determine the size of the transform block to which zeroing of high-frequency components is applied. This eliminates the need to encode control parameters for determining whether zeroing of high-frequency components is applied to the transform block. In another implementation, a zeroing flag associated with each transform block, indicating whether zeroing of high-frequency components is applied, is explicitly encoded or decoded as a control parameter. This allows for dynamic control over the application or non-application of zeroing. In another example, zeroing size information, indicating the size of the transform block to which zeroing of high-frequency components is applied, is explicitly encoded or decoded as a control parameter. This allows for dynamic control over the size to which zeroing is applied. Based on these examples, a system capable of flexibly reproducing images expressing even minute high-frequency components is provided, depending on user needs or system requirements or constraints.

[0344] According to the above implementation, when the second quantization matrix is ​​generated, a portion of the matrix is ​​loaded onto memory and temporarily stored therein. Therefore, sufficient memory resources are ensured, which optimizes device performance, such as encoder and decoder functions.

[0345] Preferred embodiments of the present disclosure have been described above with reference to the accompanying drawings. However, the technical scope of the present disclosure is not limited to the above-described embodiments. It will be apparent to those skilled in the art that various changes and modifications can be discovered within the scope of the technical concept described in the claims, and it should be understood that these changes and modifications will naturally fall within the technical scope of the present disclosure.

[0346] (Combination of implementation methods)

[0347] The embodiments described above and the various examples associated with them can be combined in any form, and the technology according to this disclosure is not limited to the combinations directly mentioned above. For example, any given method described with respect to the first embodiment can be combined with any given method described with respect to the second embodiment.

[0348] (The object to which the proposed technology is applied)

[0349] The techniques according to this disclosure can be applied to any given video coding method (video decoding method). In other words, unless the specifications of processes such as transform (inverse transform), quantization (inverse quantization), encoding (decoding), prediction, and filtering, and processes related to encoding and decoding, clearly lead to any inconsistency, such specifications do not limit the scope of the techniques of this disclosure. Furthermore, some of these processes may be omitted unless omitting some of them clearly leads to any inconsistency.

[0350] (piece)

[0351] Generally, a “block” refers to any given partial region of an image (e.g., a picture, slice, and tile) (in exceptional cases, the term may refer to a functional block that performs some function). Unless otherwise stated, the size and shape of a block do not limit the scope of the technology according to this disclosure. For example, the concept of “block” encompasses various partial regions (processing units) mentioned in references REF1 through REF3, such as TB (transform block), TU (transform unit), PB (prediction block), PU (prediction unit), SCU (minimum coding unit), CU (coding unit), LCU (maximum coding unit), CTB (coding tree block), CTU (coding tree unit), sub-block, macroblock, tile, and slice.

[0352] (Data Unit / Processing Unit)

[0353] The scope of application of the various parameters, data, or information mentioned in this specification is not limited to the scope described in the examples above, but can be any form of scope. Similarly, the scope of execution of the various processes mentioned in this specification is not limited to the scope described in the examples above, but can be any form of scope. For example, these scopes can be set up as units (data units / processing units) of at least one of TB, TU, PB, PU, ​​SCU, CU, LCU, CTB, CTU, sub-block, block, tile, slice, picture, sequence, and component. Obviously, data units / processing units can be set up for each parameter, data, and information, or process, and the data units / processing units may be common or not common to all parameters, data, and information, or processes. Parameters, data, or information can be stored in or transmitted from any given location. For example, parameters, data, or information can be stored in the header or parameter set of the aforementioned units. In another case, parameters, data, or information can be stored dispersedly in multiple locations or transmitted from multiple locations.

[0354] (Control Information)

[0355] Any control information relating to the technology according to this disclosure may be transmitted from the encoding side to the decoding side. For example, control information (e.g., enabled_flag) may be transmitted that gives instructions to enable or disable any given portion of the various functions described above. In another case, control information may be transmitted specifying the range (or the range in which any given portion of the various functions described above can or cannot be applied) of the technology according to this disclosure. For example, control information specifying the block size (upper block size or lower block size or both), frame, component, or layer to which the technology according to this disclosure is applied may be transmitted.

[0356] (Block size information)

[0357] Block sizes specified in the form of control information may not be directly expressed as size values, but rather indirectly as identifiers or indices mapped to size values. For example, a size value or its corresponding identifier or index may be expressed as a ratio or difference to a specific reference value (e.g., the size of an LCU or SCU). For instance, size information included in syntax elements can indirectly specify size values ​​using the methods described above. This method increases the possibility of reducing the amount of information encoded, transmitted, or stored to improve coding efficiency. Furthermore, the methods described above for specifying block sizes can also be used to specify a range of block sizes.

[0358] (other)

[0359] Generally, a "tag" is information that identifies multiple states. A tag can identify not only the two states "true (1)" and "false (0)", but also three or more states. In other words, a tag can take either "0" or "1", or one of three or more values. Therefore, a tag can consist of any number of bits, that is, a single bit or multiple bits. Control information that may include tags and other identifying information can itself be included in the bitstream, or control information indicating differences from some form of reference information can be included in the bitstream.

[0360] Various data and metadata concerning encoded data, such as encoded streams and encoded bitstreams, can be transmitted or recorded in any form, as long as these data and metadata are associated with the encoded data. The phrase "...data associated with...data" used in this context refers, for example, to a situation where one piece of data can be used to process another (e.g., two pieces of data are linked or mapped). In other words, data that is associated with each other can be processed as a single piece of data, or they can be processed as separate pieces of data. For example, information associated with encoded data can be transmitted via a different transmission path than the transmission path that transmits encoded data (encoded images). Furthermore, information associated with encoded data can be recorded on a different recording medium than the recording medium on which the encoded data is recorded (or recorded in different recording areas of the same recording medium on which both information and encoded data are recorded). Moreover, different parts of data, rather than the whole data, can be "associated" with each other in the manner described above. For example, an image and information about the image can be associated with each other at any given unit (e.g., multiple frames, a single frame, or a portion of a frame).

[0361] It should be noted that in this specification, terms such as “synthesis,” “multiplication,” “addition,” “integration,” “include,” “store,” “place,” “insert,” and “embed” may be used to refer to the processing of individual data as a whole or the processing of relating individual data to each other.

[0362] The technology according to this disclosure can be implemented by any kind of components that make up an apparatus or system, such as a processor for system-wide integration (LSI), a module containing multiple processors, a unit containing multiple modules, a device constructed by adding additional functionality to a module or unit, etc.

[0363] In this specification, a system refers to a collection of multiple elements (e.g., devices, units, modules, components, etc.). However, it should be noted that not all of these elements need to be housed in the same enclosure. In other words, the concept of a system includes, for example, a collection of multiple modules housed in separate enclosures and interconnected by a network, as well as a collection of multiple modules housed in a single enclosure.

[0364] For example, a component described as a single part (e.g., a processing unit) can be configured to be divided into multiple parts. Conversely, multiple parts described in this specification can be configured as a single part. Components different from those described above can be added to the parts described in this specification. If the system as a whole substantially provides the same function or performs the same operation, a portion of the configuration of one device can be included in other devices. Additionally, for example, the technology according to this disclosure can be implemented via cloud computing technology, which enables multiple devices interconnected via a network to collaboratively or distributedly perform a single function or multiple functions.

[0365] One or more steps depicted in a flowchart may be performed by multiple devices distributed among themselves, rather than by a single device. Similarly, multiple operations constituting a step may be performed by multiple devices distributed among themselves, rather than by a single device.

[0366] The program instructions that make up a program executed by a computer can cause the computer to execute two or more processing steps described in this specification in the order described above, or to execute the processing steps in parallel, or to execute the processing steps one by one in response to a triggering event (e.g., an event occurring or an external call). In other words, the processing steps described in this specification can be executed in a different order than described above unless it would cause obvious inconsistency. Furthermore, processing steps to be executed based on a particular program or program instructions can be executed in parallel or simultaneously with processing steps to be executed based on a different program or program instructions.

[0367] The effects described in this specification are illustrative or exemplary only and are not restrictive. In other words, in addition to or instead of the effects described above, the technology according to this disclosure can provide other effects that will be apparent to those skilled in the art as described in this specification.

[0368] Note that the following configurations are also within the technical scope of this disclosure.

[0369] (1) An image processing apparatus, comprising:

[0370] A decoding unit that decodes the scaled list data to generate a first quantization matrix of a first size;

[0371] A generation unit generates a second quantization matrix for a transform block of a second size with zero-reduction applied to high-frequency components by referring only to a portion of the first quantization matrix generated by the decoding unit; and

[0372] The inverse quantization unit uses the second quantization matrix generated by the generation unit to inverse quantize the quantization transform coefficients of the transform block of the second size.

[0373] (2) The image processing apparatus according to (1), wherein the ratio of the size of the partial matrix to the first size is equal to the ratio of the size of the non-zero portion to the second size.

[0374] (3) The image processing apparatus according to (1) or (2),

[0375] The decoding unit decodes size specification information indicating the size of the quantization matrix generated from the scaling list data, and

[0376] The dimension indicated by the dimension specification information includes the first dimension but excludes the second dimension.

[0377] (4) The image processing apparatus according to (1) or (2),

[0378] The decoding unit decodes scaling list data for each of one or more predefined sizes to generate quantization matrices for the one or more sizes.

[0379] Wherein, the predefined one or more dimensions include the first dimension, but do not include the second dimension.

[0380] (5) The image processing apparatus according to any one of (1) to (4), wherein the generating unit generates a third quantization matrix for a transform block of a third size with zero applied high-frequency components by referring to the entirety of the first quantization matrix generated by the decoding unit.

[0381] (6) The image processing apparatus according to any one of (1) to (5), wherein the generating unit determines whether to apply the zeroing to the transform block of the second size according to a rule that predefines the size of the transform block to which the zeroing is to be applied.

[0382] (7) The image processing apparatus according to any one of (1) to (5), wherein the generating unit determines whether to apply the zeroing to the transform block of the second size based on a zeroing marker associated with each transform block, the zeroing marker indicating whether the zeroing is applied.

[0383] (8) The image processing apparatus according to any one of (1) to (5), wherein the generating unit determines whether to apply the zeroing to the transform block of the second size based on zeroing size information indicating the size of the transform block to be applied.

[0384] (9) The image processing apparatus according to any one of (1) to (8), wherein the generating unit includes a memory that temporarily stores the partial matrix when generating the second quantization matrix.

[0385] (10) The image processing apparatus according to any one of (1) to (9) further includes

[0386] A storage unit that stores the first quantization matrix through processing involving multiple transform blocks.

[0387] The generation unit generates the second quantization matrix during the inverse quantization of the quantization transform coefficients of each transform block of the second size.

[0388] (11) The image processing apparatus according to any one of (1) to (9),

[0389] The generation unit generates the second quantization matrix before processing involving multiple transform blocks, and

[0390] The image processing apparatus further includes a storage unit, which stores a plurality of quantization matrices, including the first quantization matrix and the second quantization matrix, through processing involving the plurality of transform blocks.

[0391] (12) The image processing apparatus according to any one of (1) to (11), wherein the inverse quantization unit performs inverse quantization processing on each of a plurality of transform blocks set in an image by quadtree plus binary tree (QTBT) blocks.

[0392] (13) An image processing method performed by an image processing apparatus, the image processing method comprising:

[0393] Decode the scaled list data to generate a first quantization matrix of a first size;

[0394] A second quantization matrix is ​​generated by referring only to a portion of the generated first quantization matrix to create a transform block of a second size with zero-reduction applied to high-frequency components; and

[0395] Using the generated second quantization matrix, the quantization transform coefficients of the transform block of the second size are inversely quantized.

[0396] (14) An image processing apparatus, comprising:

[0397] The generation unit generates a second quantization matrix for a transform block of a second size with applied high-frequency components by referring only to a portion of the first quantization matrix of the first size.

[0398] A quantization unit, using the second quantization matrix generated by the generation unit, quantizes the transform coefficients of a transform block of the second size in the image to be encoded to generate quantized transform coefficients; and

[0399] An encoding unit encodes the quantization transform coefficients and a scaling list representing the first quantization matrix to generate an encoded stream.

[0400] (15) An image processing method performed by an image processing apparatus, the image processing method comprising:

[0401] A second quantization matrix is ​​generated by referencing only a portion of the first quantization matrix of the first size, for a transform block of the second size with applied high-frequency components that has been zeroed out.

[0402] Using the generated second quantization matrix, the transform coefficients of the transform block of the second size in the image to be encoded are quantized to generate quantized transform coefficients; and

[0403] The quantization transform coefficients and the scaling list representing the first quantization matrix are encoded to generate an encoded stream.

[0404] List of reference numerals

[0405] 10a, 10b Image processing device (encoder)

[0406] Quantization units 15a and 15b

[0407] 16 Reversible Encoding Units

[0408] 111 block setting unit

[0409] 113 QM Setting Unit

[0410] 115a, 115b QM storage units

[0411] 117a, 117b QM generation units

[0412] 119 SL coding unit

[0413] 60a, 60b Image processing device (decoder)

[0414] 62 reversible decoding units

[0415] 63a, 63b Inverse Quantization Units

[0416] 161 block setting units

[0417] 163 SL decoding unit

[0418] 165a, 165b QM memory units

[0419] 167a, 167b QM generation units

Claims

1. An image processing apparatus, comprising: A generation unit generates a second quantization matrix for a transform block of a second size with applied high-frequency components by referring only to a portion of a first quantization matrix of a first size. A quantization unit uses the second quantization matrix generated by the generation unit to quantize the transform coefficients of the transform block of the second size in the image to be encoded, so as to generate quantized transform coefficients; as well as An encoding unit encodes the quantization transform coefficients and the scaling list representing the first quantization matrix to generate an encoded stream. The generation unit determines whether to apply the zeroing to the transform block of the second size based on the zeroing size information indicating the size of the transform block to be zeroed.

2. The image processing apparatus according to claim 1, wherein, The ratio of the size of the partial matrix to the first size is equal to the ratio of the size of the non-zero part to the second size.

3. The image processing apparatus according to claim 1, in, The encoding unit also encodes size specification information indicating the size of the quantization matrix generated from the encoded stream, and The dimension indicated by the dimension specification information includes the first dimension but excludes the second dimension.

4. The image processing apparatus according to claim 1, in, The encoding unit encodes a list of scaling values ​​for each of one or more predefined sizes to generate the encoded stream, and Wherein, the predefined one or more dimensions include the first dimension, but do not include the second dimension.

5. The image processing apparatus according to claim 1, wherein, The generation unit generates a third quantization matrix for a zero-scale transform block of a third size that does not apply high-frequency components by referring to the entirety of the first quantization matrix.

6. The image processing apparatus according to claim 1, wherein, The generation unit determines whether to apply the zeroing to the transform block of the second size based on a predefined rule for the size of the transform block to which the zeroing is to be applied.

7. The image processing apparatus according to claim 1, wherein, The encoding unit also encodes a zeroing flag associated with each transform block, the zeroing flag indicating whether the zeroing is applied.

8. The image processing apparatus according to claim 1, wherein, The generation unit includes a memory, which temporarily stores a portion of the matrix when generating the second quantization matrix.

9. The image processing apparatus according to claim 1, further comprising: A storage unit that stores the first quantization matrix through processing involving multiple transform blocks. The generation unit generates the second quantization matrix during the quantization of the transform coefficients of each transform block of the second size.

10. The image processing apparatus according to claim 1, in, The generation unit generates the second quantization matrix before processing involving multiple transform blocks, and The image processing apparatus further includes a storage unit, which stores a plurality of quantization matrices, including the first quantization matrix and the second quantization matrix, through processing involving the plurality of transform blocks.

11. The image processing apparatus according to claim 1, wherein, The quantization unit performs quantization processing on each of the multiple transform blocks set up in the image by dividing it into quadtree plus binary tree (QTBT) blocks.

12. An image processing method executed by an image processing apparatus, the image processing method comprising: A second quantization matrix is ​​generated by referencing only a portion of the first quantization matrix of the first size, for a transform block of the second size with applied high-frequency components that has been zeroed out. Using the generated second quantization matrix, the transform coefficients of the second-size transform block in the image to be encoded are quantized to generate quantized transform coefficients; as well as The quantization transform coefficients and the scaling list representing the first quantization matrix are encoded to generate an encoded stream. In the step of generating the second quantization matrix, it is determined whether to apply the zeroing to the transform block of the second size based on the zeroing size information indicating the size of the transform block to which the zeroing is to be applied.

Images