Image decoding apparatus and method, image encoding apparatus and method, and storage medium

Abstract

The present application provides an image decoding apparatus and method, an image encoding apparatus and method, and a storage medium. A decoding unit decodes data corresponding to a first array of quantized coefficients including an N x M array of quantized coefficients corresponding to a first block and data corresponding to a second array of quantized coefficients including an N x M array of quantized coefficients corresponding to a second block. An inverse quantization unit derives a first array of orthogonal transform coefficients from the first array of quantized coefficients using at least a first quantization matrix of elements of the N x M array and derives a second array of orthogonal transform coefficients from the second array of quantized coefficients using at least a second quantization matrix of elements of the N x M array. An inverse orthogonal transform unit inverse orthogonally transforms the first array of orthogonal transform coefficients to generate a first prediction residual of pixels of a P x Q array and inverse orthogonally transforms the second array of orthogonal transform coefficients to generate a second prediction residual of pixels of the N x M array.

Description

[0001] (This application is a divisional application of application filed on February 28, 2020, with application number 202080020388.5 and title "Image Decoding Device, Image Decoding Method and Program".) Technical Field

[0002] This invention relates to image encoding technology. Background Technology

[0003] High Efficiency Video Coding (HEVC) is a known coding method for compressing moving images. HEVC uses basic blocks that are larger than traditional macroblocks (16×16 pixel arrays) to improve coding efficiency. These large basic blocks are called Code Tree Units (CTUs), and their maximum size is a 64×64 pixel array. CTUs are further divided into sub-blocks, which serve as units for prediction and transformation.

[0004] In HEVC, a quantization matrix is ​​used to assign weights to the coefficients of the orthogonal transform (hereinafter referred to as orthogonal transform coefficients) based on the frequency components. The use of a quantization matrix can improve compression efficiency while maintaining image quality by reducing the amount of data related to high-frequency components, whose degradation is less noticeable to human vision, compared to data related to low-frequency components. Japanese Patent Application Publication No. 2013-38758 discusses techniques for encoding information indicating such a quantization matrix.

[0005] International standardization of a more efficient coding method to succeed HEVC has recently begun. Specifically, the Joint Video Experts Group (JVET), established by the International Organization for Standardization (ISO / IEC) and the Telecommunication Standardization Sector of the ITU (ITU-T), is promoting the standardization of the Universal Video Coding (VVC) coding method. For this standardization, a new technique (hereinafter referred to as zero-based) is being studied to reduce code size and improve coding efficiency by forcing the orthogonal transform coefficients of high-frequency components to zero when the block size is large during orthogonal transform.

[0006] Similar to HEVC, the introduction of a quantization matrix into VVC was also investigated. However, the quantization matrix in HEVC is based on a method for quantization using a quantization matrix of the same size as the traditional orthogonal transform, and it cannot support zeroing, a new technique that sets a portion of the orthogonal transform coefficients to zero. Therefore, the existing zeroing of the orthogonal transform coefficients results in the inability to perform frequency-component-based quantization control and to improve subjective image quality.

[0007] Existing technical documents

[0008] Patent documents

[0009] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2013-38758 SUMMARY OF THE INVENTION

[0010] For example, the following structure is provided so that even when a technique of forcibly setting a part of orthogonal transform coefficients to 0 is used, the subjective image quality can be improved by enabling quantization processing using a quantization matrix corresponding to the technique. In other words, an image decoding device that can decode an image from a bitstream in units of a plurality of blocks, the plurality of blocks including a first block of pixels in a P×Q (P and Q are integers) array and a second block of pixels in an N×M (N is an integer satisfying N<P, and M is an integer satisfying M<Q) array, the image decoding device includes: a decoding unit configured to decode data corresponding to a first quantization coefficient array and data corresponding to a second quantization coefficient array from the bitstream, the first quantization coefficient array including quantization coefficients in an N×M array corresponding to the first block, the second quantization coefficient array including quantization coefficients in an N×M array corresponding to the second block; an inverse quantization unit configured to derive a first orthogonal transform coefficient array from the first quantization coefficient array by at least using a first quantization matrix of elements of the N×M array, and to derive a second orthogonal transform coefficient array from the second quantization coefficient array by at least using a second quantization matrix of elements of the N×M array; and an inverse orthogonal transform unit configured to generate a first prediction residual of pixels in a P×Q array corresponding to the first block by performing an inverse orthogonal transform on the first orthogonal transform coefficient array, and to generate a second prediction residual of pixels in an N×M array corresponding to the second block by performing an inverse orthogonal transform on the second orthogonal transform coefficient array, wherein the first quantization matrix of elements of the N×M array is different from the second quantization matrix of elements of the N×M array.

[0011] In addition, for example, the following structure is provided. An image encoding device capable of encoding an image in units of a plurality of blocks, the plurality of blocks including a first block of pixels in a P×Q (P and Q are integers) array and a second block of pixels in an N×M (N is an integer satisfying N<P, and M is an integer satisfying M<Q) array, the image encoding device including: an orthogonal transformation unit configured to generate at least a first orthogonal transformation coefficient array by performing an orthogonal transformation on a prediction residual of pixels in the P×Q array corresponding to the first block, and generate a second orthogonal transformation coefficient array by performing an orthogonal transformation on a prediction residual of pixels in the N×M array corresponding to the second block; a quantization unit configured to generate a first quantization coefficient array including quantization coefficients in an N×M array by quantizing the first orthogonal transformation coefficient array at least using a first quantization matrix of elements in the N×M array, and generate a second quantization coefficient array including quantization coefficients in an N×M array by quantizing the second orthogonal transformation coefficient array at least using a second quantization matrix of elements in the N×M array; and an encoding unit configured to encode data corresponding to the first quantization coefficient array and data corresponding to the second quantization coefficient array, the first quantization coefficient array including quantization coefficients in an N×M array corresponding to the first block, the second quantization coefficient array including quantization coefficients in an N×M array corresponding to the second block, wherein the first quantization matrix of elements in the N×M array is different from the second quantization matrix of elements in the N×M array. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram showing the structure of an image encoding device according to a first exemplary embodiment.

[0013] Figure 2 is a block diagram showing the structure of an image decoding device according to a second exemplary embodiment.

[0014] Figure 3 is a flowchart showing an image encoding process using the image encoding device according to the first exemplary embodiment.

[0015] Figure 4 is a flowchart showing an image decoding process using the image decoding device according to the second exemplary embodiment.

[0016] Figure 5 is a block diagram showing an example of the hardware structure of a computer applicable to the image encoding device and the image decoding device according to the present invention.

[0017] Figure 6A is a diagram showing an example of a bitstream output in the first exemplary embodiment.

[0018] Figure 6BThis is a diagram illustrating an example of the bitstream output in a first typical embodiment.

[0019] Figure 7A This is a diagram illustrating examples of sub-block segmentation used in the first and second typical embodiments.

[0020] Figure 7B This is a diagram illustrating examples of sub-block segmentation used in the first and second typical embodiments.

[0021] Figure 7C This is a diagram illustrating examples of sub-block segmentation used in the first and second typical embodiments.

[0022] Figure 7D This is a diagram illustrating examples of sub-block segmentation used in the first and second typical embodiments.

[0023] Figure 7E This is a diagram illustrating examples of sub-block segmentation used in the first and second typical embodiments.

[0024] Figure 7F This is a diagram illustrating examples of sub-block segmentation used in the first and second typical embodiments.

[0025] Figure 8A This is a diagram illustrating examples of the quantization matrices used in the first and second typical embodiments.

[0026] Figure 8B This is a diagram illustrating examples of the quantization matrices used in the first and second typical embodiments.

[0027] Figure 8C This is a diagram illustrating examples of the quantization matrices used in the first and second typical embodiments.

[0028] Figure 9 This is a diagram illustrating a method for scanning elements of a quantization matrix used in the first and second typical embodiments.

[0029] Figure 10 This is a diagram showing the difference matrix of the quantization matrices generated in the first and second typical embodiments.

[0030] Figure 11A This is a diagram showing an example of the encoding table used when encoding the difference in the quantization matrix.

[0031] Figure 11B This is a diagram showing an example of the encoding table used when encoding the difference in the quantization matrix.

[0032] Figure 12AThis is a diagram illustrating another example of the quantization matrix used in the first and second typical embodiments.

[0033] Figure 12B This is a diagram illustrating another example of the quantization matrix used in the first and second typical embodiments.

[0034] Figure 12C This is a diagram illustrating another example of the quantization matrix used in the first and second typical embodiments. Detailed Implementation

[0035] Typical embodiments of the invention will be described with reference to the accompanying drawings. Note that the structures described in the following typical embodiments are merely examples, and the invention is not limited to the structures described in the following typical embodiments. For convenience, terms such as basic block, sub-block, quantization matrix, and base quantization matrix are used in the typical embodiments, and other terms may be used appropriately without changing the meaning of these terms. For example, basic block and sub-block may be referred to as basic unit and sub-unit, or simply as block or unit. In the following description, as generally defined, a rectangle is a quadrilateral with four right angles and two diagonals of equal length. As generally defined, a square is a rectangle or quadrilateral with all four angles being identical and all four sides being identical. In other words, a square is a type of rectangle.

[0036] <First Typical Embodiment>

[0037] Typical embodiments of the present invention will now be described with reference to the accompanying drawings.

[0038] First, zeroing will be explained in more detail. As mentioned above, zeroing is a process used to force a portion of the orthogonal transform coefficients of the block to be encoded to be 0. For example, suppose the block to be encoded is a block of pixels in a 64×64 array in the input image (picture). Here, the orthogonal transform coefficients also have a size of 64×64 array. Zeroing is a process used, for example, to encode by assuming a portion of the orthogonal transform coefficients of the 64×64 array to be 0 even if the value of the result of the orthogonal transform is not zero. For example, low-frequency components corresponding to a predetermined upper-left range including the DC component in the two-dimensional orthogonal transform coefficients are excluded from forced zeroing, and orthogonal transform coefficients corresponding to frequency components that are higher in frequency than the low-frequency components are constantly set to 0.

[0039] Next, an image encoding device according to this exemplary embodiment will be described. Figure 1 This is a block diagram illustrating an image encoding apparatus according to this exemplary embodiment. Figure 1 In the middle, the image data is input to terminal 101.

[0040] The block segmentation unit 102 segments the input image into multiple basic blocks and outputs the image in units of basic blocks to the subsequent stage.

[0041] Quantization matrix storage unit 103 generates and stores the quantization matrix. As used herein, the quantization matrix is ​​intended to assign weights to the quantization process for the orthogonal transform coefficients based on frequency components. For example, weights are assigned to the quantization step size for the orthogonal transform coefficients in the quantization process described below by multiplying a scale value (quantization scale) based on a reference parameter value (quantization parameter) by the values ​​of each component in the quantization matrix.

[0042] There are no particular limitations on the method used to generate the quantization matrix stored in the quantization matrix storage unit 103. For example, a user can input information indicating the quantization matrix. The image encoding device can calculate the quantization matrix based on the characteristics of the input image. A quantization matrix pre-specified as its initial value can be used. In this typical embodiment, except... Figure 8A In addition to the base quantization matrix of the 8×8 array shown, the matrix generated by expanding the base quantization matrix is ​​also generated and stored. Figure 8B and Figure 8C The two types of 32×32 arrays are shown as two-dimensional quantization matrices. Figure 8B The quantization matrix is ​​obtained by... Figure 8A The quantization matrix of an 8×8 array is quadrupled by repeating each element four times in the vertical and horizontal directions, resulting in a 32×32 array quantization matrix. On the other hand, Figure 8C The quantization matrix is ​​obtained by... Figure 8A The quantization matrix is ​​obtained by expanding the elements of the 4×4 array in the upper left corner of the base quantization matrix by repeating them eight times in the vertical and horizontal directions.

[0043] As mentioned above, the base quantization matrix is ​​a quantization matrix used not only to quantize pixel sub-blocks of an 8×8 array but also to generate a quantization matrix that is larger in size than the base quantization matrix. Although the size of the base quantization matrix is ​​an 8×8 array, this size is not limiting. Furthermore, different base quantization matrices can be used depending on the sub-block size. For example, if three types of sub-blocks are used (i.e., sub-blocks of 8×8, 16×16, and 32×32 arrays), then three corresponding types of base quantization matrices can be used.

[0044] Prediction unit 104 determines sub-block segmentation related to image data in units of basic blocks. More specifically, prediction unit 104 determines whether to segment the basic block into sub-blocks, and if so, how to segment the basic block. If the basic block is not segmented into sub-blocks, the resulting sub-blocks have the same size as the basic blocks. Sub-blocks can be squares or rectangles other than squares (non-squares).

[0045] Then, the prediction unit 104 generates predicted image data by performing intra-prediction as intraframe prediction or inter-prediction as interframe prediction on a sub-block basis.

[0046] For example, prediction unit 104 selects a prediction method for the sub-block from intra-frame prediction and inter-frame prediction, and performs the selected prediction to generate predicted image data related to the sub-block. Note that the prediction method used is not limited to this, and prediction can be performed by combining intra-frame prediction and inter-frame prediction.

[0047] The prediction unit 104 also calculates prediction residuals based on the input image data and the predicted image data, and outputs these prediction residuals. For example, the prediction unit 104 calculates the difference between the pixel values ​​of a sub-block and the individual pixel values ​​of the predicted image data generated by predicting the sub-block, and calculates these differences as prediction residuals.

[0048] The prediction unit 104 also outputs the information required for prediction (such as information indicating the segmentation state of the sub-blocks, the prediction mode of the prediction method for the sub-blocks, and information such as motion vectors) along with the prediction residuals. This information required for prediction will be collectively referred to as prediction information below.

[0049] The transformation and quantization unit 105 performs orthogonal transformations on the prediction residuals calculated by the prediction unit 104 on a sub-block basis to obtain orthogonal transformation coefficients indicating each frequency component of the prediction residuals. The transformation and quantization unit 105 also performs quantization using quantization parameters and the quantization matrix stored in the quantization matrix storage unit 103 to obtain quantization coefficients as the quantized orthogonal transformation coefficients. The orthogonal transformation function and the quantization function can be configured as separate structures.

[0050] The inverse quantization and inverse transform unit 106 reconstructs the orthogonal transform coefficients by inverse quantization of the quantization coefficients output from the transform and quantization unit 105 using the quantization coefficients and the quantization matrix stored in the quantization matrix storage unit 103. The inverse quantization and inverse transform unit 106 further performs an inverse orthogonal transform to reconstruct the prediction residual. This process of reconstructing (deriving) the orthogonal transform coefficients using the quantization matrix and quantization parameters is referred to as inverse quantization. The functions of inverse quantization and inverse transform can be configured as separate structures. The information used by the image decoding device to derive the quantization parameters is also encoded into a bitstream by the encoding unit 110.

[0051] Frame memory 108 stores the reconstructed image data.

[0052] The image reconstruction unit 107 generates predicted image data based on the prediction information output from the prediction unit 104, by appropriately referencing the frame memory 108. The image reconstruction unit 107 generates reconstructed image data based on the predicted image data and the input prediction residual, and outputs the reconstructed image data.

[0053] The in-loop filtering unit 109 performs in-loop filtering processing on the reconstructed image, such as unblocking filtering and adaptive sampling offset, and outputs the filtered image.

[0054] The encoding unit 110 encodes the quantization coefficients output from the transform and quantization unit 105 and the prediction information output from the prediction unit 104 to generate code data, and outputs the code data.

[0055] The quantization matrix encoding unit 113 encodes the base quantization matrix output from the quantization matrix storage unit 103 to generate quantization matrix code data for the image decoding device to export the base quantization matrix, and outputs the quantization matrix code data.

[0056] The integration and encoding unit 111 generates header code data by using quantization matrix code data as the output of the quantization matrix encoding unit 113. The integration and encoding unit 111 also combines the header code data with the code data output from the encoding unit 110 to generate a bit stream, and outputs the bit stream.

[0057] Terminal 112 outputs the bit stream generated by the integration and encoding unit 111 to the outside.

[0058] The image encoding operation using the aforementioned image encoding device will now be described. In this exemplary embodiment, the image encoding device is configured to input motion picture data in frames. Furthermore, in the description of this exemplary embodiment, the block segmentation unit 102 is described as dividing the motion picture data into basic blocks of pixels in a 64×64 array. However, this is not limiting. For example, the basic block may be a block of pixels in a 128×128 array. The basic block may be a block of pixels in a 32×32 array.

[0059] Before encoding an image, the image encoding device generates and encodes a quantization matrix. In the following description, for example, the horizontal direction in quantization matrix 800 or its blocks will be assumed to be the x-axis and the vertical direction will be assumed to be the y-axis, with the right and down directions being positive. The top-left element of quantization matrix 800 has coordinates (0,0). In other words, the bottom-right element of the base quantization matrix of an 8×8 array has coordinates (7,7). The bottom-right element of the quantization matrix of a 32×32 array has coordinates (31,31).

[0060] The quantization matrix storage unit 103 initially generates the quantization matrix. The quantization matrix is ​​generated based on the sub-block size, the magnitude of the orthogonal transform coefficients to be quantized, and the type of prediction method. In this typical embodiment, the initial generation is used to generate the quantization matrix described below. Figure 8A The base quantization matrix of the 8×8 array is shown. Next, this base quantization matrix is ​​enlarged to generate... Figure 8B and Figure 8C The quantization matrices of the two types of 32×32 arrays are shown. Figure 8B The quantization matrix is ​​obtained by... Figure 8A The quantization matrix of the 8×8 array is obtained by repeating the base quantization matrix four times in the vertical and horizontal directions and then amplifying it by four times.

[0061] More specifically, in Figure 8B In the example shown, the leftmost element of the base quantization matrix is ​​assigned the value 1 to each element in the 32×32 array quantization matrix with x-coordinates of 0–3 and y-coordinates of 0–3. The rightmost element of the base quantization matrix is ​​assigned the value 15 to each element in the 32×32 array quantization matrix with x-coordinates of 28–31 and y-coordinates of 28–31. Figure 8B In the example, the values ​​of the elements in the base quantization matrix are each assigned to a portion of the elements in the quantization matrix of the 32×32 array.

[0062] on the other hand, Figure 8C The quantization matrix is ​​obtained by... Figure 8A The quantization matrix is ​​a 32×32 array obtained by magnifying the elements of the 4×4 array in the upper left corner of the base quantization matrix by repeating each element eight times in the vertical and horizontal directions.

[0063] More specifically, in Figure 8C In the example shown, the elements in the 32×32 quantization matrix with x-coordinates of 0–7 and y-coordinates of 0–7 are assigned the value 1 to the top left element of the top left 4×4 array portion of the base quantization matrix. The elements in the 32×32 quantization matrix with x-coordinates of 24–31 and y-coordinates of 24–31 are assigned the value 7 to the bottom right element of the top left 4×4 array portion of the base quantization matrix. Figure 8C In the example, the values ​​of the elements in the base quantization matrix that correspond only to the top-left 4×4 array portion (x-coordinates from 0 to 3 and y-coordinates from 0 to 3) are assigned to the elements of the 32×32 array quantization matrix.

[0064] Note that the quantization matrix to be generated is not limited to this. If there are orthogonal transform coefficients of a size other than a 32×32 array to be quantized, a quantization matrix corresponding to the size of the orthogonal transform coefficients to be quantized (such as 16×16, 8×8, and 4×4 arrays) can be generated. There are no particular restrictions on the methods used to determine the elements constituting the base quantization matrix and the quantization matrix. For example, predetermined initial values ​​can be used. These elements can be set individually. The quantization matrix can be generated based on image characteristics.

[0065] The generated base quantization matrix and quantization matrix are thus stored in quantization matrix storage unit 103. Figure 8B An example of the quantization matrix to be used when quantizing the orthogonal transform coefficients corresponding to the sub-blocks of the 32×32 array to be described below is shown, and Figure 8C This illustrates an example of a quantization matrix used when quantizing orthogonal transform coefficients corresponding to sub-blocks of a 64×64 array. The bold box 800 represents the quantization matrix. For ease of illustration, each quantization matrix 800 should comprise a 32×32 array (i.e., 1024 pixels), and each square within the bold box should represent an element constituting the quantization matrix. In this typical embodiment, Figure 8B and Figure 8C The three types of quantization matrices shown are stored in two-dimensional form. However, it will be understood that the elements in the quantization matrix are not limited to this. Furthermore, multiple quantization matrices can be stored for the same prediction method, depending on the magnitude of the orthogonal transform coefficients to be quantized or whether the encoded object is a luma block or a chromaticity block. Typically, since quantization matrices are designed to achieve quantization processing based on human visual characteristics, therefore... Figure 8B and Figure 8C As shown, the elements in the low-frequency part corresponding to the upper left part of the quantization matrix are small, while the elements in the high-frequency part corresponding to the lower right part are large.

[0066] The quantization matrix encoding unit 113 sequentially reads the elements of the base quantization matrix stored in two-dimensional form from the quantization matrix storage unit 103, scans these elements to calculate the difference, and arranges these differences into a one-dimensional matrix. In this typical embodiment, regarding Figure 8A The base quantization matrix shown uses Figure 9 The scanning method shown calculates the difference between each element and its immediate preceding element in the scanning order. For example, by... Figure 9 The scanning method shown is used for scanning. Figure 8A The 8×8 array shown is the base quantization matrix. After the first element 1 in the upper left corner, the element 2 immediately below it is scanned, and the difference +1 is calculated. The first element of the quantization matrix (1 in this typical embodiment) is encoded by calculating the difference from a predetermined initial value (e.g., 8). However, this is not limiting, and the difference from any value or the value of the first element itself can be used.

[0067] In this typical embodiment, thus using Figure 9 The scanning method is based on Figure 8A to generate the basis quantization matrix Figure 10 The difference matrix is ​​shown. The quantization matrix encoding unit 113 further encodes the difference matrix to generate quantization matrix code data. In this typical embodiment, by using... Figure 11A The encoding table shown is used for encoding, but the encoding table is not limited to this. For example, it can be used... Figure 11B The encoding table shown is used. The resulting quantization matrix code data is output to the subsequent integration and encoding unit 111.

[0068] return Figure 1 The integration and encoding unit 111 encodes the header information required for encoding the image data and integrates the code data on the quantization matrix with the encoded header information.

[0069] Then, the image data is encoded. A frame of image data input from terminal 101 is input to block segmentation unit 102.

[0070] The block segmentation unit 102 divides the input image data into multiple basic blocks and outputs the image in basic block units to the prediction unit 104. In this typical embodiment, the image is output in basic blocks of pixels in a 64×64 array.

[0071] The prediction unit 104 performs prediction processing on the image data input from the block segmentation unit 102 in units of basic blocks. Specifically, the prediction unit 104 determines sub-block segmentation related to whether to segment the basic blocks into smaller sub-blocks, and also determines prediction modes such as intra-frame prediction and inter-frame prediction on a sub-block basis.

[0072] Figures 7A to 7F An example of a sub-block segmentation method is shown. The thick box 700 represents a base block. For ease of description, a base block should consist of a 64×64 array of pixels, and each quadrilateral in the thick box should represent a sub-block. Figure 7B This illustrates an example of quadtree square sub-block partitioning. A basic block of pixels in a 64×64 array is partitioned into sub-blocks in a 32×32 array. Figures 7C to 7F An example of rectangular sub-block segmentation is shown. Figure 7C In this model, the basic block is divided into vertically elongated rectangular sub-blocks of pixels in a 32×64 array. Figure 7D In this model, the basic block is divided into horizontally elongated rectangular sub-blocks of pixels in a 64×32 array. Figure 7E and Figure 7FIn this model, the basic block is divided into rectangular sub-blocks in a 1:2:1 ratio. Encoding is performed using not only square sub-blocks but also non-square rectangular sub-blocks. Furthermore, the basic block can be further divided into multiple square blocks, and sub-block segmentation can be performed based on the resulting square blocks. In other words, the size of the basic block is not limited to a 64×64 pixel array, and multiple sizes of basic blocks can be used.

[0073] In this typical embodiment, the basic blocks of pixels in a 64×64 array are as follows: Figure 7A That was not divided, or by using only such Figure 7B Such a quadtree partitioning is used for segmentation. However, sub-block partitioning methods are not limited to this. Methods such as... Figure 7E and Figure 7F Such ternary tree partitioning and such Figure 7C and Figure 7D That kind of binary tree partitioning. If we also use division... Figure 7A and Figure 7B If a sub-block is partitioned other than the sub-block partitioned, the quantization matrix storage unit 103 generates a quantization matrix corresponding to the sub-block to be used. If one or more new base quantization matrices corresponding to the generated quantization matrix are also generated, the quantization matrix encoding unit 113 also encodes these one or more new base quantization matrices.

[0074] The prediction method of the prediction unit 104 used in this exemplary embodiment will be described in more detail. In this exemplary embodiment, for example, two types of prediction methods are used: intra-frame prediction and inter-frame prediction. Intra-frame prediction generates prediction pixels for the block to be encoded by using encoded pixels spatially located around the block to be encoded, and also generates intra-frame prediction mode information indicating the intra-frame prediction method used in intra-frame prediction methods such as horizontal prediction, vertical prediction, and DC prediction. Inter-frame prediction generates prediction pixels for the block to be encoded by using encoded pixels in one or more frames that are temporally different from the block to be encoded, and also generates motion information indicating the frames(one or more) to be referenced and motion vectors, etc. As described above, the prediction unit 104 can use a prediction method that combines intra-frame prediction and inter-frame prediction.

[0075] Predictive image data is generated based on the determined prediction pattern and encoded pixels. Further, prediction residuals are generated based on the input image data and the predicted image data, and these prediction residuals are output to the transform and quantization unit 105. Information related to sub-block segmentation and prediction pattern, etc., is output as prediction information to the encoding unit 110 and the image reconstruction unit 107.

[0076] Transform and quantization unit 105 performs orthogonal transformation and quantization on the input prediction residual to generate quantization coefficients. Transform and quantization unit 105 first applies orthogonal transformation processing corresponding to the sub-block size to generate orthogonal transformation coefficients. Then, transform and quantization unit 105 generates quantization coefficients by quantizing the orthogonal transformation coefficients using the quantization matrix stored in quantization matrix storage unit 103 based on the prediction mode. More specific details of the orthogonal transformation and quantization processing are described below.

[0077] If you choose Figure 7B The 32×32 array is divided into sub-blocks. The transform and quantization unit 105 applies an orthogonal transformation of the orthogonal transformation matrix of the 32×32 array to the prediction residual of the 32×32 array to generate orthogonal transformation coefficients for the 32×32 array. Specifically, the transform and quantization unit 105 multiplies the orthogonal transformation matrix of the 32×32 array (represented by Discrete Cosine Transform (DCT)) with the prediction residual of the 32×32 array to calculate intermediate coefficients in matrix form. The transform and quantization unit 105 further multiplies the intermediate coefficients in matrix form with the transpose of the aforementioned orthogonal transformation matrix of the 32×32 array to generate orthogonal transformation coefficients for the 32×32 array. The transform and quantization unit 105 uses quantization parameters and Figure 8B The quantization matrix of the 32×32 array shown is used to quantize the orthogonal transform coefficients of the generated 32×32 array to produce the quantization coefficients of the 32×32 array. Since the basic block of the 64×64 array consists of four sub-blocks of the 32×32 array, this process is repeated four times.

[0078] On the other hand, if you choose Figure 7A The 64×64 array shown is in a segmented state (unsegmented). For the prediction residuals of the 64×64 array, an orthogonal transformation matrix for a 64×32 array is generated by interleaving the odd-numbered rows (hereinafter referred to as odd rows) of the orthogonal transformation matrix of the 64×64 array. In other words, the orthogonal transformation coefficients of the 32×32 array are generated by applying the orthogonal transformation to the orthogonal transformation matrix of the 64×32 array generated by interleaving the odd rows.

[0079] Specifically, the transformation and quantization unit 105 performs interval removal of odd-numbered rows in the orthogonal transformation matrix of the 64×64 array to generate an orthogonal transformation matrix of the 64×32 array. Then, the transformation and quantization unit 105 multiplies the orthogonal transformation matrix of the 64×32 array with the prediction residual of the 64×64 array to generate intermediate coefficients in the form of a 64×32 array matrix. The transformation and quantization unit 105 multiplies the intermediate coefficients in the form of a 64×32 array matrix with the transpose of the 32×64 array obtained by transposing the aforementioned orthogonal transformation matrix of the 64×32 array to generate orthogonal transformation coefficients of the 32×32 array. Then, the transformation and quantization unit 105 zeroes out the coefficients in the upper left portion (x-coordinate range of 0-31 and y-coordinate range of 0-31) of the orthogonal transformation coefficients of the 64×64 array by using the generated orthogonal transformation coefficients of the 32×32 array as the coefficients, and sets the other coefficients to 0.

[0080] In this typical embodiment, the prediction residuals of the 64×64 array are orthogonally transformed using an orthogonal transformation matrix of a 64×32 array and a transpose of the 32×64 array obtained by transposing the orthogonal transformation matrix of the 64×32 array. The orthogonal transformation coefficients of the 32×32 array are then zeroed out. Therefore, compared to the technique of forcibly setting a portion of the orthogonal transformation coefficients of the 64×64 array to 0 if a portion of the values ​​are not zero, the orthogonal transformation coefficients of the 32×32 array can be generated with less computation. In other words, the computational cost of the orthogonal transformation can be reduced compared to performing the orthogonal transformation using an orthogonal transformation matrix of a 64×64 array and encoding the result by assuming that the orthogonal transformation coefficients to be zero are zero regardless of whether they are actually zero. Note that although the method of calculating the orthogonal transformation coefficients of the 32×32 array from the prediction residuals of the 64×64 array by using orthogonal transformation can reduce the amount of computation, the method for zeroing is not limited to this and various methods can be used.

[0081] When performing zeroing, information indicating that the orthogonal transform coefficients in the range of objects to be zeroed are 0 can be encoded, or information indicating the execution of zeroing (such as flags) can be encoded simply. By decoding this information, the image decoding device can decode the block when the zeroed object is 0.

[0082] Next, the transformation and quantization unit 105 uses quantization parameters and Figure 8C The quantization matrix of the 32×32 array shown quantizes the orthogonal transform coefficients of the 32×32 array thus generated to produce the quantization coefficients of the 32×32 array.

[0083] In this typical embodiment, for the orthogonal transformation coefficients of the 32×32 array corresponding to the sub-blocks of the 32×32 array, the following is used: Figure 8B The quantization matrix, and for the orthogonal transform coefficients of the 32×32 array corresponding to the sub-blocks of the 64×64 array, using Figure 8C The quantization matrix. In other words, for the orthogonal transform coefficients of a 32×32 array that has not been zeroed out, use... Figure 8B Furthermore, for the orthogonal transformation coefficients of the 32×32 array corresponding to the sub-blocks of the 64×64 array that have been zeroed out, use Figure 8C The quantization matrix is ​​used. Note that the quantization matrix used is not limited to this. The generated quantization coefficients are output to the encoding unit 110 and the inverse quantization and inverse transform unit 106.

[0084] The inverse quantization and inverse transform unit 106 reconstructs orthogonal transform coefficients by inverse quantization of the input quantization coefficients using quantization parameters and the quantization matrix stored in the quantization matrix storage unit 103. The inverse quantization and inverse transform unit 106 further performs an inverse orthogonal transform on the reconstructed orthogonal transform coefficients to reconstruct the prediction residual. Similar to the transform and quantization unit 105, the inverse quantization and inverse transform unit 106 performs inverse quantization using a quantization matrix corresponding to the size of the sub-block to be encoded. The following describes the inverse quantization and inverse orthogonal transform processing using the inverse quantization and inverse transform unit 106 in more detail.

[0085] If you choose Figure 7B The 32×32 array is divided into sub-blocks, and the inverse quantization and inverse transform unit 106 is used by... Figure 8B The quantization matrix is ​​used to inverse quantize the quantization coefficients of the 32×32 array generated by the transformation and quantization unit 105 to reconstruct the orthogonal transformation coefficients of the 32×32 array. Then, the inverse quantization and inverse transformation unit 106 multiplies the transpose of the aforementioned 32×32 array with the orthogonal transformation coefficients of the 32×32 array to calculate intermediate coefficients in the form of a matrix of the 32×32 array. Then, the inverse quantization and inverse transformation unit 106 multiplies the intermediate coefficients in the form of a matrix of the 32×32 array with the aforementioned orthogonal transformation matrix of the 32×32 array to reconstruct the prediction residual of the 32×32 array. The inverse quantization and inverse transformation unit 106 performs similar processing on each sub-block of the 32×32 array. On the other hand, if as Figure 7A If unsegmented is selected, then the inverse quantization and inverse transform unit 106 is used by Figure 8CThe quantization matrix is ​​used to inverse quantize the quantization coefficients of the 32×32 array generated by the transform and quantization unit 105 to reconstruct the orthogonal transform coefficients of the 32×32 array. Then, the inverse quantization and inverse transform unit 106 multiplies the transpose of the aforementioned 32×64 array with the orthogonal transform coefficients of the 32×32 array to calculate intermediate coefficients in the form of a matrix of the 32×64 array. The inverse quantization and inverse transform unit 106 multiplies the intermediate coefficients in the form of a matrix of the 32×64 array with the aforementioned orthogonal transform matrix of the 64×32 array to reconstruct the prediction residual of the 64×64 array. In this typical embodiment, inverse quantization is performed based on the size of the sub-blocks using the same quantization matrix as used in the transform and quantization unit 105. The reconstructed prediction residual is output to the image reconstruction unit 107.

[0086] The image reconstruction unit 107 reconstructs the predicted image based on the prediction information input from the prediction unit 104 by appropriately referencing the data required for reconstructing the predicted image stored in the frame memory 108. Then, the image reconstruction unit 107 reconstructs image data based on the reconstructed predicted image and the reconstructed prediction residual input from the inverse quantization and inverse transform unit 106, and inputs and stores the image data in the frame memory 108.

[0087] The in-loop filtering unit 109 reads the reconstructed image from the frame memory 108 and performs in-loop filtering processing such as deblocking filtering. The in-loop filtering unit 109 then inputs the filtered image back into the frame memory 108.

[0088] Encoding unit 110 performs entropy encoding on a block-by-block basis on the quantization coefficients generated by transform and quantization unit 105 and the prediction information input from prediction unit 104 to generate code data. No specific entropy encoding method is specified, and Golomb coding, arithmetic coding, and Huffman coding can be used. The generated code data is output to integration and encoding unit 111.

[0089] The integration and encoding unit 111 generates a bit stream by multiplexing the aforementioned header code data and the code data input from the encoding unit 110. This bit stream is ultimately output to the outside from the terminal 112.

[0090] Figure 6AAn example of the bitstream output in the first exemplary embodiment is shown. The sequence header includes code data on the base quantization matrix, which includes the encoding results of the individual elements. Note that the location where the code data on the base quantization matrix, etc., is encoded is not limited to this, and it will be understood that the bitstream can be configured such that this data is encoded in the image header or other headers. If the quantization matrix is ​​switched within the sequence, the new base quantization matrix can be encoded for updating. In this case, all quantization matrices can be rewritten, or a portion of these quantization matrices can be changed by specifying the size of a sub-block of the quantization matrix corresponding to the quantization matrix to be rewritten.

[0091] Figure 3 This is a flowchart illustrating the encoding process using an image encoding device according to a first exemplary embodiment.

[0092] In step S301, before image encoding, the quantization matrix storage unit 103 first generates and stores two-dimensional quantization matrices. In this typical embodiment, the quantization matrix storage unit 103 generates... Figure 8A The base quantization matrix shown, and the quantization matrix generated based on the base quantization matrix. Figure 8B and Figure 8C The quantization matrix is ​​shown, and the base quantization matrix and these quantization matrices are stored.

[0093] In step S302, the quantization matrix encoding unit 113 scans the base quantization matrix used in step S301 when generating the quantization matrix to calculate the difference between adjacent elements in the scan order, and generates a one-dimensional difference matrix. In this typical embodiment, regarding Figure 8A The base quantization matrix shown uses Figure 9 The scanning method generates Figure 10 The difference matrix is ​​shown. The quantization matrix encoding unit 113 further encodes the generated difference matrix to generate quantization matrix code data.

[0094] In step S303, the integration and encoding unit 111 encodes the header information required to encode the image data together with the generated quantization matrix code data and outputs the result.

[0095] In step S304, the block segmentation unit 102 segments the input image in frames into basic blocks of pixels in a 64×64 array.

[0096] In step S305, the prediction unit 104 performs prediction processing on the image data generated in step S304 in units of basic blocks using the above-described prediction method to generate prediction information such as sub-block segmentation information and prediction patterns, as well as prediction image data. In this typical embodiment, two types of block sizes are used, namely... Figure 7BThe block size of the pixel sub-blocks in the 32×32 array shown is... Figure 7A The block size of the pixel sub-blocks in the 64×64 array is shown. The prediction unit 104 further calculates the prediction residual based on the input image data and the prediction image data.

[0097] In step S306, the transform and quantization unit 105 performs an orthogonal transform on the prediction residual calculated in step S305 to generate orthogonal transform coefficients. The transform and quantization unit 105 further generates quantization coefficients by quantizing using quantization parameters and the quantization matrix generated and stored in step S301. Specifically, the transform and quantization unit 105 uses a 32×32 array of orthogonal transform matrices and their transpose to quantize... Figure 7B The prediction residuals of the pixel sub-blocks of the 32×32 array shown are multiplied to generate orthogonal transform coefficients for the 32×32 array. On the other hand, the transform and quantization unit 105 uses the orthogonal transform matrix of the 64×32 array and its transpose to... Figure 7A The prediction residuals of the pixel sub-blocks of the 64×64 array shown are multiplied to generate orthogonal transform coefficients for a 32×32 array. In this typical embodiment, the transform and quantization unit 105 multiplies the prediction residuals of the pixel sub-blocks of the 64×64 array to generate orthogonal transform coefficients for a 32×32 array. Figure 8B The quantization matrix is ​​used for Figure 7B The orthogonal transformation coefficients of the sub-blocks of the 32×32 array shown, and by... Figure 8C The quantization matrix is ​​used for Figure 7A The orthogonal transformation coefficients of the sub-blocks of the 64×64 array shown are used to quantize the orthogonal transformation coefficients of the 32×32 array.

[0098] In step S307, the inverse quantization and inverse transform unit 106 reconstructs the orthogonal transform coefficients by inverse quantization of the quantization coefficients generated in step S306 using quantization parameters and the quantization matrix generated and stored in step S301. The inverse quantization and inverse transform unit 106 further performs an inverse orthogonal transform on the orthogonal transform coefficients to reconstruct the prediction residuals. In this step, inverse quantization is performed using the same quantization matrix as used in step S306. Specifically, for the quantization coefficients generated in step S301, the inverse quantization process is performed using the same quantization matrix as used in step S306. Figure 7B The quantization coefficients of the 32×32 array corresponding to the pixel sub-blocks of the 32×32 array are used by the inverse quantization and inverse transform unit 106. Figure 8B The quantization matrix is ​​used for inverse quantization to reconstruct the orthogonal transform coefficients of the 32×32 array. Then, the inverse quantization and inverse transform unit 106 multiplies the orthogonal transform coefficients of the 32×32 array using the orthogonal transform matrix of the 32×32 array and its transpose to reconstruct the prediction residuals of the pixels in the 32×32 array. On the other hand, for... Figure 7A The quantization coefficients of the 32×32 array corresponding to the sub-blocks of pixels in the 64×64 array are used by the inverse quantization and inverse transform unit 106. Figure 8C The quantization matrix is ​​used for inverse quantization to reconstruct the orthogonal transform coefficients of the 32×32 array. Then, the inverse quantization and inverse transform unit 106 multiplies the orthogonal transform coefficients of the 32×32 array by using the orthogonal transform matrix of the 64×32 array and its transpose to reconstruct the prediction residuals of the pixels of the 64×64 array.

[0099] In step S308, the image reconstruction unit 107 reconstructs the predicted image based on the prediction information generated in step S305. The image reconstruction unit 107 further reconstructs image data based on the reconstructed predicted image and the prediction residual generated in step S307.

[0100] In step S309, encoding unit 110 encodes the prediction information generated in step S305 and the quantization coefficients generated in step S306 to generate code data. Encoding unit 110 generates a bit stream by also including other code data.

[0101] In step S310, the image encoding device determines whether all basic blocks in the frame have been encoded. If all basic blocks have been encoded, the process proceeds to step S311. If not all basic blocks have been encoded, the process returns to step S304 with the next block as the target.

[0102] In step S311, the in-loop filtering unit 109 performs in-loop filtering on the image data reconstructed in step S308 to generate a filtered image. The process ends.

[0103] Through the above structure and operation, the quantization of each frequency component can be controlled while reducing computational load to improve subjective image quality. Specifically, in step S306, by reducing the number of orthogonal transform coefficients and performing quantization processing using a quantization matrix corresponding to the reduced orthogonal transform coefficients, the quantization of each frequency component can be controlled while reducing computational load to improve subjective image quality. Furthermore, if the number of orthogonal transform coefficients is reduced to quantize and encode only the low-frequency components, this can be achieved by using methods such as... Figure 8C Such a quantization matrix, obtained by only expanding the low-frequency portion of the base quantization matrix, achieves optimal quantization control over the low-frequency range. Figure 8C In the example, the low-frequency part here refers to the range where the x-coordinate is 0 to 3 and the y-coordinate is 0 to 3.

[0104] In this typical embodiment, in order to reduce the amount of code, the quantization matrix encoding unit 113 is configured to only encode the generated... Figure 8B and Figure 8C The quantization matrix is ​​used in common Figure 8A The base quantization matrix is ​​encoded. However, the quantization matrix encoding unit 113 can be configured to encode the base quantization matrix. Figure 8B and Figure 8C The quantization matrix itself is encoded. Since unique values ​​can be set for each frequency component of the quantization matrix, this allows for finer quantization control of each frequency component. Optionally, it can be used for... Figure 8B and Figure 8C Each quantization matrix has its own distinct base quantization matrix, and the quantization matrix encoding unit 113 can be configured to encode each base quantization matrix. In this case, different quantization controls can be applied to the orthogonal transform coefficients of the 32×32 array and the orthogonal transform coefficients of the 64×64 array to achieve more precise control over subjective image quality. Furthermore, in this case, instead of scaling up the upper left 4×4 array portion of the 8×8 array's base quantization matrix by a factor of eight, the entire base quantization matrix of the 8×8 array can be scaled up by a factor of four to obtain the quantization matrix corresponding to the orthogonal transform coefficients of the 64×64 array. This also enables finer quantization control over the orthogonal transform coefficients of the 64×64 array.

[0105] Furthermore, although this typical embodiment is configured such that the quantization matrix for a sub-block of the 64×64 array to be zeroed is uniquely determined, identifiers can be introduced to enable selection. For example, Figure 6B The bitstream is shown below: A new quantization matrix encoding method information code is introduced to make the quantization matrix encoding of sub-blocks of the 64×64 array to be zeroed selectable. For example, if the quantization matrix encoding method information code indicates 0, then it will be used as an independent quantization matrix. Figure 8C Orthogonal transform coefficients are used for sub-blocks of pixels in a 64×64 array that are to be zeroed. If the encoding method information code indicates 1, these will be used as the quantization matrix for ordinary sub-blocks that are not to be zeroed. Figure 8B This is used for sub-blocks of pixels in a 64×64 array that are to be zeroed. If the encoding method information code indicates 2, then instead of the elements of the base quantization matrix of the 8×8 array, all elements of the quantization matrix to be used for the sub-blocks of pixels in the 64×64 array that are to be zeroed are encoded. This allows for selective reduction of the amount of quantization matrix code and unique quantization control for the sub-blocks to be zeroed.

[0106] In this typical embodiment, the sub-blocks processed by zeroing are only sub-blocks of a 64×64 array. However, the sub-blocks to be processed by zeroing are not limited to this. For example, in conjunction with... Figure 7C or Figure 7D In the orthogonal transform coefficients corresponding to the sub-blocks of the 32×64 or 64×32 arrays shown, the orthogonal transform coefficients of the lower or right half of the 32×32 array can be forcibly set to 0. In this case, only the orthogonal transform coefficients of the upper or left half of the 32×32 array are quantized and encoded. By using... Figure 8BDifferent quantization matrices are used to quantize the orthogonal transformation coefficients of the 32×32 array in the upper or left half.

[0107] Furthermore, the values ​​of the quantization matrix corresponding to the top-left DC coefficients, which are considered to have the greatest impact on image quality, can be set and encoded separately from the values ​​of the elements of the 8×8 array's basis matrix. Figure 12B and Figure 12C Showing the value of the top left element corresponding to the DC component from Figure 8B and Figure 8C Examples of value changes in [the context]. In this case, it can be achieved by removing the [partial] pair from the [context]. Figure 8A In addition to encoding information related to the base quantization matrix, the information of the indicator "2" at the position of the DC part is also encoded to set... Figure 12B and Figure 12C The quantization matrix shown is used. This allows for finer quantization control of the DC components of the orthogonal transform coefficients, which have the greatest impact on image quality.

[0108] <Second Typical Embodiment>

[0109] Figure 2 This is a block diagram illustrating the structure of an image decoding apparatus according to a second exemplary embodiment of the present invention. This exemplary embodiment will be described by way of an image decoding apparatus for decoding encoded data generated in a first exemplary embodiment.

[0110] Input the encoded bit stream to terminal 201.

[0111] The separation and decoding unit 202 separates the bitstream into information related to decoding processing and code data related to coefficients, and decodes the code data included in the header of the bitstream. In this typical embodiment, the separation and decoding unit 202 separates the quantization matrix code and outputs the quantization matrix code to the subsequent stage. The separation and decoding unit 202 performs... Figure 1 The operation is the opposite of the operation of the integration and encoding unit 111.

[0112] The quantization matrix decoding unit 209 performs the following processing to decode the quantization matrix code from the bit stream to reconstruct the base quantization matrix, and further generates a quantization matrix based on the base quantization matrix.

[0113] Decoding unit 203 decodes the code data output from separation and decoding unit 202 to reconstruct (derive) quantization coefficients and prediction information.

[0114] Inverse quantization and inverse transform unit 204 and Figure 1Similar to the inverse quantization and inverse transform unit 106, the inverse quantization coefficients are obtained by inverse quantization of the quantization coefficients using quantization parameters and the reconstructed quantization matrix, and then further inverse orthogonal transform is performed to reconstruct the prediction residuals. The information used to derive the quantization parameters is also decoded from the bitstream by the decoding unit 203. The functions of inverse quantization and inverse transform can be configured as separate structures.

[0115] Frame memory 206 stores image data related to the reconstructed image.

[0116] The image reconstruction unit 205 generates predicted image data based on the input prediction information by appropriately referencing the frame memory 206. Then, the image reconstruction unit 205 generates reconstructed image data based on the predicted image data and the prediction residual reconstructed by the inverse quantization and inverse transform unit 204, and outputs the reconstructed image data.

[0117] In-loop filter unit 207 and Figure 1 Similar to 109, the reconstructed image undergoes in-loop filtering processes such as unblocking filtering, and the filtered image is output.

[0118] Terminal 208 outputs the reconstructed image data to the outside.

[0119] The image decoding operation of the aforementioned image decoding device will be described below. In this typical embodiment, the image decoding device is configured to input the bitstream generated in the first typical embodiment frame by frame (in units of images).

[0120] exist Figure 2 In this process, a one-frame bitstream input from terminal 201 is fed into the separation and decoding unit 202. The separation and decoding unit 202 separates the bitstream into information related to the decoding process and coefficient-related code data, and decodes the code data included in the header of the bitstream. More specifically, the separation and decoding unit 202 reconstructs the quantization matrix code data. In this typical embodiment, the separation and decoding unit 202 first... Figure 6A The quantization matrix code data is extracted from the sequence header of the bitstream shown, and this quantization matrix code data is output to the quantization matrix decoding unit 209. In this typical embodiment, the extracted and output data is related to... Figure 8A The quantization matrix code data corresponding to the base quantization matrix is ​​shown. Then, the separation and decoding unit 202 reconstructs the code data of the image data in units of basic blocks and outputs the code data to the decoding unit 203.

[0121] The quantization matrix decoding unit 209 first decodes the input quantization matrix code data to reconstruct the quantization matrix. Figure 10 The one-dimensional difference matrix is ​​shown. In this typical embodiment, as in the first typical embodiment, it is achieved by using... Figure 11AThe encoding table shown is used to decode the quantization matrix code data. However, the encoding table is not limited to this, and other encoding tables can be used, as long as the same encoding table is used in the first exemplary embodiment. The quantization matrix decoding unit 209 further reconstructs a two-dimensional quantization matrix based on the reconstructed one-dimensional difference matrix. Here, the quantization matrix decoding unit 209 performs the opposite operation to that of the quantization matrix encoding unit 113 according to the first exemplary embodiment. More specifically, in this exemplary embodiment, regarding Figure 10 The difference matrix shown is obtained by using... Figure 9 The scanning method shown reconstructs and stores Figure 8A The base quantization matrix is ​​shown. Specifically, the quantization matrix decoding unit 209 reconstructs the elements of the quantization matrix by sequentially adding the differences in the difference matrix to the aforementioned initial values. Then, the quantization matrix decoding unit 209, based on... Figure 9 The scanning method shown reconstructs the two-dimensional quantization matrix by sequentially associating the reconstructed one-dimensional elements with the elements of the two-dimensional quantization matrix.

[0122] The quantization matrix decoding unit 209 further expands the reconstructed base quantization matrix as in the first exemplary embodiment to generate Figure 8B and Figure 8C The quantization matrices of the two types of 32×32 arrays are shown. Figure 8B The quantization matrix is ​​obtained by... Figure 8A The quantization matrix of a 32×32 array is obtained by repeating each element of the base quantization matrix of an 8×8 array four times in the vertical and horizontal directions and then amplifying it by four times.

[0123] on the other hand, Figure 8C The quantization matrix is ​​obtained by... Figure 8A The quantization matrix is ​​obtained by magnifying the elements of the top-left 4×4 array portion of the base quantization matrix by repeating them eight times in both the vertical and horizontal directions, resulting in a 32×32 array quantization matrix. Note that the quantization matrix to be generated is not limited to this. If any size of quantization coefficients to be dequantized in subsequent stages exists other than 32×32, array quantization matrices corresponding to the size of the quantization coefficients to be dequantized, such as 16×16, 8×8, and 4×4 arrays, can be generated. The generated quantization matrix is ​​stored and used in subsequent dequantization processing.

[0124] Decoding unit 203 decodes code data from the bitstream and reconstructs quantization coefficients and prediction information. Based on the decoded prediction information, decoding unit 203 determines the size of one or more sub-blocks to be decoded, and further outputs the reconstructed quantization coefficients to inverse quantization and inverse transform unit 204, and outputs the reconstructed prediction information to image reconstruction unit 205. In this typical embodiment, regardless of the size of the sub-blocks (i.e., the size is...), Figure 7A Such a 64×64 array is still like Figure 7BFor such a 32×32 array, the quantization coefficients of the 32×32 array are reconstructed for each sub-block to be decoded.

[0125] The inverse quantization and inverse transform unit 204 generates orthogonal transform coefficients by inverse quantization of the input quantization coefficients using quantization parameters and the quantization matrix reconstructed by the quantization matrix decoding unit 209, and further applies an inverse orthogonal transform to reconstruct the prediction residuals. The inverse quantization and inverse orthogonal transform processing is described in more detail below.

[0126] If you choose Figure 7B The sub-block partitioning of the 32×32 array is achieved by using... Figure 8B The quantization matrix is ​​used to inversely quantize the quantization coefficients of the 32×32 array reconstructed by decoding unit 203 to reconstruct the orthogonal transform coefficients of the 32×32 array. Then, the transpose of the aforementioned 32×32 array and the orthogonal transform coefficients of the 32×32 array are multiplied to calculate intermediate coefficients in the form of a matrix of the 32×32 array. The intermediate coefficients in the form of a matrix of the 32×32 array are multiplied with the aforementioned orthogonal transform matrix of the 32×32 array to reconstruct the prediction residual of the 32×32 array. Similar processing is performed on each sub-block of the 32×32 array.

[0127] On the other hand, if Figure 7A Choosing to not split would then be achieved by using... Figure 8C The quantization matrix is ​​used to inversely quantize the array of quantization coefficients reconstructed by decoding unit 203 to reconstruct the orthogonal transform coefficients of the 32×32 array. The transpose of the aforementioned 32×64 array and the orthogonal transform coefficients of the 32×32 array are multiplied to calculate intermediate coefficients in matrix form using the 32×64 array. These intermediate coefficients in matrix form using the 32×64 array are multiplied by the aforementioned orthogonal transform matrix of the 64×32 array to reconstruct the prediction residuals of the 64×64 array.

[0128] The reconstructed prediction residual is output to image reconstruction unit 205. In this typical embodiment, the quantization matrix to be used in the inverse quantization process is determined based on the size of the sub-block to be decoded, as determined by the prediction information reconstructed by decoding unit 203. More specifically, for Figure 7B The individual sub-blocks of the 32×32 array are used in the inverse quantization process. Figure 8B The quantization matrix. For Figure 7A The 64x64 array of sub-blocks, using Figure 8C The quantization matrix is ​​specified. Note that the quantization matrix used is not limited to this; the same quantization matrix may be used as long as the transform and quantization unit 105 and the inverse quantization and inverse transform unit 106 in the first typical embodiment use the same quantization matrix.

[0129] The image reconstruction unit 205, based on the prediction information input from the decoding unit 203, obtains the data required to reconstruct the prediction image by appropriately referencing the frame memory 206, and reconstructs the prediction image. In this exemplary embodiment, the image reconstruction unit 205 uses both intra-frame prediction and inter-frame prediction methods, similar to the prediction unit 104 in the first exemplary embodiment. As described above, a combination of intra-frame prediction and inter-frame prediction methods can be used. As in the first exemplary embodiment, prediction processing is performed on a sub-block basis.

[0130] Since the specific prediction processing is similar to that of the prediction unit 104 according to the first exemplary embodiment, a description of the prediction processing will be omitted. The image reconstruction unit 205 reconstructs image data based on the prediction image generated by the prediction processing and the prediction residual input from the inverse quantization and inverse transform unit 204. Specifically, the image reconstruction unit 205 reconstructs image data by adding the prediction image and the prediction residual. The reconstructed image data is appropriately stored in the frame memory 206. The stored image data is appropriately referenced when predicting other sub-blocks.

[0131] and Figure 1 Similar to the in-loop filtering unit 109, the in-loop filtering unit 207 reads the reconstructed image from the frame memory 206 and performs in-loop filtering processes such as deblocking filtering. The filtered image is then input back into the frame memory 206.

[0132] The reconstructed image stored in frame memory 206 is eventually output to an external device via terminal 208. For example, the reconstructed image is output to an external display device, etc.

[0133] Figure 4 This is a flowchart illustrating image decoding processing using an image decoding device according to a second exemplary embodiment.

[0134] In step S401, the separation and decoding unit 202 first separates the bitstream into information related to the decoding process and code data related to the coefficients, and then decodes the code data in the header portion. More specifically, the separation and decoding unit 202 reconstructs the quantization matrix code data.

[0135] In step S402, the quantization matrix decoding unit 209 first decodes the quantization matrix code data reconstructed in step S401 to reconstruct... Figure 10 The one-dimensional difference matrix is ​​shown. Next, the quantization matrix decoding unit 209 reconstructs a two-dimensional basis quantization matrix based on the reconstructed one-dimensional difference matrix. The quantization matrix decoding unit 209 further expands the reconstructed two-dimensional basis quantization matrix to generate a quantization matrix.

[0136] More specifically, in this typical embodiment, the quantization matrix decoding unit 209 uses... Figure 9The scanning method shown is based on Figure 10 The difference matrix reconstruction shown Figure 8A The base quantization matrix is ​​shown. The quantization matrix decoding unit 209 further expands the reconstructed base quantization matrix to generate... Figure 8B and Figure 8C The quantization matrices shown are used to store these quantization matrices.

[0137] In step S403, decoding unit 203 decodes the code data separated in step S401 to reconstruct quantization coefficients and prediction information. Decoding unit 203 further determines the size of one or more sub-blocks to be decoded based on the decoded prediction information. In this typical embodiment, regardless of the size of the sub-block (i.e., the size is...), Figure 7A Such a 64×64 array is still like Figure 7B For such a 32×32 array, the quantization coefficients of the 32×32 array are reconstructed for each sub-block to be decoded.

[0138] In step S404, the inverse quantization and inverse transform unit 204 obtains orthogonal transform coefficients by inverse quantization of the quantization coefficients using the quantization matrix reconstructed in step S402, and further performs an inverse orthogonal transform to reconstruct the prediction residual. In this typical embodiment, the quantization matrix to be used in the inverse quantization process is determined based on the size of the sub-block to be decoded, as determined by the prediction information reconstructed in step S403. More specifically, for Figure 7B The individual sub-blocks of the 32×32 array are used in the inverse quantization process. Figure 8B The quantization matrix. For Figure 7A The 64x64 array of sub-blocks, using Figure 8C The quantization matrix is ​​used. Note that the quantization matrix used is not limited to this; the same quantization matrix can be used in steps S306 and S307 of the first typical embodiment.

[0139] In step S405, the image reconstruction unit 205 reconstructs the predicted image based on the prediction information generated in step S403. In this typical embodiment, as in step S305 of the first typical embodiment, both intra-frame prediction and inter-frame prediction methods are used. The image reconstruction unit 205 further reconstructs image data based on the reconstructed predicted image and the prediction residual generated in step S404.

[0140] In step S406, the image decoding device determines whether all basic blocks in the frame have been decoded. If all basic blocks have been decoded, the process proceeds to step S407. If not all basic blocks have been decoded, the process returns to step S403 with the next basic block as the object.

[0141] In step S407, the in-loop filtering unit 207 performs in-loop filtering on the image data reconstructed in step S405 to generate a filtered image. Processing ends.

[0142] Through the above structure and operation, the bitstream generated in the first typical embodiment can be decoded, wherein for this bitstream, even in sub-blocks where only low-frequency orthogonal transform coefficients are quantized and encoded, a quantization matrix is ​​used to control the quantization of each frequency component to improve subjective image quality. Furthermore, by using, for example... Figure 8C The quantization matrix obtained by expanding only the low-frequency part of the base quantization matrix is ​​used to quantize and encode only the low-frequency orthogonal transform coefficients of the sub-blocks, so as to decode the bitstream in which optimal quantization control is applied to its low-frequency part.

[0143] In this typical embodiment, in order to reduce the amount of code, only the code to be used in common for generation is used. Figure 8B and Figure 8C quantization matrix Figure 8A Decoding is performed using the base quantization matrix. However, Figure 8B and Figure 8C The quantization matrix itself can be decoded. Because unique values ​​can be set for each frequency component of the quantization matrix, this enables the decoding of bitstreams with finer quantization control for each frequency component.

[0144] Can target Figure 8B and Figure 8C Each base quantization matrix can be set separately, and each base quantization matrix can be encoded. In this case, the orthogonal transform coefficients of the 32×32 array and the orthogonal transform coefficients of the 64×64 array can be quantized separately to decode the bitstream that achieves more nuanced control over subjective image quality. Furthermore, in this case, instead of scaling the upper left 4×4 array portion of the 8×8 array's base quantization matrix by a factor of eight, the entire base quantization matrix of the 8×8 array can be scaled by a factor of four to obtain the quantization matrix corresponding to the orthogonal transform coefficients of the 64×64 array. This allows for even finer quantization control over the orthogonal transform coefficients of the 64×64 array.

[0145] Furthermore, although this typical embodiment is configured such that the quantization matrix for a sub-block of the 64×64 array to be zeroed is uniquely determined, identifiers can be introduced to enable selection. For example, Figure 6B The bitstream is shown below: A new quantization matrix encoding method information code is introduced to make the quantization matrix encoding of sub-blocks of the 64×64 array to be zeroed selectable. For example, if the quantization matrix encoding method information code indicates 0, then it will be used as an independent quantization matrix. Figure 8COrthogonal transform coefficients are used for the sub-blocks of the 64×64 array to be zeroed. If the encoding method information code indicates 1, these will be used as the quantization matrix for ordinary sub-blocks that are not to be zeroed. Figure 8B For a sub-block of the 64×64 array to be zeroed. If the encoding method information code indicates 2, then instead of the elements of the base quantization matrix of the 8×8 array, all elements of the quantization matrix to be used for the sub-block of the 64×64 array to be zeroed are encoded. This makes it possible to decode a bitstream for which the amount of quantization matrix code can be reduced and unique quantization control for the sub-block to be zeroed can be implemented in an optional manner.

[0146] In this typical embodiment, the sub-blocks processed by zeroing are only sub-blocks of a 64×64 array. However, the sub-blocks processed by zeroing are not limited to this. For example, in conjunction with... Figure 7C or Figure 7D In the orthogonal transform coefficients corresponding to the sub-blocks of the 32×64 or 64×32 array shown, decoding can be performed only on the quantization coefficients of the upper or left half of the 32×32 array without needing to decode the orthogonal transform coefficients of the lower or right half of the 32×32 array. In this case, only the orthogonal transform coefficients of the upper or left half of the 32×32 array are decoded and inverse quantized. By using... Figure 8B Different quantization matrices are used to quantize the orthogonal transformation coefficients of the 32×32 array in the upper or left half.

[0147] Furthermore, the values ​​of the quantization matrix corresponding to the top-left DC coefficients, which are considered to have the greatest impact on image quality, can be decoded and set separately from the values ​​of the elements of the 8×8 array's basis matrix. Figure 12B and Figure 12C Showing the value of the top left element corresponding to the DC component from Figure 8B and Figure 8C Examples of value changes in [the context]. In this case, it can be achieved by removing the [partial] pair from the [context]. Figure 8A In addition to decoding information related to the base quantization matrix, the information of the indicator "2" at the position of the DC part is also decoded to set... Figure 12B and Figure 12C The quantization matrix is ​​shown. This enables decoding of bitstreams where finer quantization control is achieved for the DC components of the orthogonal transform coefficients, which have the greatest impact on image quality.

[0148] <Third Embodiment>

[0149] In the aforementioned typical embodiments, Figure 1 and Figure 2The processing units shown have been described as being constructed of hardware. However, the processing performed by the various processing units shown in these figures can be constructed by computer programs.

[0150] Figure 5 This is a block diagram illustrating an example of the hardware structure of a computer that can be applied to the image encoding device and image decoding device according to the foregoing exemplary embodiments.

[0151] The Central Processing Unit (CPU) 501 controls the entire computer by using computer programs and data stored in Random Access Memory (RAM) 502 and Read-Only Memory (ROM) 503, and performs the aforementioned processing to be performed by the device according to a typical embodiment. In other words, the CPU 501 serves as... Figure 1 and Figure 2 The processing unit shown.

[0152] RAM 502 includes areas for temporarily storing computer programs and data loaded from external storage device 506, as well as data obtained from the outside via interface (I / F) 507. RAM 502 also includes working areas for CPU 501 to use when performing various processes. In other words, RAM 502 can be allocated, for example, as frame memory, and appropriately provided with various other areas.

[0153] ROM 503 stores computer settings data and boot programs, etc. Operation unit 504 includes a keyboard and mouse. The computer user can input various instructions into CPU 501 by operating operation unit 504. Output unit 505 outputs the results processed by CPU 501. Output unit 505 may include, for example, a liquid crystal display (LCD).

[0154] External storage device 506 is a large-capacity information storage device, represented by a hard disk drive. External storage device 506 stores the operating system (OS) and the components used by the CPU 501 to implement... Figure 1 and Figure 2 The computer program shows the functions of the unit. External storage device 506 can also store image data to be processed.

[0155] The computer programs and data stored in external storage device 506 are appropriately loaded into RAM 502 under the control of CPU 501 and processed by CPU 501. Networks such as local area networks (LANs) and the Internet, as well as other devices such as projection devices and display devices, can be connected to I / F 507. The computer can obtain and send various types of information via I / F 507. Bus 508 connects the above-mentioned units.

[0156] The activities of the aforementioned structure or the activities described in the aforementioned flowchart are primarily controlled by CPU 501.

[0157] (Other typical embodiments)

[0158] A typical embodiment can also be implemented by supplying a storage medium containing computer program code for implementing the aforementioned functions to the system, and having the system read and execute the computer program code. In this case, the computer program code read from the storage medium itself implements the functions of the aforementioned typical embodiment, and the storage medium storing the computer program code constitutes the present invention. It also includes cases where an operating system (OS) running on a computer performs some or all of the actual processing based on instructions in the program code, and implements the aforementioned functions through this processing.

[0159] A typical embodiment can also be implemented in the following way. That is, computer program code read from a storage medium is written into a memory included in a function expansion card inserted into a computer or a function expansion unit connected to the computer. Then, the CPU or other components included in the function expansion card or function expansion unit perform some or all of the actual processing based on the instructions in the computer program code, thereby realizing the above-mentioned function. Such cases are also included.

[0160] If the present invention is applied to the above-described storage medium, the storage medium stores computer program code corresponding to the above flowchart.

[0161] According to the aforementioned typical embodiments, although a technique is used to force a portion of the orthogonal transform coefficients to be 0, subjective image quality can also be improved by enabling quantization processing using a quantization matrix corresponding to the technique.

[0162] This invention is not limited to the embodiments described above, and various changes and modifications can be made within the spirit and scope of this invention. Therefore, the appended claims have been added to inform the public of the scope of this invention.

[0163] This application claims priority to Japanese Patent Application 2019-044275, filed on March 11, 2019, which is incorporated herein by reference in its entirety.

Claims

1. An image decoding device capable of decoding an image from a bitstream using multiple blocks, said multiple blocks comprising a first block of pixels in a P×Q array and a second block of pixels in an N×M array, wherein, P and Q are integers, N is an integer satisfying N < P, and M is an integer satisfying M < Q. The image decoding device includes: a decoding unit configured to decode data corresponding to a first quantized transform coefficient array and data corresponding to a second quantized transform coefficient array from the bitstream, the first quantized transform coefficient array corresponding to the first block and the second quantized transform coefficient array corresponding to the second block; an inverse quantization unit configured to derive a first transform coefficient array from the first quantized transform coefficient array by using a first quantization matrix of elements of an N×M array, and to derive a second transform coefficient array from the second quantized transform coefficient array by using a second quantization matrix of elements of the N×M array, wherein the first transform coefficient array represents frequency components, and wherein the second transform coefficient array represents frequency components; an inverse transform unit configured to derive a first prediction residual array corresponding to the first block by performing an inverse transform process on the first transform coefficient array, and to derive a second prediction residual array corresponding to the second block by performing an inverse transform process on the second transform coefficient array; and a picture reconstruction unit configured to at least reconstruct picture data based on the first prediction residual array and prediction picture data, the prediction picture data being able to be derived by using a prediction method that combines intra-prediction and inter-prediction, wherein the first quantization matrix of elements of the N×M array is a quantization matrix that includes some elements of a third quantization matrix of elements of an R×S array but does not include other elements of the third quantization matrix, wherein R is an integer satisfying R ≤ N, and S is an integer satisfying S ≤ M, wherein the second quantization matrix of elements of the N×M array is a quantization matrix that includes all elements of a fourth quantization matrix of elements of the R×S array, and wherein the third quantization matrix is different from the fourth quantization matrix.

2. An image decoding method capable of decoding an image from a bitstream using multiple blocks, said multiple blocks comprising a first block of pixels in a P×Q array and a second block of pixels in an N×M array, wherein, P and Q are integers, N is an integer satisfying N < P, and M is an integer satisfying M < Q. The image decoding method includes: decoding data corresponding to a first quantized transform coefficient array and data corresponding to a second quantized transform coefficient array from the bitstream, the first quantized transform coefficient array corresponding to the first block and the second quantized transform coefficient array corresponding to the second block; deriving a first transform coefficient array from the first quantized transform coefficient array by using a first quantization matrix of elements of an N×M array, and deriving a second transform coefficient array from the second quantized transform coefficient array by using a second quantization matrix of elements of the N×M array, wherein the first transform coefficient array represents frequency components, and wherein the second transform coefficient array represents frequency components; deriving a first prediction residual array corresponding to the first block by performing an inverse transform process on the first transform coefficient array, and deriving a second prediction residual array corresponding to the second block by performing an inverse transform process on the second transform coefficient array; and At least reconstruct picture data based on the first prediction residual array and prediction image data, the prediction image data being derivable by using a prediction method that combines intra-frame prediction and inter-frame prediction. Wherein, the first quantization matrix of elements of the N×M array is a quantization matrix as follows, the quantization matrix including a part of the elements of a third quantization matrix of elements of an R×S array but not including other elements of the third quantization matrix, wherein R is an integer satisfying R≤N, and S is an integer satisfying S≤M. Wherein, the second quantization matrix of elements of the N×M array is a quantization matrix including all elements of a fourth quantization matrix of elements of an R×S array, and wherein, the third quantization matrix is different from the fourth quantization matrix.

3. The image decoding method according to claim 2, in, the first quantization matrix is a quantization matrix in which elements other than the element corresponding to the DC component include the part of the elements of the third quantization matrix, and wherein, the second quantization matrix is a quantization matrix in which elements other than the element corresponding to the DC component include all elements of the fourth quantization matrix.

4. The image decoding method according to claim 2, in, the first transform coefficient array includes transform coefficients of an N×M array, and wherein, the second transform coefficient array includes transform coefficients of an N×M array.

5. The image decoding method according to claim 2, wherein, The first block and the second block are non-square blocks.

6. The image decoding method according to claim 2, in, the first prediction residual array is prediction residuals of a P×Q array, and wherein, the second prediction residual array is prediction residuals of an N×M array.

7. The image decoding method according to claim 2, in, the first transform coefficient array is transform coefficients of an N×M array, wherein, the first prediction residual array is prediction residuals of a P×Q array, and wherein, the first prediction residual array is derived from the first transform coefficient array by an inverse transform process performed on the first transform coefficient array.

8. The image decoding method according to claim 7, wherein, The inverse transform process includes multiplying the first transform coefficient array by a matrix of an M×Q array to thereby derive an intermediate value of an N×Q array, and the inverse transform process includes multiplying a matrix of a P×N array by the intermediate value of the N×Q array to thereby derive the first prediction residual array from the first transform coefficient array.

9. A non-transitory computer-readable storage medium storing a program, the program being for causing a computer to execute the image decoding method according to claim 2.

10. An image encoding apparatus capable of encoding an image using multiple blocks, said multiple blocks comprising a first block of pixels in a P×Q array and a second block of pixels in an N×M array, wherein, P and Q are integers, N is an integer satisfying N<P, and M is an integer satisfying M<Q, and the image encoding device includes: A transform unit configured to derive a first transform coefficient array by performing a transform process on a first prediction residual array corresponding to the first block, and to derive a second transform coefficient array by performing a transform process on a second prediction residual array corresponding to the second block, at least the first prediction residual array being derivable by using a prediction method that combines intra-frame prediction and inter-frame prediction; A quantization unit configured to derive a first quantized transform coefficient array by quantizing the first transform coefficient array using a first quantization matrix of elements of an N×M array, and to derive a second quantized transform coefficient array by quantizing the second transform coefficient array using a second quantization matrix of elements of the N×M array; and An encoding unit configured to encode data corresponding to the first quantized transform coefficient array corresponding to the first block and data corresponding to the second quantized transform coefficient array corresponding to the second block, wherein the first quantization matrix of elements of the N×M array is a quantization matrix that includes a part of the elements of a third quantization matrix of elements of an R×S array but does not include other elements of the third quantization matrix, where R is an integer satisfying R≤N and S is an integer satisfying S≤M, wherein the second quantization matrix of elements of the N×M array is a quantization matrix that includes all elements of a fourth quantization matrix of elements of the R×S array, and wherein the third quantization matrix is different from the fourth quantization matrix.

11. An image encoding method capable of encoding an image using multiple blocks, said multiple blocks comprising a first block of pixels in a P×Q array and a second block of pixels in an N×M array, wherein, P and Q are integers, N is an integer satisfying N<P, and M is an integer satisfying M<Q. The image encoding method includes: Deriving a first transform coefficient array by performing a transform process on a first prediction residual array corresponding to the first block, and deriving a second transform coefficient array by performing a transform process on a second prediction residual array corresponding to the second block, wherein at least the first prediction residual array can be derived by using a prediction method that combines intra-frame prediction and inter-frame prediction; Deriving a first quantized transform coefficient array by quantizing the first transform coefficient array using a first quantization matrix of elements of an N×M array, and deriving a second quantized transform coefficient array by quantizing the second transform coefficient array using a second quantization matrix of elements of the N×M array; and Encoding data corresponding to the first quantized transform coefficient array corresponding to the first block and data corresponding to the second quantized transform coefficient array corresponding to the second block, wherein the first quantization matrix of elements of the N×M array is a quantization matrix that includes a part of the elements of a third quantization matrix of elements of an R×S array but does not include other elements of the third quantization matrix, where R is an integer satisfying R≤N and S is an integer satisfying S≤M, wherein the second quantization matrix of elements of the N×M array is a quantization matrix that includes all elements of a fourth quantization matrix of elements of the R×S array, and wherein the third quantization matrix is different from the fourth quantization matrix.

12. The image encoding method according to claim 11, in, wherein the first quantization matrix is a quantization matrix in which elements other than the element corresponding to the DC component include the part of the elements of the third quantization matrix, and wherein the second quantization matrix is a quantization matrix in which elements other than the element corresponding to the DC component include all elements of the fourth quantization matrix.

13. The image encoding method according to claim 11, in, The first transform coefficient array includes transform coefficients of an N×M array, and The second transformation coefficient array includes transformation coefficients of an N×M array.

14. The image encoding method according to claim 11, wherein, The first and second blocks are non-square blocks.

15. The image encoding method according to claim 11, in, The first prediction residual array is the prediction residual of a P×Q array, and The second prediction residual array is the prediction residual of an N×M array.

16. The image encoding method according to claim 11, in, The first prediction residual array is the prediction residual of a P×Q array. Wherein, the first transform coefficient array is the transform coefficient of an N×M array, and Specifically, the first transformation coefficient array is derived from the first prediction residual array by performing a transformation process on the first prediction residual array.

17. The image encoding method according to claim 16, wherein, The transformation process includes multiplying the matrix of the first prediction residual array with the matrix of the Q×M array to derive the intermediate values ​​of the P×M array, and the transformation process includes multiplying the matrix of the N×P array with the intermediate values ​​of the P×M array to derive the first transformation coefficient array from the first prediction residual array.

18. A non-transitory computer-readable storage medium storing a program for causing a computer to perform the image encoding method according to claim 11.

Images