Image encoding device, image encoding method, and program

Abstract

To more efficiently perform a method that forcibly sets some orthogonal conversion coefficients to 0.SOLUTION: Orthogonal conversion means performs orthogonal conversion for a prediction error of a rectangular block of P×Q pixels, and thereby generates N×M orthogonal conversion coefficients, where N is an integer satisfying N<P and M is an integer satisfying M<Q. Quantization means quantizes the N×M orthogonal conversion coefficients using at least a quantization matrix including N×M elements, and generates N×M quantization coefficients.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to image coding technology. [Background technology]

[0002] The High Efficiency Video Coding (HEVC) coding method (hereafter referred to as HEVC) is known as a coding method for compressing moving images. To improve coding efficiency, HEVC uses basic blocks larger than conventional macroblocks (16x16 pixels). These large basic blocks are called Coding Tree Units (CTUs) and can be up to 64x64 pixels in size. CTUs are further divided into sub-blocks, which serve as units for prediction and transformation.

[0003] Furthermore, in HEVC, a quantization matrix is ​​used to weight coefficients after orthogonal transformation (hereinafter referred to as orthogonal transform coefficients) according to frequency components. The use of a quantization matrix makes it possible to improve compression efficiency while maintaining image quality by reducing high-frequency component data, degradation of which is less noticeable to human vision, more than low-frequency component data. Japanese Patent Application Laid-Open No. 2013-38758 (Patent Document 1) discloses a technology for encoding information indicating such a quantization matrix.

[0004] Recently, efforts have been initiated to internationally standardize a more efficient coding method as a successor to HEVC. Specifically, the Joint Video Experts Team (JVET), established by ISO / IEC and ITU-T, is working on standardizing the Versatile Video Coding (VVC) coding method (hereinafter referred to as VVC). To improve coding efficiency, this standardization is studying a new method (hereinafter referred to as zero-out) for reducing the amount of code by forcibly setting the orthogonal transform coefficients of high-frequency components to zero when the block size during orthogonal transform is large. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2013-38758 Summary of the Invention [Problem to be solved by the invention]

[0006] Therefore, an object of the present invention is to more efficiently execute a method for forcibly setting some orthogonal transform coefficients to 0. [Means for solving the problem]

[0007] To solve the above problems, the image decoding apparatus of the present invention has the following configuration. That is, in an image decoding apparatus capable of decoding an image from a bit stream using a plurality of blocks including a first block of P×Q pixels (P and Q are integers) and a second block of N×M pixels (N is an integer satisfying N<P, and M is an integer satisfying M<Q), decoding means for decoding from the bit stream data corresponding to a first quantization conversion coefficient group corresponding to the first block and data corresponding to a second quantization conversion coefficient group corresponding to the second block; inverse quantization means for deriving a first conversion coefficient group representing frequency components from the first quantization conversion coefficient group using a first quantization matrix having N×M elements and deriving a second conversion coefficient group representing frequency components from the second quantization conversion coefficient group using a second quantization matrix having N×M elements; inverse conversion means for deriving a first prediction error group corresponding to the first block by performing an inverse conversion process on the first conversion coefficient group and deriving a second prediction error group corresponding to the second block by performing an inverse conversion process on the second conversion coefficient group; and a reproduction unit for at least reproducing image data based on the first prediction error group and prediction image data, wherein the prediction image data can be derived using a prediction method combining intra prediction and inter prediction. When the block to be decoded is the first block, the inverse conversion means derives N×Q intermediate values by multiplying the first conversion coefficient group, which is N×M conversion coefficients, by a matrix of M×Q, and further derives the first prediction error group, which is P×Q prediction errors from the first conversion coefficient group, by multiplying a matrix of P×N by the N×Q intermediate values. The first quantization matrix having N×M elements is a quantization matrix that includes some elements of a third quantization matrix having R×S (R is an integer satisfying R≦N, and S is an integer satisfying S≦M) elements and does not include other elements of the third quantization matrix. The second quantization matrix having N×M elements is a quantization matrix that includes all elements of a fourth quantization matrix having R×S elements. The third quantization matrix is different from the fourth quantization matrix.the first quantization matrix is ​​a quantization matrix that is configured with some of the elements in the third quantization matrix except for elements corresponding to DC components, and the second quantization matrix is ​​a quantization matrix that is configured with all of the elements in the fourth quantization matrix except for elements corresponding to DC components, The decoding means further decodes information indicating the value of the element corresponding to the DC component.

[0008] To solve the above problems, the image encoding apparatus of the present invention has the following configuration. That is, in an image encoding apparatus capable of encoding an image using a plurality of blocks including a first block of P×Q pixels (P and Q are integers) and a second block of N×M pixels (N is an integer satisfying N<P, and M is an integer satisfying M<Q), a first conversion coefficient group is derived by performing a conversion process on a first prediction error group corresponding to the first block, and a second conversion coefficient group is derived by performing a conversion process on a second prediction error group corresponding to the second block. A conversion means, a quantization means for quantizing the first conversion coefficient group using a first quantization matrix having N×M elements to derive a first quantized conversion coefficient group, and quantizing the second conversion coefficient group using a second quantization matrix having N×M elements to derive a second quantized conversion coefficient group, and an encoding means for encoding data corresponding to the first quantized conversion coefficient group corresponding to the first block and data corresponding to the second quantized conversion coefficient group corresponding to the second block. At least the first prediction error group can be derived using a prediction method that combines intra prediction and inter prediction. When the block to be encoded is the first block, the conversion means derives P×M intermediate values by multiplying the first prediction error group, which is P×Q prediction errors, by a Q×M matrix, and further derives the first conversion coefficient group, which is N×M conversion coefficients from the first prediction error group, by multiplying an N×P matrix by the P×M intermediate values. The first quantization matrix having N×M elements is a quantization matrix that includes some elements in a third quantization matrix having R×S (R is an integer satisfying R≦N, and S is an integer satisfying S≦M) elements and does not include other elements in the third quantization matrix. The second quantization matrix having N×M elements is a quantization matrix that includes all elements in a fourth quantization matrix having R×S elements. The third quantization matrix is different from the fourth quantization matrix. The first quantization matrix, except for the element corresponding to the DC component,the second quantization matrix is ​​a quantization matrix configured with all of the elements in the fourth quantization matrix except for elements corresponding to DC components; The encoding means further encodes information indicating the value of the element corresponding to the DC component. [Effects of the Invention]

[0009] The present invention aims to more efficiently execute a method for forcibly setting some orthogonal transform coefficients to 0. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing the configuration of an image encoding device according to a first embodiment. [Figure 2] FIG. 10 is a block diagram showing the configuration of an image decoding device according to a second embodiment. [Figure 3] 4 is a flowchart showing an image encoding process in the image encoding device according to the first embodiment. [Figure 4] 10 is a flowchart showing an image decoding process in the image decoding device according to the second embodiment. [Figure 5] FIG. 1 is a block diagram showing an example of the hardware configuration of a computer applicable to the image encoding device and image decoding device of the present invention. [Figure 6] FIG. 10 is a diagram showing an example of a bitstream output in the first embodiment. [Figure 7] FIG. 2 is a diagram showing an example of sub-block division used in the first and second embodiments. [Figure 8] FIG. 2 is a diagram showing an example of a quantization matrix used in the first and second embodiments. [Figure 9] FIG. 3 is a diagram showing a method for scanning each element of a quantization matrix used in the first and second embodiments. [Figure 10] FIG. 10 is a diagram showing a difference value matrix of a quantization matrix generated in the first and second embodiments. [Figure 11]FIG. 10 is a diagram showing an example of a coding table used for coding difference values ​​of a quantization matrix. [Figure 12] FIG. 10 is a diagram showing another example of the quantization matrix used in the first and second embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0011] Embodiments of the present invention will be described with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the configurations described in the following embodiments. Note that terms such as basic block, sub-block, quantization matrix, and base quantization matrix are used for convenience in each embodiment, and other terms may be used as appropriate as long as their meanings remain unchanged. For example, basic blocks and sub-blocks may be referred to as basic units or sub-units, or simply as blocks or units. In the following description, a rectangle is generally defined as a quadrilateral having four right-angled interior angles and two diagonals of equal length. A square is generally defined as a quadrilateral having four equal corners and four equal sides. In other words, a square is a type of rectangle.

[0012] <Embodiment 1> Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0013] First, zeroing out will be described in more detail. As described above, zeroing out is a process of forcibly setting some of the orthogonal transform coefficients of a block to be coded to 0. For example, assume that a 64x64 pixel block in an input image (picture) is the block to be coded. In this case, the orthogonal transform coefficients also have a size of 64x64. Zeroing out is a process of encoding some of these 64x64 orthogonal transform coefficients by regarding them as 0 even if they have a non-zero value as a result of orthogonal transform. For example, low-frequency components corresponding to a predetermined range in the upper left corner including DC components in two-dimensional orthogonal transform coefficients are not forcibly set to 0, and orthogonal transform coefficients corresponding to frequency components higher than these low-frequency components are always set to 0.

[0014] Next, the image coding apparatus of this embodiment will be described. Fig. 1 is a block diagram showing the image coding apparatus of this embodiment. In Fig. 1, reference numeral 101 denotes a terminal for inputting image data.

[0015] A block dividing unit 102 divides an input image into a plurality of basic blocks and outputs an image in units of basic blocks to a subsequent stage.

[0016] Reference numeral 103 denotes a quantization matrix storage unit that generates and stores a quantization matrix. Here, the quantization matrix is ​​used to weight the quantization process for orthogonal transform coefficients according to frequency components. In the quantization process described below, the quantization step for each orthogonal transform coefficient is weighted by, for example, multiplying a scale value (quantization scale) based on a reference parameter value (quantization parameter) by the value of each element in the quantization matrix.

[0017] The method for generating the quantization matrices stored by the quantization matrix storage unit 110 is not particularly limited. For example, the user may input information indicating the quantization matrix, or the image encoding device may calculate the quantization matrix based on the characteristics of the input image. Alternatively, a predetermined initial value may be used. In this embodiment, in addition to the 8×8 base quantization matrix shown in FIG. 8(a), two types of 32×32 two-dimensional quantization matrices shown in FIGS. 8(b) and 8(c) are generated and stored by enlarging the base quantization matrix. The quantization matrix in FIG. 8(b) is a 32×32 quantization matrix obtained by enlarging the 8×8 base quantization matrix in FIG. 8(a) by four times by repeating each element in the vertical and horizontal directions. On the other hand, the quantization matrix in FIG. 8(c) is a 32×32 quantization matrix obtained by enlarging the 8×8 base quantization matrix in FIG. 8(a) by eight times by repeating each element in the upper left 4×4 portion.

[0018] As described above, the base quantization matrix is ​​a quantization matrix that is used not only for quantization of 8x8 pixel sub-blocks, but also for creating quantization matrices of a larger size than the base quantization matrix. It is assumed that the size of the base quantization matrix is ​​8x8, but this size is not limited to this. Also, different base quantization matrices may be used depending on the size of the sub-block. For example, if three types of sub-blocks, 8x8, 16x16, and 32x32, are used, three corresponding types of base quantization matrices may be used.

[0019] A prediction unit 104 determines subblock division for image data in units of basic blocks. That is, it determines whether to divide a basic block into subblocks, and if so, how to divide it. If not divided into subblocks, the subblocks will have the same size as the basic block. The subblocks may be square or may be rectangular in shape.

[0020] Then, the prediction unit 104 performs intra-prediction, which is intra-frame prediction, or inter-prediction, which is inter-frame prediction, on a sub-block basis to generate predicted image data.

[0021] For example, the prediction unit 104 selects a prediction method to be performed on one sub-block from intra prediction and inter prediction, performs the selected prediction, and generates predicted image data for the sub-block. However, the prediction method used is not limited to these, and a prediction that combines intra prediction and inter prediction may also be used.

[0022] Furthermore, the prediction unit 104 calculates and outputs a prediction error from the input image data and the predicted image data. For example, the prediction unit 104 calculates the difference between each pixel value of a sub-block and each pixel value of the predicted image data generated by predicting the sub-block, and calculates the difference as a prediction error.

[0023] The prediction unit 104 also outputs information necessary for prediction, such as information indicating the division state of the sub-block, a prediction mode indicating the prediction method for the sub-block, and information such as a motion vector, together with the prediction error. Hereinafter, this information necessary for prediction will be collectively referred to as prediction information.

[0024] Reference numeral 105 denotes a transform / quantization unit. The transform / quantization unit 105 performs orthogonal transform on the prediction error calculated by the prediction unit 104 in units of sub-blocks to obtain orthogonal transform coefficients representing each frequency component of the prediction error. The transform / quantization unit 105 then performs quantization using the quantization matrix stored in the quantization matrix storage unit 103 and the quantization parameter to obtain quantized coefficients, which are quantized orthogonal transform coefficients. Note that the function for performing orthogonal transform and the function for performing quantization may be configured separately.

[0025] Reference numeral 106 denotes an inverse quantization and inverse transform unit. The inverse quantization and inverse transform unit 106 inverse quantizes the quantized coefficients output from the transform and quantization unit 105 using the quantization matrix stored in the quantization matrix storage unit 103 and the quantization parameter to reproduce orthogonal transform coefficients. The inverse quantization and inverse transform unit 106 then performs inverse orthogonal transform to reproduce prediction errors. This process of reproducing (deriving) orthogonal transform coefficients using the quantization matrix and the quantization parameter is referred to as inverse quantization. Note that the function of performing inverse quantization and the function of performing inverse quantization may be configured separately. Information used by the image decoding device to derive the quantization parameter is also coded into a bitstream by the coding unit 110.

[0026] Reference numeral 108 denotes a frame memory for storing reproduced image data.

[0027] An image reproduction unit 107 generates predicted image data by appropriately referencing a frame memory 108 based on the prediction information output from the prediction unit 104, and generates and outputs reproduced image data from this and the input prediction error.

[0028] An in-loop filter unit 109 performs in-loop filtering such as deblocking filtering and sample adaptive offset on the reconstructed image, and outputs the filtered image.

[0029] An encoding unit 110 encodes the quantization coefficients output from the transform / quantization unit 105 and the prediction information output from the prediction unit 104 to generate and output coded data.

[0030] A quantization matrix encoding unit 113 encodes the base quantization matrix output from the quantization matrix storage unit 103, and generates and outputs quantization matrix code data for use by the image decoding device to derive the base quantization matrix.

[0031] An integrated coding unit 111 generates header code data using the quantization matrix code data output from the quantization matrix coding unit 113. The integrated coding unit 111 then combines the header code data with the code data output from the coding unit 110 to form a bit stream and outputs it.

[0032] A terminal 112 outputs the bit stream generated by the integrated encoding unit 111 to the outside.

[0033] The image encoding operation in the image encoding device is described below. In this embodiment, moving image data is input in frame units. Furthermore, for the sake of explanation, this embodiment will be described assuming that the block division unit 101 divides the data into basic blocks of 64 x 64 pixels, but is not limited to this. For example, the basic blocks may be blocks of 128 x 128 pixels or blocks of 32 x 32 pixels.

[0034] Before encoding an image, an image encoding device generates and encodes a quantization matrix. In the following description, as an example, the horizontal direction of the quantization matrix 800 and each block is referred to as the x coordinate and the vertical direction as the y coordinate, with the rightward direction being positive and the downward direction being positive, respectively. The coordinates of the upper left element of the quantization matrix 800 are (0,0). That is, the coordinates of the lower right element of an 8x8 base quantization matrix are (7,7). The coordinates of the lower right element of a 32x32 quantization matrix are (31,31).

[0035] First, the quantization matrix storage unit 103 generates a quantization matrix. The quantization matrix is ​​generated according to the size of the sub-block, the size of the orthogonal transform coefficients to be quantized, and the type of prediction method. In this embodiment, first, an 8×8 base quantization matrix is ​​generated that is used to generate a quantization matrix (described later) and is shown in FIG. 8(a). Next, this base quantization matrix is ​​enlarged to generate two types of 32×32 quantization matrices shown in FIG. 8(b) and FIG. 8(c). The quantization matrix in FIG. 8(b) is a 32×32 quantization matrix obtained by enlarging each element of the 8×8 base quantization matrix in FIG. 8(a) by four times in the vertical and horizontal directions.

[0036] That is, in the example shown in Fig. 8(b), each element in the 32x32 quantization matrix whose x coordinate is in the range of 0 to 3 and whose y coordinate is in the range of 0 to 3 is assigned the value of 1, which is the value of the element at the top left of the base quantization matrix. Also, each element in the 32x32 quantization matrix whose x coordinate is in the range of 28 to 31 and whose y coordinate is in the range of 28 to 31 is assigned the value of 15, which is the value of the element at the bottom right of the base quantization matrix. In the example of Fig. 8(b), all of the values ​​of each element in the base quantization matrix are assigned to one of the elements of the 32x32 quantization matrix.

[0037] On the other hand, the quantization matrix in Figure 8(c) is a 32x32 quantization matrix that is expanded by repeating each element of the top left 4x4 part of the base quantization matrix in Figure 8(a) eight times in the vertical and horizontal directions.

[0038] That is, in the example shown in Fig. 8(c), each element in the 32x32 quantization matrix whose x coordinate is in the range of 0 to 7 and whose y coordinate is in the range of 0 to 7 is assigned 1, which is the value of the upper left element in the upper left 4x4 section of the base quantization matrix. Also, each element in the 32x32 quantization matrix whose x coordinate is in the range of 24 to 31 and whose y coordinate is in the range of 24 to 31 is assigned 7, which is the value of the lower right element in the upper left 4x4 section of the base quantization matrix. In the example of Fig. 8(c), only the element values ​​corresponding to the upper left 4x4 section (x coordinate range of 0 to 3 and y coordinate range of 0 to 3) of the values ​​of each element in the base quantization matrix are assigned to each element of the 32x32 quantization matrix.

[0039] However, the generated quantization matrix is ​​not limited to this, and if the size of the orthogonal transform coefficients to be quantized is other than 32×32, a quantization matrix corresponding to the size of the orthogonal transform coefficients to be quantized, such as 16×16, 8×8, or 4×4, may be generated. The method of determining the base quantization matrix and each element constituting the quantization matrix is ​​not particularly limited. For example, a predetermined initial value may be used, or each element may be set individually. Alternatively, the quantization matrix may be generated according to the characteristics of the image.

[0040] The quantization matrix storage unit 103 stores the base quantization matrix and quantization matrices generated in this manner. FIG. 8(b) shows an example of a quantization matrix used to quantize orthogonal transform coefficients corresponding to a 32×32 sub-block (described later), and FIG. 8(c) shows an example of a quantization matrix used to quantize orthogonal transform coefficients corresponding to a 64×64 sub-block. The bold frame 800 represents a quantization matrix. For ease of explanation, each is assumed to have a configuration of 1024 pixels (32×32), and each square within the bold frame represents an element constituting the quantization matrix. In this embodiment, the three types of quantization matrices shown in FIGS. 8(b) and 8(c) are stored in a two-dimensional form, but the elements in the quantization matrix are not limited to this. It is also possible to store multiple quantization matrices for the same prediction method depending on the size of the orthogonal transform coefficients to be quantized or whether the encoding target is a luminance block or a chrominance block. Generally, a quantization matrix realizes quantization processing that corresponds to the characteristics of human vision, so as shown in Figures 8(b) and (c), the elements in the low-frequency part corresponding to the upper left part of the quantization matrix are small, and the elements in the high-frequency part corresponding to the lower right part are large.

[0041] The quantization matrix encoding unit 113 sequentially reads each element of the base quantization matrix stored in two-dimensional form from the quantization matrix storage unit 106, scans each element to calculate a difference, and arranges each difference in a one-dimensional matrix. In this embodiment, the base quantization matrix shown in FIG. 8(a) uses the scanning method shown in FIG. 9, and calculates the difference between each element and the previous element in the scanning order. For example, the 8×8 base quantization matrix shown in FIG. 8(a) is scanned using the scanning method shown in FIG. 9, but after the first element 1 located in the upper left, the element 2 located immediately below it is scanned and the difference, +1, is calculated. Furthermore, to encode the first element of the quantization matrix (1 in this embodiment), the difference from a predetermined initial value (e.g., 8) is calculated; however, this is not limited to this, and a difference from an arbitrary value or the value of the first element itself may also be used.

[0042] In this manner, in this embodiment, the base quantization matrix of Fig. 8(a) is scanned using the scanning method of Fig. 9, and the difference matrix shown in Fig. 10 is generated. The quantization matrix encoding unit 113 further encodes the difference matrix to generate quantization matrix code data. In this embodiment, the encoding table shown in Fig. 11(a) is used for encoding, but the encoding table is not limited to this, and for example, the encoding table shown in Fig. 11(b) may also be used. The quantization matrix code data generated in this manner is output to the subsequent integrated encoding unit 111.

[0043] Returning to FIG. 1, the integration encoding unit 111 encodes header information required for encoding image data and integrates the coded data of the quantization matrix.

[0044] Next, the image data is coded. Image data for one frame input from a terminal 101 is input to a block division unit 102.

[0045] The block division unit 102 divides the input image data into a plurality of basic blocks, and outputs an image in units of basic blocks to the prediction unit 104. In this embodiment, it is assumed that an image in units of basic blocks of 64×64 pixels is output.

[0046] The prediction unit 104 performs prediction processing on the image data in units of basic blocks input from the block division unit 102. Specifically, the prediction unit 104 determines sub-block division for dividing the basic blocks into smaller sub-blocks, and further determines a prediction mode such as intra prediction or inter prediction for each sub-block.

[0047] FIG. 7 shows an example of a subblock division method. The bold-framed reference numeral 700 represents a basic block. For ease of explanation, it is assumed to be a 64x64 pixel configuration, with each rectangle within the bold frame representing a subblock. FIG. 7(b) shows an example of square subblock division in a quadtree, in which a 648x64 pixel basic block is divided into 32x32 pixel subblocks. Meanwhile, FIGS. 7(c) to 7(f) show examples of rectangular subblock division. In FIG. 7(c), the basic block is divided into 32x64 pixel vertically elongated rectangular subblocks, and in FIG. 7(d), it is divided into 64x32 pixel horizontally elongated rectangular subblocks. Furthermore, in FIGS. 7(e) and 7(f), the basic block is divided into rectangular subblocks at a ratio of 1:2:1. In this way, encoding processing is performed using not only square subblocks but also rectangular subblocks other than square ones. Alternatively, a basic block may be further divided into multiple square blocks, and subblock division may be performed based on the resulting square blocks. In other words, the size of the basic block is not limited to 64×64 pixels, and basic blocks of a plurality of sizes may be used.

[0048] In this embodiment, only quadtree partitioning as shown in Figure 7(a) and Figure 7(b), in which basic blocks of 64 x 64 pixels are not divided, is used, but the subblock partitioning method is not limited to this. Tri-tree partitioning as shown in Figures 7(e) and 7(f) or binary tree partitioning as shown in Figures 7(c) and 7(d) may also be used. When subblock partitioning other than that shown in Figures 7(a) and 7(b) is also used, the quantization matrix storage unit 103 generates quantization matrices corresponding to the subblocks to be used. Furthermore, when a new base quantization matrix corresponding to the generated quantization matrix is ​​also generated, the new base quantization matrix is ​​also coded by the quantization matrix coding unit 113.

[0049] The prediction method used by the prediction unit 194 in this embodiment will now be described in more detail. In this embodiment, two types of prediction methods, intra prediction and inter prediction, are used, as an example. Intra prediction generates predicted pixels for the block to be coded using coded pixels spatially surrounding the block to be coded, and also generates intra prediction mode information indicating the intra prediction method used, among intra prediction methods such as horizontal prediction, vertical prediction, and DC prediction. Inter prediction generates predicted pixels for the block to be coded using coded pixels of a frame temporally different from the block to be coded, and also generates motion information indicating reference frames, motion vectors, and the like. As mentioned above, the prediction unit 194 may use a prediction method that combines intra prediction and inter prediction.

[0050] Prediction image data is generated from the determined prediction mode and encoded pixels, and a prediction error is generated from the input image data and the prediction image data, and output to the transform / quantization unit 105. Information such as sub-block division and prediction mode is output to the encoding unit 110 and image reproduction unit 107 as prediction information.

[0051] The transform / quantization unit 105 performs orthogonal transform / quantization on the input prediction error to generate quantized coefficients. First, an orthogonal transform process corresponding to the size of the sub-block is performed to generate orthogonal transform coefficients, and then the orthogonal transform coefficients are quantized using a quantization matrix stored in the quantization matrix storage unit 103 according to the prediction mode to generate quantized coefficients. The orthogonal transform / quantization process will be described in more detail below.

[0052] When the 32×32 sub-block division shown in FIG. 7(b) is selected, the 32×32 prediction error is subjected to an orthogonal transform using a 32×32 orthogonal transform matrix to generate 32×32 orthogonal transform coefficients. Specifically, a 32×32 orthogonal transform matrix, such as a discrete cosine transform (DCT), is multiplied by the 32×32 prediction error to generate a 32×32 matrix of intermediate coefficients. This 32×32 matrix of intermediate coefficients is further multiplied by the transpose of the aforementioned 32×32 orthogonal transform matrix to generate 32×32 orthogonal transform coefficients. The 32×32 orthogonal transform coefficients thus generated are quantized using the 32×32 quantization matrix and quantization parameter shown in FIG. 8(b) to generate 32×32 quantized coefficients. Because there are four 32×32 sub-blocks in a 64×64 basic block, the above process is repeated four times.

[0053] On the other hand, when the 64×64 division state (no division) shown in Figure 7(a) is selected, a 32×64 orthogonal transform matrix generated by thinning out odd-numbered rows (hereinafter referred to as odd rows) in the 64×64 orthogonal transform matrix is ​​used for the 64×64 prediction error. In other words, 32×32 orthogonal transform coefficients are generated by performing an orthogonal transform using the 32×64 orthogonal transform matrix generated by thinning out the odd rows.

[0054] Specifically, first, odd-numbered rows are thinned out from a 64×64 orthogonal transform matrix to generate a 64×32 orthogonal transform matrix. This 64×32 orthogonal transform matrix is ​​then multiplied by the 64×64 prediction error to generate a 64×32 matrix of intermediate coefficients. This 64×32 matrix of intermediate coefficients is then multiplied by a 32×64 transposed matrix obtained by transposing the 64×32 orthogonal transform matrix to generate 32×32 orthogonal transform coefficients. The transform / quantization unit 105 then performs zeroing out by using the generated 32×32 orthogonal transform coefficients as the coefficients in the upper left part (x coordinate range 0 to 31 and y coordinate range 0 to 31) of the 64×64 orthogonal transform coefficients and setting the rest to 0.

[0055] In this embodiment, an orthogonal transform is performed on a 64×64 prediction error using a 64×32 orthogonal transform matrix and a 32×64 transposed matrix obtained by transposing the 64×32 orthogonal transform matrix. Zeroing out is performed by generating 32×32 orthogonal transform coefficients in this manner. This allows 32×32 orthogonal transform coefficients to be generated with less computational complexity than a technique in which some of the 64×64 orthogonal transform coefficients generated by a 64×64 orthogonal transform are forcibly set to zero even if their values ​​are not zero. In other words, the computational complexity of the orthogonal transform can be reduced compared to a case in which an orthogonal transform is performed using a 64×64 orthogonal transform matrix and the orthogonal transform coefficients to be zeroed out are coded as zero regardless of whether they are zero or not. Note that the computational complexity can be reduced by using a method of calculating 32×32 orthogonal transform coefficients from 64×64 prediction errors using orthogonal transform coefficients, but the zeroing out method is not limited to this method and various other methods can be used.

[0056] Furthermore, when zeroing out is performed, information indicating that the orthogonal transform coefficients in the range subject to zeroing out are 0 may be coded, or information (such as a flag) indicating that zeroing out has been performed may simply be coded. By decoding this information, the image decoding device can decode each block by regarding the range subject to zeroing out as 0.

[0057] Next, the transform / quantization unit 105 quantizes the 32×32 orthogonal transform coefficients generated in this manner using the 32×32 quantization matrix shown in Figure 8(c) and the quantization parameter, to generate 32×32 quantized coefficients.

[0058] In this embodiment, the quantization matrix of Figure 8(b) is used for 32x32 orthogonal transform coefficients corresponding to 32x32 sub-blocks, and the quantization matrix of Figure 8(c) is used for 32x32 orthogonal transform coefficients corresponding to 64x64 sub-blocks. In other words, the quantization matrix of Figure 8(b) is used for 32x32 orthogonal transform coefficients that have not been zeroed out, and the quantization matrix of Figure 8(c) is used for 32x32 orthogonal transform coefficients that have been zeroed out corresponding to 64x64 sub-blocks. However, the quantization matrix used is not limited to this. The generated quantization coefficients are output to the encoding unit 110 and the inverse quantization and inverse transform unit 106.

[0059] The inverse quantization and inverse transform unit 106 inverse quantizes the input quantized coefficients using the quantization matrix stored in the quantization matrix storage unit 103 and the quantization parameters to reproduce orthogonal transform coefficients. The inverse quantization and inverse transform unit 106 then performs inverse orthogonal transform on the reproduced orthogonal transform coefficients to reproduce prediction errors. As with the transform and quantization unit 105, the inverse quantization process uses a quantization matrix corresponding to the size of the sub-block to be coded. The inverse quantization and inverse orthogonal transform process performed by the inverse quantization and inverse transform unit 106 will be described in more detail below.

[0060] When the 32×32 sub-block division shown in FIG. 7(b) is selected, the inverse quantization and inverse transform unit 106 inverse quantizes the 32×32 quantized coefficients generated by the transform and quantization unit 105 using the quantization matrix shown in FIG. 8(b) to reproduce 32×32 orthogonal transform coefficients. The inverse quantization and inverse transform unit 106 then multiplies the aforementioned 32×32 transposed matrix by the 32×32 orthogonal transform to calculate intermediate coefficients in a 32×32 matrix. The inverse quantization and inverse transform unit 106 then multiplies this 32×32 intermediate coefficient matrix by the aforementioned 32×32 orthogonal transform matrix to reproduce 32×32 prediction errors. Similar processing is performed on each 32×32 sub-block. On the other hand, when no division is selected as shown in FIG. 7(a), the 32×32 quantized coefficients generated by the transform / quantization unit 105 are inversely quantized using the quantization matrix shown in FIG. 8(c) to reproduce 32×32 orthogonal transform coefficients. Then, the aforementioned 32×64 transposed matrix is ​​multiplied by the 32×32 orthogonal transform to calculate 32×64 matrix-shaped intermediate coefficients. This 32×64 matrix-shaped intermediate coefficients is multiplied by the aforementioned 64×32 orthogonal transform matrix to reproduce 64×64 prediction errors. In this embodiment, the same quantization matrix as used by the transform / quantization unit 105 is used to perform the inverse quantization process, depending on the size of the sub-block. The reproduced prediction errors are output to the image reproduction unit 107.

[0061] The image reproduction unit 107 reproduces a predicted image by appropriately referencing data necessary for reproducing the predicted image stored in the frame memory 108 based on the prediction information input from the prediction unit 104. Then, image data is reproduced from the reproduced predicted image and the reproduced prediction error input from the inverse quantization and inverse transform unit 106, and is input to the frame memory 108 for storage.

[0062] The in-loop filter unit 109 reads out the reconstructed image from the frame memory 108 and performs in-loop filtering such as deblocking filtering on the image, and then inputs the filtered image back into the frame memory 108 and stores it again.

[0063] The encoding unit 110 entropy-encodes, on a block-by-block basis, the quantization coefficients generated by the transform / quantization unit 105 and the prediction information input from the prediction unit 104, to generate coded data. There is no particular specification as to the entropy coding method, but Golomb coding, arithmetic coding, Huffman coding, etc. can be used. The generated coded data is output to the integrated coding unit 111.

[0064] The integrated encoding unit 111 multiplexes the encoded data of the header and the encoded data input from the encoding unit 110 to form a bit stream. Finally, the bit stream is output from a terminal 112 to the outside.

[0065] FIG. 6(a) shows an example of a bitstream output in the first embodiment. The sequence header contains coded data of the base quantization matrix and is composed of the coded results of each element. However, the location where the coded data of the base quantization matrix is ​​coded is not limited to this, and it may of course be coded in the picture header section or another header section. Furthermore, when changing the quantization matrix within a sequence, it is also possible to update the base quantization matrix by newly coding it. In this case, all quantization matrices may be rewritten, or it is also possible to change only a portion of the quantization matrix by specifying the size of the sub-block of the quantization matrix corresponding to the quantization matrix to be rewritten.

[0066] FIG. 3 is a flowchart showing the encoding process in the image encoding device according to the first embodiment.

[0067] First, prior to encoding an image, in step S301, the quantization matrix storage unit 103 generates and stores a two-dimensional quantization matrix. In this embodiment, the base quantization matrix shown in Fig. 8(a) and the quantization matrices shown in Fig. 8(b) and (c) generated from the base quantization matrix are generated and stored.

[0068] In step S302, the quantization matrix encoding unit 113 scans the base quantization matrix used to generate the quantization matrix in step S301, calculates the difference between each element that comes before and after in the scanning order, and generates a one-dimensional difference matrix. In this embodiment, the base quantization matrix shown in Fig. 8(a) is scanned using the scanning method of Fig. 9, and the difference matrix shown in Fig. 10 is generated. The quantization matrix encoding unit 113 further encodes the generated difference matrix to generate quantization matrix code data.

[0069] In step S303, the integral encoding unit 111 encodes and outputs the generated quantization matrix code data as well as header information required for encoding the image data.

[0070] In step S304, the block dividing unit 102 divides the input image in units of frames into basic blocks of 64×64 pixels.

[0071] In step S305, the prediction unit 104 performs prediction processing on the basic block-based image data generated in step S304 using the prediction method described above, generating prediction information such as sub-block division information and a prediction mode, as well as predicted image data. In this embodiment, two types of sub-block sizes are used: the 32x32 pixel sub-block division shown in Figure 7(b) and the 64x64 pixel sub-blocks shown in Figure 7(a). Furthermore, a prediction error is calculated from the input image data and the predicted image data.

[0072] In step S306, the transform / quantization unit 105 performs an orthogonal transform on the prediction errors calculated in step S305 to generate orthogonal transform coefficients. The transform / quantization unit 105 then performs quantization using the quantization matrix and quantization parameters generated and stored in step S301 to generate quantized coefficients. Specifically, the prediction errors of the 32×32 pixel sub-blocks in FIG. 7(b) are multiplied by a 32×32 orthogonal transform matrix and its transpose matrix to generate 32×32 orthogonal transform coefficients. Meanwhile, the prediction errors of the 64×64 pixel sub-blocks in FIG. 7(a) are multiplied by a 64×32 orthogonal transform matrix and its transpose matrix to generate 32×32 orthogonal transform coefficients. In this embodiment, the quantization matrix of Figure 8(b) is used for the orthogonal transform coefficients of the 32x32 sub-blocks of Figure 7(b), and the quantization matrix of Figure 8(c) is used for the orthogonal transform coefficients corresponding to the 64x64 sub-blocks of Figure 7(a), to quantize the 32x32 orthogonal transform coefficients.

[0073] In step S307, the inverse quantization / inverse transform unit 106 inverse quantizes the quantized coefficients generated in step S306 using the quantization matrix and quantization parameters generated and stored in step S301 to regenerate orthogonal transform coefficients. Furthermore, the orthogonal transform coefficients are subjected to inverse orthogonal transform to regenerate prediction errors. In this step, the same quantization matrix as used in step S306 is used for inverse quantization. Specifically, the 32×32 quantized coefficients corresponding to the 32×32 pixel sub-blocks in FIG. 7(b) are inverse quantized using the quantization matrix in FIG. 8(b) to regenerate 32×32 orthogonal transform coefficients. These 32×32 orthogonal transform coefficients are then multiplied by the 32×32 orthogonal transform matrix and its transpose matrix to regenerate 32×32 pixel prediction errors. On the other hand, the 32x32 quantized coefficients corresponding to the 64x64 pixel sub-blocks in Figure 7(a) are inverse quantized using the quantization matrix in Figure 8(c) to regenerate 32x32 orthogonal transform coefficients. These 32x32 orthogonal transform coefficients are then multiplied by a 64x32 orthogonal transform matrix and its transpose matrix to regenerate 64x64 pixel prediction errors.

[0074] In step S308, the image reproduction unit 107 reproduces a predicted image based on the prediction information generated in step S305, and further reproduces image data from the reproduced predicted image and the prediction error generated in step S307.

[0075] In step S309, the encoding unit 110 encodes the prediction information generated in step S305 and the quantized coefficients generated in step S306 to generate coded data, and also generates a bitstream including other coded data.

[0076] In step S310, the image encoding device determines whether encoding of all basic blocks in the frame has been completed. If so, the process proceeds to step S311; if not, the process returns to step S304 for the next basic block.

[0077] In step S311, the in-loop filter unit 109 performs in-loop filtering on the image data reproduced in step S308 to generate a filtered image, and then the process ends.

[0078] The above configuration and operation enable the control of quantization for each frequency component while reducing the amount of calculation, thereby improving subjective image quality. In particular, in step S305, by reducing the number of orthogonal transform coefficients and performing quantization processing using a quantization matrix corresponding to the reduced orthogonal transform coefficients, the control of quantization for each frequency component while reducing the amount of calculation can be improved, thereby improving subjective image quality. Furthermore, when the number of orthogonal transform coefficients is reduced and only the low-frequency portion is quantized and encoded, optimal quantization control for the low-frequency portion can be achieved by using a quantization matrix that expands only the low-frequency portion of the base quantization matrix, as shown in FIG. 8(c). The low-frequency portion here refers to the range of x coordinates from 0 to 3 and y coordinates from 0 to 3 in the example of FIG. 8(c).

[0079] In this embodiment, in order to reduce the amount of code, only the base quantization matrix of FIG. 8(a), which is commonly used to generate the quantization matrices of FIG. 8(b) and FIG. 8(c), is coded. However, the quantization matrices of FIG. 8(b) and FIG. 8(c) themselves may also be coded. In this case, a unique value can be set for each frequency component of each quantization matrix, enabling more precise quantization control for each frequency component. Alternatively, a separate base quantization matrix can be set for each of FIG. 8(b) and FIG. 8(c), and each base quantization matrix can be coded. In this case, different quantization controls can be applied to the 32×32 orthogonal transform coefficients and the 64×64 orthogonal transform coefficients, enabling more precise control of subjective image quality. Furthermore, in this case, the quantization matrix corresponding to the 64×64 orthogonal transform coefficients may be enlarged by a factor of four, instead of enlarging the upper left 4×4 portion of the 8×8 base quantization matrix by a factor of eight. In this way, even for 64×64 orthogonal transform coefficients, more precise quantization control can be achieved.

[0080] Furthermore, although this embodiment is configured such that a quantization matrix for a 64×64 sub-block using zero-out is uniquely determined, a configuration in which an identifier can be introduced to enable selection may also be adopted. For example, FIG. 6(b) illustrates selective quantization matrix coding for a 64×64 sub-block using zero-out by newly introducing a quantization matrix coding method information code. For example, when the quantization matrix coding method information code indicates 0, an independent quantization matrix shown in FIG. 8(c) is used for the orthogonal transform coefficients corresponding to a 64×64 pixel sub-block using zero-out. Furthermore, when the coding method information code indicates 1, a quantization matrix shown in FIG. 8(b) for a normal sub-block that is not zeroed out is used for a 64×64 pixel sub-block using zero-out. On the other hand, when the coding method information code indicates 2, all elements of the quantization matrix used for a 64×64 pixel sub-block using zero-out are coded, instead of the 8×8 base quantization matrix. This makes it possible to selectively achieve quantization matrix code amount reduction and unique quantization control for a sub-block using zero-out.

[0081] Furthermore, although the sub-blocks processed using zero-out are limited to 64x64 in this embodiment, the sub-blocks processed using zero-out are not limited to this. For example, among the orthogonal transform coefficients corresponding to the 32x64 and 64x32 sub-blocks shown in Figures 7(c) and 7(b), the 32x32 orthogonal transform coefficients in the lower half and right half may be forcibly set to 0. In this case, only the 32x32 orthogonal transform coefficients in the upper half and left half are subject to quantization and encoding, and the quantization process is performed on the 32x32 orthogonal transform coefficients in the upper half and left half using a quantization matrix different from that in Figure 8(b).

[0082] Furthermore, the value of the quantization matrix corresponding to the DC coefficient located at the upper left corner of the generated orthogonal transform coefficients, which is considered to have the greatest impact on image quality, may be set and coded separately from the values ​​of each element of the 8×8 base matrix. Figures 12(b) and 12(c) show an example in which the value of the element located at the upper left corner, which corresponds to the DC component, is changed compared to Figures 8(b) and 8(c). In this case, the quantization matrices shown in Figures 12(b) and 12(c) can be set by separately coding information indicating "2" located in the DC portion in addition to the information on the base quantization matrix of Figure 8(a). This allows for more precise quantization control of the DC component of the orthogonal transform coefficients, which have the greatest impact on image quality.

[0083] <Embodiment 2> 2 is a block diagram showing the configuration of an image decoding device according to a second embodiment of the present invention. In this embodiment, an image decoding device that decodes coded data generated in the first embodiment will be described as an example.

[0084] 201 is a terminal to which an encoded bitstream is input.

[0085] Reference numeral 202 denotes a separate decoding unit, which separates the bitstream into coded data relating to information on decoding processing and coefficients, and decodes the coded data present in the header part of the bitstream. In this embodiment, it separates the quantization matrix code and outputs it to the subsequent stage. The separate decoding unit 202 performs the reverse operation of the integrated coding unit 111 in FIG. 1.

[0086] 209 is a quantization matrix decoding unit that decodes the quantization matrix code from the bit stream to reproduce the base quantization matrix, and further executes processing to generate each quantization matrix from the base quantization matrix.

[0087] A decoding unit 203 decodes the coded data output from the demultiplexing decoding unit 202 and reproduces (derives) the quantization coefficients and prediction information.

[0088] An inverse quantization and inverse transform unit 204, like the inverse quantization and inverse transform unit 106 in Fig. 1, performs inverse quantization on the quantized coefficients using the reproduced quantization matrix and quantization parameters to obtain orthogonal transform coefficients, and then performs inverse orthogonal transform to reproduce prediction errors. Note that information for deriving the quantization parameters is also decoded from the bitstream by the decoding unit 203. Furthermore, the function for performing inverse quantization and the function for performing inverse quantization may be configured separately.

[0089] A frame memory 206 stores image data of reproduced pictures.

[0090] An image reproduction unit 205 generates predicted image data based on the input prediction information by appropriately referencing the frame memory 206. Then, the image reproduction unit 205 generates and outputs reproduced image data from this predicted image data and the prediction error reproduced by the inverse quantization and inverse transform unit 204.

[0091] 207 is an in-loop filter unit. Similar to 109 in Fig. 1, it performs in-loop filtering such as deblocking filtering on the reconstructed image and outputs the filtered image.

[0092] Reference numeral 208 denotes a terminal for outputting the reproduced image data to the outside.

[0093] The image decoding operation in the image decoding device will be described below. In this embodiment, the bit stream generated in the first embodiment is input in frame units (picture units).

[0094] In FIG. 2, a bitstream for one frame input from a terminal 201 is input to a separate decoding unit 202. The separate decoding unit 202 separates the bitstream into coded data related to information on decoding processing and coefficients, and decodes the coded data present in the header part of the bitstream. More specifically, quantization matrix coded data is reproduced. In this embodiment, first, quantization matrix coded data is extracted from the sequence header of the bitstream shown in FIG. 6(a) and output to a quantization matrix decoding unit 209. In this embodiment, quantization matrix coded data corresponding to the base quantization matrix shown in FIG. 8(a) is extracted and output. Next, coded data for each basic block of picture data is reproduced and output to a decoding unit 203.

[0095] The quantization matrix decoding unit 209 first decodes the input quantization matrix code data to reconstruct the one-dimensional difference matrix shown in FIG. 10. In this embodiment, as in the first embodiment, decoding is performed using the coding table shown in FIG. 11(a). However, the coding table is not limited to this, and other coding tables may be used as long as they are the same as those in the first embodiment. Furthermore, the quantization matrix decoding unit 209 reconstructs a two-dimensional quantization matrix from the reconstructed one-dimensional difference matrix. Here, the operation is the reverse of the operation of the quantization matrix encoding unit 113 in the first embodiment. That is, in this embodiment, the difference matrix shown in FIG. 10 is reconstructed and stored from the base quantization matrix shown in FIG. 8(a) using the scanning method shown in FIG. 9. Specifically, the quantization matrix decoding unit 209 reconstructs each element of the quantization matrix by sequentially adding each difference value in the difference matrix starting from the initial value described above. Then, the quantization matrix decoding unit 209 reproduces the two-dimensional quantization matrix by sequentially associating each of the reproduced one-dimensional elements with each of the elements of the two-dimensional quantization matrix according to the scanning method shown in FIG.

[0096] Furthermore, the quantization matrix decoding unit 209 enlarges this reconstructed base quantization matrix in the same way as in the first embodiment, to generate two types of 32 × 32 quantization matrices shown in Figure 8(b) and Figure 8(c). The quantization matrix in Figure 8(b) is a 32 × 32 quantization matrix obtained by enlarging each element of the 8 × 8 base quantization matrix in Figure 8(a) by four times in the vertical and horizontal directions.

[0097] On the other hand, the quantization matrix in Figure 8(c) is a 32x32 quantization matrix obtained by enlarging each element of the 4x4 portion in the upper left corner of the base quantization matrix in Figure 8(a) by repeating it eight times vertically and horizontally. However, the generated quantization matrix is ​​not limited to this, and if the size of the quantized coefficients to be dequantized later is other than 32x32, a quantization matrix corresponding to the size of the quantized coefficients to be dequantized, such as 16x16, 8x8, or 4x4, may be generated. These generated quantization matrices are retained and used in the dequantization process later.

[0098] The decoding unit 203 decodes the coded data from the bitstream and reproduces the quantization coefficients and prediction information. The size of the sub-block to be decoded is determined based on the decoded prediction information, and the reproduced quantization coefficients are output to the inverse quantization and inverse transform unit 204, and the reproduced prediction information is output to the image reproduction unit 205. In this embodiment, regardless of the size of the sub-block to be decoded, i.e., whether it is 64×64 as in FIG. 7(a) or 32×32 as in FIG. 7(b), 32×32 quantization coefficients are reproduced for each sub-block.

[0099] The inverse quantization and inverse transform unit 204 performs inverse quantization on the input quantized coefficients using the quantization matrix reproduced by the quantization matrix decoding unit 209 and the quantization parameters to generate orthogonal transform coefficients, and then performs inverse orthogonal transform to reproduce prediction errors. More specific inverse quantization and inverse orthogonal transform processing will be described below.

[0100] When the 32×32 sub-block division shown in FIG. 7(b) is selected, the 32×3 quantized coefficients reproduced by the decoding unit 203 are inverse quantized using the quantization matrix shown in FIG. 8(b) to reproduce 32×32 orthogonal transform coefficients. Then, the aforementioned 32×32 transposed matrix is ​​multiplied by the 32×32 orthogonal transform to calculate intermediate coefficients in a 32×32 matrix. This 32×32 intermediate coefficient matrix is ​​multiplied by the aforementioned 32×32 orthogonal transform matrix to reproduce 32×32 prediction errors. Similar processing is performed for each 32×32 sub-block.

[0101] On the other hand, when no division is selected as shown in Figure 7(a), the 32 x 32 quantized coefficients reproduced by the decoding unit 203 are inverse quantized using the quantization matrix of Figure 8(c) to reproduce 32 x 32 orthogonal transform coefficients.Then, the aforementioned 32 x 64 transposed matrix is ​​multiplied by the 32 x 32 orthogonal transform to calculate a 32 x 64 matrix of intermediate coefficients.This 32 x 64 matrix of intermediate coefficients is multiplied by the aforementioned 64 x 32 orthogonal transform matrix to reproduce 64 x 64 prediction errors.

[0102] The reconstructed prediction error is output to the image reconstruction unit 205. In this embodiment, the quantization matrix to be used in the inverse quantization process is determined according to the size of the sub-block to be decoded, which is determined by the prediction information reconstructed by the decoding unit 203. That is, the quantization matrix of FIG. 8(b) is used in the inverse quantization process for each 32×32 sub-block of FIG. 7(b), and the quantization matrix of FIG. 8(c) is used for the 64×64 sub-block of FIG. 7(a). However, the quantization matrix to be used is not limited to this, and it may be the same as the quantization matrix used in the transform / quantization unit 105 and the inverse quantization / inverse transform unit 106 of the first embodiment.

[0103] The image reproduction unit 205 appropriately refers to the frame memory 206 based on the prediction information input from the decoding unit 203, acquires data necessary for reproducing the predicted image, and reproduces the predicted image. In this embodiment, as with the prediction unit 104 in the first embodiment, two types of prediction methods, intra prediction and inter prediction, are used. Also, as described above, a prediction method that combines intra prediction and inter prediction may be used. Also, as with the first embodiment, the prediction process is performed in sub-block units.

[0104] The specific prediction process is the same as that of the prediction unit 104 in the first embodiment, and therefore a description thereof will be omitted. The image reproduction unit 205 reproduces image data from the predicted image generated by the prediction process and the prediction error input from the inverse quantization and inverse transform unit 204. Specifically, the image reproduction unit 205 reproduces image data by adding the predicted image and the prediction error. The reproduced image data is stored in the frame memory 206 as appropriate. The stored image data is referenced as appropriate when predicting other sub-blocks.

[0105] 1, the in-loop filter unit 207 reads the reconstructed image from the frame memory 206 and performs in-loop filtering such as deblocking filtering. The filtered image is then input back to the frame memory 206.

[0106] The reproduced image stored in the frame memory 206 is finally output to the outside from a terminal 208. The reproduced image is output to, for example, an external display device.

[0107] FIG. 4 is a flowchart showing the image decoding process in the image decoding device according to the second embodiment.

[0108] First, in step S401, the demultiplexing / decoding unit 202 demultiplexes the bitstream into coded data related to information on decoding processing and coefficients, and decodes the coded data of the header portion. More specifically, it reproduces quantization matrix coded data.

[0109] In step S402, the quantization matrix decoding unit 209 first decodes the quantization matrix code data reproduced in step S401 to reproduce the one-dimensional difference matrix shown in Fig. 10. Next, the quantization matrix decoding unit 209 reproduces a two-dimensional base quantization matrix from the reproduced one-dimensional difference matrix. Furthermore, the quantization matrix decoding unit 209 enlarges the reproduced two-dimensional base quantization matrix to generate a quantization matrix.

[0110] That is, in this embodiment, the quantization matrix decoding unit 209 reproduces the base quantization matrix shown in Fig. 8(a) using the difference matrix shown in Fig. 10 and the scanning method shown in Fig. 9. Furthermore, the quantization matrix decoding unit 209 expands the reproduced base quantization matrix to generate and store the quantization matrices shown in Fig. 8(b) and Fig. 8(c).

[0111] In step S403, the decoding unit 203 decodes the coded data separated in step S401 to reproduce the quantization coefficients and prediction information. Furthermore, the size of the sub-blocks to be decoded is determined based on the decoded prediction information. In this embodiment, regardless of the size of the sub-blocks to be decoded, i.e., whether it is 64x64 in FIG. 7(a) or 32x32 in FIG. 7(b), 32x32 quantization coefficients are reproduced for each sub-block.

[0112] In step S404, the inverse quantization / inverse transform unit 204 performs inverse quantization on the quantized coefficients using the quantization matrix reconstructed in step S402 to obtain orthogonal transform coefficients, and then performs inverse orthogonal transform to reconstruct prediction errors. In this embodiment, the quantization matrix to be used in the inverse quantization process is determined according to the size of the sub-block to be decoded, which is determined by the prediction information reconstructed in step S403. That is, the quantization matrix of FIG. 8(b) is used in the inverse quantization process for each 32×32 sub-block in FIG. 7(b), and the quantization matrix of FIG. 8(c) is used for each 64×64 sub-block in FIG. 7(a). However, the quantization matrix to be used is not limited to this, and may be the same as the quantization matrix used in steps S306 and S307 in the first embodiment.

[0113] In step S405, the image reproduction unit 205 reproduces a predicted image from the prediction information generated in step S403. In this embodiment, two types of prediction methods, intra prediction and inter prediction, are used, as in step S305 in embodiment 1. Furthermore, image data is reproduced from the reproduced predicted image and the prediction error generated in step S404.

[0114] In step S406, the image decoding device determines whether or not decoding of all basic blocks in the frame has been completed. If so, the process proceeds to step S407; if not, the process returns to step S403 for the next basic block.

[0115] In step S407, the in-loop filter unit 207 performs in-loop filtering on the image data reproduced in step S405 to generate a filtered image, and then the process ends.

[0116] With the above configuration and operation, it is possible to decode a bitstream in which quantization is controlled for each frequency component using a quantization matrix, and improve the subjective image quality, even for sub-blocks in which only low-frequency orthogonal transform coefficients are quantized and coded, as generated in embodiment 1. Furthermore, for sub-blocks in which only low-frequency orthogonal transform coefficients are quantized and coded, a quantization matrix in which only the low-frequency part of the base quantization matrix as shown in Figure 8(c) is expanded is used, and a bitstream in which optimal quantization control is applied to the low-frequency part can be decoded.

[0117] In this embodiment, in order to reduce the amount of code, only the base quantization matrix of Fig. 8(a) that is commonly used to generate the quantization matrices of Fig. 8(b) and (c) is decoded, but it is also possible to decode the quantization matrices of Fig. 8(b) and (c) themselves. In this case, a unique value can be set for each frequency component of each quantization matrix, making it possible to decode a bitstream that achieves more detailed quantization control for each frequency component.

[0118] It is also possible to set separate base quantization matrices for each of Figures 8(b) and 8(c) and encode each base quantization matrix. In this case, different quantization controls are performed on the 32x32 orthogonal transform coefficients and the 64x64 orthogonal transform coefficients, allowing for the decoding of bitstreams that achieve more precise control of subjective image quality. Furthermore, in this case, the quantization matrix corresponding to the 64x64 orthogonal transform coefficients may be enlarged by a factor of four, instead of enlarging the upper left 4x4 portion of the 8x8 base quantization matrix by a factor of eight. In this way, more precise quantization control can be achieved even for the 64x64 orthogonal transform coefficients.

[0119] Furthermore, although this embodiment is configured such that a quantization matrix for a 64×64 sub-block using zero-out is uniquely determined, a configuration in which an identifier is introduced to enable selection may also be adopted. For example, FIG. 6(b) shows a configuration in which a quantization matrix encoding method information code is newly introduced to selectively encode the quantization matrix for a 64×64 sub-block using zero-out. For example, when the quantization matrix encoding method information code indicates 0, an independent quantization matrix shown in FIG. 8(c) is used for the quantization coefficients corresponding to the 64×64 sub-block using zero-out. When the encoding method information code indicates 1, a quantization matrix shown in FIG. 8(b) for a normal sub-block that is not zeroed out is used for the 64×64 sub-block using zero-out. On the other hand, when the encoding method information code indicates 2, all elements of the quantization matrix used for the 64×64 sub-block using zero-out are encoded instead of the 8×8 base quantization matrix. This makes it possible to decode a bitstream that selectively achieves quantization matrix code amount reduction and unique quantization control for sub-blocks using zero-out.

[0120] Furthermore, although the sub-blocks processed using zero-out are limited to 64x64 in this embodiment, the sub-blocks processed using zero-out are not limited to this. For example, among the orthogonal transform coefficients corresponding to the 32x64 and 64x32 sub-blocks shown in Figures 7(c) and 7(b), the 32x32 orthogonal transform coefficients in the lower and right halves may not be decoded, and only the quantized coefficients in the upper and left halves may be decoded. In this case, only the 32x32 orthogonal transform coefficients in the upper and left halves are subject to decoding and inverse quantization, and the 32x32 orthogonal transform coefficients in the upper and left halves are quantized using a quantization matrix different from that in Figure 8(b).

[0121] Furthermore, a configuration may be adopted in which the value of the quantization matrix corresponding to the DC coefficient located at the upper left corner of the generated orthogonal transform coefficients, which is considered to have the greatest impact on image quality, is decoded and set separately from the values ​​of each element of the 8 × 8 base matrix. Figures 12(b) and 12(c) show an example in which the value of the element located at the upper left corner, which corresponds to the DC component, is changed compared to Figures 8(b) and 8(c). In this case, the quantization matrices shown in Figures 12(b) and 12(c) can be set by separately decoding information indicating "2" located in the DC portion in addition to information on the base quantization matrix of Figure 8(a). This makes it possible to decode a bitstream in which more precise quantization control has been applied to the DC components of the orthogonal transform coefficients, which have the greatest impact on image quality.

[0122] <Embodiment 3> In the above embodiment, each processing unit shown in Figures 1 and 2 has been described as being configured by hardware. However, the processing performed by each processing unit shown in these figures may be configured by a computer program.

[0123] FIG. 5 is a block diagram showing an example of the hardware configuration of a computer that can be applied to the image display device according to each of the above embodiments.

[0124] The CPU 501 controls the entire computer using computer programs and data stored in the RAM 502 and the ROM 503, and also executes the processes described above as being performed by the image processing device according to each of the above embodiments. That is, the CPU 501 functions as each processing unit shown in FIGS. 1 and 2.

[0125] The RAM 502 has an area for temporarily storing computer programs and data loaded from an external storage device 506, data acquired from the outside via an I / F (interface) 507, etc. The RAM 502 also has a work area used when the CPU 501 executes various processes. That is, the RAM 502 can be allocated as a frame memory, for example, or can provide various other areas as needed.

[0126] The ROM 503 stores setting data for the computer, a boot program, etc. The operation unit 504 is made up of a keyboard, a mouse, etc., and can be operated by a user of the computer to input various instructions to the CPU 501. The display unit 505 displays the results of processing by the CPU 501. The display unit 505 is made up of, for example, a liquid crystal display.

[0127] The external storage device 506 is a large-capacity information storage device, such as a hard disk drive. The external storage device 506 stores an operating system (OS) and computer programs for causing the CPU 501 to implement the functions of the various units shown in Figures 1 and 2. Furthermore, the external storage device 506 may also store image data to be processed.

[0128] Computer programs and data stored in the external storage device 506 are loaded into the RAM 502 as appropriate under the control of the CPU 501 and become the subject of processing by the CPU 501. An I / F 507 can be connected to a network such as a LAN or the Internet, or other devices such as a projector or display device, and the computer can obtain and send various information via this I / F 507. 508 is a bus that connects the above-mentioned components.

[0129] The operation of the above-described configuration is controlled mainly by the CPU 501 as explained in the flow chart above.

[0130] (Other Examples) Each embodiment can also be achieved by providing a storage medium containing computer program code that realizes the above-described functions to a system, and having the system read and execute the computer program code. In this case, the computer program code read from the storage medium itself realizes the functions of the above-described embodiments, and the storage medium containing the computer program code constitutes the present invention. This also includes cases where an operating system (OS) running on a computer performs some or all of the actual processing based on the instructions of the program code, thereby realizing the above-described functions.

[0131] Furthermore, the present invention may be realized in the following form: That is, computer program code read from a storage medium is written to memory in a function expansion card inserted into a computer or in a function expansion unit connected to the computer, and the CPU in the function expansion card or function expansion unit then performs some or all of the actual processing based on the instructions of the computer program code, thereby realizing the above-mentioned functions.

[0132] When the present invention is applied to the storage medium, the storage medium stores computer program code corresponding to the flowcharts described above. [Explanation of symbols]

[0133] 101, 112, 201, 208 terminals 102 Block division section 103 Quantization matrix storage unit 104 Prediction Department 105 Transformation and Quantization Unit 106, 204 Inverse quantization and inverse transform unit 107, 205 Image playback unit 108, 206 frame memory 109, 207 In-loop filter section 110 Encoding section 111 Integrated coding unit 113 Quantization matrix coding unit 202 Separate Decoding Unit 203 Decoding Unit 209 Quantization matrix decoding unit

Claims

1. An image decoding device capable of decoding an image from a bitstream using a plurality of blocks including a first block of P×Q pixels (P and Q are integers) and a second block of N×M pixels (N is an integer satisfying N<P and M is an integer satisfying M<Q), decoding means for decoding data corresponding to a first set of quantized transform coefficients corresponding to the first block and data corresponding to a second set of quantized transform coefficients corresponding to the second block from the bitstream; inverse quantization means for deriving a first set of transform coefficients representing frequency components from the first set of quantized transform coefficients using a first quantization matrix having N×M elements, and for deriving a second set of transform coefficients representing frequency components from the second set of quantized transform coefficients using a second quantization matrix having N×M elements; inverse transform means for performing an inverse transform process on the first set of transform coefficients to derive a first set of prediction errors corresponding to the first block, and for performing an inverse transform process on the second set of transform coefficients to derive a second set of prediction errors corresponding to the second block; a reproduction unit that reproduces at least image data based on the first group of prediction errors and predicted image data; and The predicted image data can be derived using a prediction method that combines intra prediction and inter prediction, When the block to be decoded is the first block, the inverse transform means multiplies the first transform coefficient group, which is N×M transform coefficients, by an M×Q matrix to derive N×Q intermediate values, and further multiplies the N×Q intermediate values ​​by a P×N matrix to derive the first prediction error group, which is P×Q prediction errors, from the first transform coefficient group; the first quantization matrix having the N×M elements is a quantization matrix that includes some elements of a third quantization matrix having R×S elements (R is an integer satisfying R≦N and S is an integer satisfying S≦M) but does not include other elements of the third quantization matrix, the second quantization matrix having the N×M elements is a quantization matrix that includes all elements in a fourth quantization matrix having R×S elements; the third quantization matrix is ​​different from the fourth quantization matrix; the first quantization matrix is ​​a quantization matrix configured with the part of elements in the third quantization matrix except for elements corresponding to DC components, the second quantization matrix is ​​a quantization matrix that is configured from all of the elements in the fourth quantization matrix except for an element corresponding to a DC component, The decoding means further decodes information indicating the value of the element corresponding to the DC component.

1. An image decoding device comprising:

2. 1. An image decoding method capable of decoding an image from a bitstream using a plurality of blocks including a first block of P×Q pixels (P and Q are integers) and a second block of N×M pixels (N is an integer satisfying N<P and M is an integer satisfying M<Q), a decoding step of decoding data corresponding to a first set of quantized transform coefficients corresponding to the first block and data corresponding to a second set of quantized transform coefficients corresponding to the second block from the bitstream; an inverse quantization step of deriving a first set of transform coefficients representing frequency components from the first set of quantized transform coefficients using a first quantization matrix having N×M elements, and deriving a second set of transform coefficients representing frequency components from the second set of quantized transform coefficients using a second quantization matrix having N×M elements; an inverse transform process for deriving a first set of prediction errors corresponding to the first block by performing an inverse transform process on the first set of transform coefficients, and for deriving a second set of prediction errors corresponding to the second block by performing an inverse transform process on the second set of transform coefficients; a reproduction step of reproducing at least image data based on the first group of prediction errors and predicted image data; and The predicted image data can be derived using a prediction method that combines intra prediction and inter prediction, When the block to be decoded is the first block, in the inverse transform step, the first transform coefficient group, which is N×M transform coefficients, is multiplied by an M×Q matrix to derive N×Q intermediate values, and further, the first prediction error group, which is P×Q prediction errors, is derive from the first transform coefficient group by multiplying the N×Q intermediate values ​​by a P×N matrix; the first quantization matrix having the N×M elements is a quantization matrix that includes some elements of a third quantization matrix having R×S elements (R is an integer satisfying R≦N and S is an integer satisfying S≦M) but does not include other elements of the third quantization matrix, the second quantization matrix having the N×M elements is a quantization matrix that includes all elements in a fourth quantization matrix having R×S elements; the third quantization matrix is ​​different from the fourth quantization matrix; the first quantization matrix is ​​a quantization matrix configured with the part of elements in the third quantization matrix except for elements corresponding to DC components, the second quantization matrix is ​​a quantization matrix that is configured from all of the elements in the fourth quantization matrix except for an element corresponding to a DC component, In the decoding step, information indicating the value of the element corresponding to the DC component is further decoded.

1. An image decoding method comprising:

3. The first and second blocks are square blocks.

3. The image decoding method according to claim 2.

4. The P and the Q are 64, and the N and the M are 32.

3. The image decoding method according to claim 2.

5. The P and Q are 128, and the N and M are 32.

3. The image decoding method according to claim 2.

6. the first set of transform coefficients is N×M transform coefficients; The second set of transform coefficients is N×M transform coefficients.

3. The image decoding method according to claim 2.

7. The first and second blocks are non-square blocks.

3. The image decoding method according to claim 2.

8. the first set of prediction errors is P×Q prediction errors; The second group of prediction errors is N×M prediction errors.

3. The image decoding method according to claim 2.

9. 1. An image encoding device capable of encoding an image using a plurality of blocks including a first block of P×Q pixels (P and Q are integers) and a second block of N×M pixels (N is an integer satisfying N<P and M is an integer satisfying M<Q), a transform means for deriving a first set of transform coefficients by performing a transform process on a first set of prediction errors corresponding to the first block, and for deriving a second set of transform coefficients by performing a transform process on a second set of prediction errors corresponding to the second block; a quantization means for quantizing the first set of transform coefficients using a first quantization matrix having N×M elements to derive a first set of quantized transform coefficients, and for quantizing the second set of transform coefficients using a second quantization matrix having N×M elements to derive a second set of quantized transform coefficients; encoding means for encoding data corresponding to the first set of quantized transform coefficients corresponding to the first block and data corresponding to the second set of quantized transform coefficients corresponding to the second block; and at least the first set of prediction errors can be derived using a prediction method that combines intra-prediction and inter-prediction, When the block to be coded is the first block, the transform means multiplies the first group of prediction errors, which are P×Q prediction errors, by a Q×M matrix to derive P×M intermediate values, and further multiplies the P×M intermediate values ​​by an N×P matrix to derive the first group of transform coefficients, which are N×M transform coefficients, from the first group of prediction errors; the first quantization matrix having the N×M elements is a quantization matrix that includes some elements of a third quantization matrix having R×S elements (R is an integer satisfying R≦N and S is an integer satisfying S≦M) but does not include other elements of the third quantization matrix, the second quantization matrix having the N×M elements is a quantization matrix that includes all elements in a fourth quantization matrix having R×S elements; the third quantization matrix is ​​different from the fourth quantization matrix; the first quantization matrix is ​​a quantization matrix configured with the part of elements in the third quantization matrix except for elements corresponding to DC components, the second quantization matrix is ​​a quantization matrix that is configured from all of the elements in the fourth quantization matrix except for an element corresponding to a DC component, The encoding means further encodes information indicating the value of the element corresponding to the DC component. An image encoding device comprising:

10. 1. An image coding method capable of coding an image using a plurality of blocks including a first block of P×Q pixels (P and Q are integers) and a second block of N×M pixels (N is an integer satisfying N<P and M is an integer satisfying M<Q), a transform step of deriving a first set of transform coefficients by performing a transform process on a first set of prediction errors corresponding to the first block, and deriving a second set of transform coefficients by performing a transform process on a second set of prediction errors corresponding to the second block; a quantization step of quantizing the first set of transform coefficients using a first quantization matrix having N×M elements to derive a first set of quantized transform coefficients, and quantizing the second set of transform coefficients using a second quantization matrix having N×M elements to derive a second set of quantized transform coefficients; an encoding step of encoding data corresponding to the first set of quantized transform coefficients corresponding to the first block and data corresponding to the second set of quantized transform coefficients corresponding to the second block; and at least the first set of prediction errors can be derived using a prediction method that combines intra-prediction and inter-prediction, When the block to be coded is the first block, in the transform step, the first group of prediction errors, which are P×Q prediction errors, are multiplied by a Q×M matrix to derive P×M intermediate values, and further, the first group of transform coefficients, which are N×M transform coefficients, are derive from the first group of prediction errors by multiplying the P×M intermediate values ​​by an N×P matrix; the first quantization matrix having the N×M elements is a quantization matrix that includes some elements of a third quantization matrix having R×S elements (R is an integer satisfying R≦N and S is an integer satisfying S≦M) but does not include other elements of the third quantization matrix, the second quantization matrix having the N×M elements is a quantization matrix that includes all elements in a fourth quantization matrix having R×S elements; the third quantization matrix is ​​different from the fourth quantization matrix; the first quantization matrix is ​​a quantization matrix configured with the part of elements in the third quantization matrix except for elements corresponding to DC components, the second quantization matrix is ​​a quantization matrix that is configured from all of the elements in the fourth quantization matrix except for an element corresponding to a DC component, In the encoding step, information indicating the value of the element corresponding to the DC component is further encoded.

1. An image encoding method comprising:

11. A program for causing a computer to execute the image decoding method according to any one of claims 2 to 8.

12. A program for causing a computer to execute the image coding method according to claim 10.

Images