Image encoding apparatus and method, and image decoding apparatus and method
By adopting the quantized matrix reference mode, the differential coding mode between quantized matrices and the differential coding mode within the quantized matrix in video encoding technology, the problem of excessive quantized matrix code in VVC is solved, and the encoding efficiency is improved.
Patent Information
- Application Number
- CN202411315495.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2019-09-19
- Filing Date
- 2020-08-14
- Publication Date
- 2024-11-12
AI Technical Summary
In video encoding technology, the amount of code of VVC's quantization matrix is too large, resulting in low encoding efficiency, especially due to the different distributions of orthogonal transformation coefficients of different sub-block shapes, resulting in an increase in the amount of code when defining the optimal quantization matrix separately.
Three quantized matrix coding modes are adopted: the quantized matrix reference mode, the differential coding mode between quantized matrices and the differential coding mode within the quantized matrix. By encoding the element differences of the quantized matrix, the code volume is reduced.
The code amount of quantization matrix is effectively suppressed and the encoding efficiency of image data is improved.
Smart Images

Figure CN118945353A_ABST
Abstract
Description
[0001] (This application is a divisional application of an application filed on August 14, 2020, with application number 202080065877.2 and invention name “Image encoding device, image decoding device, control method and program thereof”.) Technical Field
[0002] The present invention relates to an image encoding device, an image decoding device, a control method and a program thereof. Background Art
[0003] Known encoding methods for compressing and recording moving images include the HEVC (High Efficiency Video Coding) encoding method (hereinafter referred to as HEVC). In order to improve encoding efficiency, HEVC uses basic blocks that are larger than traditional macroblocks (16×16 pixels). These large basic blocks are called CTUs (Coding Tree Units) and have a size of up to 64×64 pixels. CTUs are further divided into sub-blocks that are units for prediction and transformation.
[0004] In addition, HEVC uses the following processing: weighting the coefficients of the orthogonal transformation called the quantization matrix (hereinafter referred to as the orthogonal transformation coefficients) according to the frequency component. By further reducing the data of the high-frequency components whose degradation is not easily visible to the human eye, the compression efficiency can be improved while maintaining the image quality. Patent document 1 discloses a technology for encoding such a quantization matrix.
[0005] In recent years, activities to perform international standardization of more efficient coding methods have been started as a successor to HEVC. JVET (Joint Video Experts Group) has been established between ISO / IEC and ITU-T, and standardization is being promoted as a VVC (Versatile Video Coding) coding method (hereinafter referred to as VVC). In order to improve coding efficiency, the size of the basic block in VVC is as high as 128×128 pixels, and in addition to the traditional square sub-block division, rectangular sub-block division is also under consideration.
[0006] Prior art literature
[0007] Patent Literature
[0008] Patent Document 1: Japanese Patent Application Laid-Open No. 2013-38758 Summary of the invention
[0009] Problem that the invention aims to solve
[0010] It is also considered to introduce a quantization matrix into VVC similar to HEVC. In addition, for VVC, more types of sub-block partitioning than HEVC are being considered, including rectangular sub-blocks. Since the distribution of orthogonal transform coefficients corresponding to each sub-block partitioning differs depending on the size and shape of the orthogonal transform, it is desirable to define an optimal quantization matrix according to the size and shape of the sub-block. However, if separate quantization matrices are defined for all sub-block shapes, the amount of code of the quantization matrix will increase unnecessarily.
[0011] The present invention has been made in view of these problems, and provides a technique for suppressing the code amount of a quantization matrix.
[0012] Solutions for solving problems
[0013] In order to solve this problem, the image encoding device of the present invention is equipped with the following configuration, for example. That is, an image encoding device for dividing an image into a plurality of blocks, quantizing a prediction error in units of the divided blocks, and encoding the image encoding device, characterized in that it includes: a quantization component for quantizing the prediction error of the block using a quantization matrix; and an encoding component for encoding the quantization matrix, the encoding component having: a first mode, in which information indicating that the quantization matrix to be encoded is the same as the encoded quantization matrix, and the elements of the quantization matrix to be encoded are not encoded, a second mode, in which information indicating that the difference between each element of the quantization matrix to be encoded and the encoded quantization matrix is encoded, and the difference between each element of the quantization matrix to be encoded and the encoded quantization matrix is encoded, and a third mode, in which information indicating that the difference between elements within the quantization matrix to be encoded is encoded, and the difference between elements within the quantization matrix to be encoded is encoded.
[0014] Effects of the Invention
[0015] According to the present invention, it is possible to improve the encoding efficiency of image data by suppressing the code amount of the quantization matrix more than before.
[0016] Other features and advantages of the present invention will be apparent from the following description in conjunction with the accompanying drawings. Note that throughout the accompanying drawings, the same reference numerals represent the same or similar components. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.
[0018] Figure 1 It is a block diagram showing the configuration of the image encoding device of the first embodiment.
[0019] Figure 2 It is a block diagram of the configuration of the image decoding device of the first embodiment.
[0020] Figure 3 is a flowchart showing the encoding process in the image encoding device of the first embodiment.
[0021] Figure 4 is a flowchart showing a decoding process in the image decoding device of the first embodiment.
[0022] Figure 5 is a diagram showing the configuration of computer hardware used in the second embodiment.
[0023] Fig. 6A is a diagram showing an example of the data structure of a bit stream generated by the image encoding device of the first embodiment.
[0024] Figure 6B is a diagram showing an example of the data structure of a bit stream generated by the image encoding device of the first embodiment.
[0025] Fig. 7A is a diagram showing an example sub-block partitioning in an embodiment.
[0026] Figure 7B is a diagram showing an example sub-block partitioning in an embodiment.
[0027] Figure 7C is a diagram showing an example sub-block partitioning in an embodiment.
[0028] Fig.7D is a diagram showing an example sub-block partitioning in an embodiment.
[0029] Fig. 7E is a diagram showing an example sub-block partitioning in an embodiment.
[0030] Figure 7F is a diagram showing an example sub-block partitioning in an embodiment.
[0031] Fig. 8A is a diagram showing an example of a quantization matrix used in the embodiment.
[0032] Figure 8B is a diagram showing an example of a quantization matrix used in the embodiment.
[0033] Figure 8C is a diagram showing an example of a quantization matrix used in the embodiment.
[0034] Fig.8D is a diagram showing an example of a quantization matrix used in the embodiment.
[0035] Fig. 8Eis a diagram showing an example of a quantization matrix used in the embodiment.
[0036] Figure 8F is a diagram showing an example of a quantization matrix used in the embodiment.
[0037] Figure 8G is a diagram showing an example of a quantization matrix used in the embodiment.
[0038] Figure 8H is a diagram showing an example of a quantization matrix used in the embodiment.
[0039] Figure 8I is a diagram showing an example of a quantization matrix used in the embodiment.
[0040] Figure 8J is a diagram showing an example of a quantization matrix used in the embodiment.
[0041] Figure 8K is a diagram showing an example of a quantization matrix used in the embodiment.
[0042] Figure 8L is a diagram showing an example of a quantization matrix used in the embodiment.
[0043] Fig.9A is a diagram showing a method for scanning elements of a quantization matrix used in the embodiment.
[0044] Fig. 9B is a diagram showing a method for scanning elements of a quantization matrix used in the embodiment.
[0045] Fig. 9C is a diagram showing a method for scanning elements of a quantization matrix used in the embodiment.
[0046] Fig. 10A FIG. 4 is a diagram showing a difference matrix of a quantization matrix generated in the embodiment.
[0047] Fig. 10B FIG. 4 is a diagram showing a difference matrix of a quantization matrix generated in the embodiment.
[0048] Fig. 10C FIG. 4 is a diagram showing a difference matrix of a quantization matrix generated in the embodiment.
[0049] Fig.11A is a diagram showing an example of a coding table for encoding a difference value of a quantization matrix.
[0050] Fig. 11B is a diagram showing an example of a coding table for encoding a difference value of a quantization matrix.
[0051] Fig.12: is a diagram showing an example of a syntax table used for a quantization matrix used in the embodiment.
[0052] Fig.13A is a diagram showing an example of a reference quantization matrix used in the embodiment.
[0053] Fig. 13B is a diagram showing an example of a reference quantization matrix used in the embodiment.
[0054] Fig. 13C is a diagram showing an example of a reference quantization matrix used in the embodiment.
[0055] Fig.14 is a flow chart illustrating a process for encoding a quantization matrix. DETAILED DESCRIPTION
[0056] The embodiments will be described in detail below with reference to the accompanying drawings. Note that the following embodiments do not limit the present invention according to the patent claims. The embodiments describe a plurality of features, but not all of these features are necessary for the present invention, and these features can be combined arbitrarily. In addition, in the accompanying drawings, the same reference numerals are given to the same or similar constituent elements, and their redundant descriptions will be omitted.
[0057] First embodiment
[0058] The embodiments describe a plurality of features, but not all of these features are essential to the present invention, and these features may be arbitrarily combined. In addition, the same reference numerals are given to the same or similar constituent elements, and redundant descriptions thereof will be omitted.
[0059] Figure 1 1 is a block diagram of the configuration of the image encoding device of the first embodiment. The image encoding device has a control unit 150 that manages the overall control of the device. The control unit 150 has a CPU, a ROM for storing programs executed by the CPU, and a RAM used as a work area of the CPU. The image encoding device has an input terminal 101, a block division unit 102, a quantization matrix holding unit 103, a prediction unit 104, a transform / quantization unit 105, an inverse quantization / inverse transform unit 106, an image reconstruction unit 107, a frame memory 108, a loop filtering unit 109, an encoding unit 110, an integrated encoding unit 111, an output terminal 112, and a quantization matrix encoding unit 113.
[0060] The input terminal 101 inputs the image data for encoding generated by the image data source in units of frames. The image data source is not limited in type, and is a camera, or a file server or storage medium storing the image data for encoding, etc. The output terminal 112 outputs the encoded data to an output destination device, and the output destination device is also not particularly limited, and is a storage medium or a file server, etc.
[0061] The block division unit 102 divides the input frame image into a plurality of basic blocks, and sequentially outputs one basic block of each frame to the downstream prediction unit 104 .
[0062] The quantization matrix holding unit 103 generates a plurality of quantization matrices before encoding, and holds the generated quantization matrices in an internal memory not shown. The method for generating the quantization matrix is not particularly limited, and may involve input by the user or calculation of the quantization matrix based on the characteristics of the input image, or use of a quantization matrix specified in advance as an initial value. The quantization matrix holding unit 103 in this embodiment generates and holds FIG. 8A to FIG. 8L The two-dimensional quantization matrices 800 to 811 of 8×8, 4×4 and 2×2 pixel sizes suitable for orthogonal transformation are shown. Here, quantization matrices 800, 803, 806 and 809 are quantization matrices used for the luminance component, and quantization matrices 801, 802, 804, 805, 807, 808, 810 and 811 are quantization matrices used for the two chrominance components.
[0063] The prediction unit 104 determines sub-block division for the image data in units of basic blocks, performs processing such as intra-frame prediction (which is prediction within a frame in units of sub-blocks) and inter-frame prediction (which is prediction between frames), and generates predicted image data. In addition, the prediction unit 104 calculates and outputs a prediction error in units of pixels based on the sub-blocks in the input image data and the corresponding predicted image data. In addition, the prediction unit 104 outputs information required for prediction (such as sub-block division, prediction mode, and motion vector, etc.) together with the prediction error. Hereinafter, the information required for prediction will be referred to as prediction information.
[0064] The transform / quantization unit 105 obtains an orthogonal transform coefficient by performing an orthogonal transform on the prediction error of the sub-block input from the prediction unit 104 in units of sub-blocks. In addition, the transform / quantization unit 105 quantizes the orthogonal transform coefficient using the quantization matrix stored in the quantization matrix holding unit 103, and obtains a residual coefficient (quantized orthogonal transform coefficient).
[0065] The inverse quantization / inverse transform unit 106 inputs the residual coefficient from the transform / quantization unit 105, performs inverse quantization using the corresponding quantization matrix stored in the quantization matrix holding unit 103, and reconstructs the orthogonal transform coefficient. The inverse quantization / inverse transform unit 106 further performs inverse orthogonal transform on the orthogonal transform coefficient to reconstruct the prediction error.
[0066] The image reconstruction unit 107 generates predicted image data by appropriately referring to the frame memory 108 based on the prediction information output from the prediction unit 104. The image reconstruction unit 107 generates reconstructed image data by adding the prediction error input from the inverse quantization / inverse transform unit 106 to the generated predicted image data, and stores the generated reconstructed image data in the frame memory 108.
[0067] The loop filtering unit 109 performs loop filtering such as deblocking filtering and sample adaptive offset on the reconstructed image stored in the frame memory 108 , and stores the filtered image data in the frame memory 108 again.
[0068] The encoding unit 110 encodes the residual coefficient output from the transform / quantization unit 105 and the prediction information output from the prediction unit 104 to generate encoded data, and outputs the encoded data to the integrated encoding unit 111 .
[0069] The quantization matrix encoding unit 113 converts the quantization matrix (see FIG. 8A to FIG. 8L ) is encoded to generate quantization matrix encoding data, and the quantization matrix encoding data is output to the integrated encoding unit 111.
[0070] The integrated coding unit 111 generates header coded data including the quantization matrix coded data output from the quantization matrix coding unit 113. Then, the integrated coding unit 111 positions the coded data output from the coding unit 110 after the header coded data and forms a bit stream. Then, the integrated coding unit 111 outputs the formed bit stream via the output terminal 112.
[0071] Here, the image encoding operation in the image encoding device will be described in more detail below. In the present embodiment, a configuration is adopted in which moving image data having a 4:2:0 color format is input in units of frames at a predetermined frame rate (e.g., 30 frames / second) from the input terminal 101, but a configuration in which still image data is input at 1 frame / minute may also be adopted. In addition, in the present embodiment, for the purpose of description, the block division unit 101 will be described as dividing the image data input from the input terminal 101 into basic blocks of 8×8 pixels. That is, the basic block of 8×8 pixels includes luminance (Y) components of 8×8 pixels and chrominance (Cb and Cr) components of 4×4 pixels. Note that this is for ease of understanding, and the present invention is not limited to the above numerical values (sizes).
[0072] Before encoding an image, a quantization matrix is generated and encoded.
[0073] The quantization matrix holding unit 103 first generates and holds a quantization matrix. Specifically, the quantization matrix holding unit 103 generates a quantization matrix according to the size of the sub-block to be encoded and the type of prediction method. Fig. 7A The quantization matrix corresponding to the basic block of 8×8 pixels not divided into sub-blocks shown in FIG. Figure 7B 4×4 pixel sub-blocks obtained by quadtree segmentation of the basic block shown in FIG. 4. That is, the quantization matrix holding unit 103 generates quantization matrices of 8×8 and 4×4 pixel sizes used for the luminance (Y) component, and quantization matrices of 4×4 and 2×2 pixel sizes used for the chrominance (Cb and Cr) components. However, the generated quantization matrix is not limited to this, and a quantization matrix corresponding to the shape of a sub-block such as 4×8 and 8×4 can be generated. The method for determining the elements constituting the quantization matrix is not particularly limited. For example, a predetermined initial value can be used, or the elements can be set separately. In addition, the quantization matrix can be generated according to the characteristics of the image.
[0074] The quantization matrix holding unit 103 holds the plurality of types of quantization matrices thus generated in an internal memory not shown. Fig. 8A A quantization matrix 800 of 8×8 pixel size for the Y component using intra prediction is shown. Figure 8B 4×4 pixel quantization matrix 801 is also shown for the Cb component using intra prediction, and Figure 8C A quantization matrix 802 of 4x4 pixel size is shown for the Cr component using intra prediction.
[0075] Similarly, Fig.8D 8×8 pixel quantization matrix 803 is shown for the Y component using inter-frame prediction. Fig. 8EAlso shown is a quantization matrix 804 of 4×4 pixel size used for the Cb component using inter-frame prediction, and Figure 8F A quantization matrix 805 of 4×4 pixel size is shown for the Cr component using inter-frame prediction.
[0076] also, Figure 8G A quantization matrix 806 of 4×4 pixel size is shown for the Y component using intra prediction. Figure 8H Also shown is a quantization matrix 807 of 2×2 pixel size used for the Cb component using intra prediction, and Figure 8I A quantization matrix 808 of 2x2 pixel size is shown for the Cr component using intra prediction.
[0077] Similarly, Figure 8J A quantization matrix 809 of size 4x4 pixels is shown for the Y component using inter-frame prediction. Figure 8K Also shown is a quantization matrix 810 of 2×2 pixel size used for the Cb component using inter-frame prediction, and Figure 8L A quantization matrix 811 of 2×2 pixel size used for the Cr component using inter-frame prediction is shown.
[0078] In order to simplify the description, configurations of 64 pixels (8×8), 16 pixels (4×4), and 4 pixels (2×2) are given, and each thick-framed square represents an element constituting a quantization matrix. FIG. 8A to FIG. 8L The 12 types of quantization matrices shown are maintained in a two-dimensional form, but the individual elements in the quantization matrix are of course not limited to this. In addition, according to the size of the sub-block, multiple quantization matrices can also be maintained for the same color component and the same prediction mode. Generally, since the quantization matrix implements a quantization process that depends on human visual characteristics, such as FIG. 8A to FIG. 8L As shown, the elements of the low-frequency components corresponding to the upper left portion of the quantization matrix are smaller, and the elements of the high-frequency components corresponding to the lower right portion are larger.
[0079] The quantization matrix encoding unit 113 sequentially reads out the two-dimensional quantization matrix held in the quantization matrix holding unit 103, and determines the encoding mode of each quantization matrix to be used when encoding the quantization matrix. In the present embodiment, three types of quantization matrix encoding modes are used, namely, "quantization matrix reference mode", "quantization matrix difference encoding mode" and "quantization matrix intra-difference encoding mode". The quantization matrix encoding unit 113 determines the quantization matrix encoding mode that minimizes the amount of data of the encoding result for each quantization matrix.
[0080] Here, three types of quantization matrix encoding modes used by the quantization matrix encoding unit 113 of this embodiment will be described in more detail.
[0081] The "quantization matrix reference mode" is a mode used when there is an encoded quantization matrix that matches the quantization matrix to be encoded. In the case of using this mode, the quantization matrix encoding unit 113 encodes a quantization matrix reference index (which is an identifier indicating the encoded quantization matrix that matches the quantization matrix to be encoded) instead of encoding the elements of the quantization matrix to be encoded.
[0082] Next, the "quantization matrix difference encoding mode" is a mode in which an encoded quantization matrix for approximating a quantization matrix to be encoded is selected, a difference value of each element is calculated, and the calculated difference value is encoded. When this mode is selected, the quantization matrix encoding unit 113 encodes a quantization matrix reference index (which is an identifier indicating a quantization matrix to be referenced), and then encodes difference information (which is a set of difference values of each element).
[0083] In addition, in the case where there is no encoded quantization matrix or in the case where there is no encoded quantization matrix similar to the quantization matrix to be encoded, the "intra-quantization matrix difference encoding mode" is used. The quantization matrix encoding unit scans each element of the quantization matrix to be encoded, calculates the difference between the elements, and encodes the difference information which is a set of the difference values.
[0084] Fig.12 This is a syntax table used when encoding the quantization matrix in this embodiment. This syntax table will be used to specifically explain the present embodiment. FIG. 8A to FIG. 8L First, we will explain Fig.12 The structure of the syntax table in . At the outermost side of the syntax table is a "for" loop for the parameter "sizeId". sizeId indicates the size of the quantization matrix, and quantization matrices of the same size are encoded continuously. sizeId=1 indicates a quantization matrix of 2×2 pixel size. Similarly, sizeId=2 indicates a quantization matrix of 4×4 pixel size, and sizeId=3 indicates a quantization matrix of 8×8 pixel size.
[0085] The initial value of sizeId in the for loop is "1", and it is defined that the value will be incremented by "1" for each loop, and the loop will be repeated as long as sizeId is less than 4. Therefore, in the present embodiment, this means that after encoding a group of quantization matrices of 2×2 pixel size, a group of quantization matrices of 4×4 pixel size will be encoded, and then a group of quantization matrices of 8×8 pixel size will be encoded. In VVC, an orthogonal transform from a minimum 2×2 pixel size to a 64×64 pixel size is used, and the following configuration can also be adopted: a for loop from a 2×2 pixel size with sizeId of 1 to a 64×64 pixel size with sizeId of 6 is set, and the quantization matrices corresponding to these pixel sizes are encoded respectively. However, in the present embodiment, the maximum orthogonal transform size (i.e., the maximum size of the sub-block) is 8×8, so the upper limit value of the for loop is sizeId=3 corresponding to the 8×8 pixel size. This enables unnecessary quantization matrix encoding to be omitted and redundant code generation to be prevented by setting the upper limit value of the for loop based on the maximum value of the orthogonal transform size actually used.
[0086] Within the for loop for sizeId, there is a for loop for the parameter matrixId. The matrixId is a parameter based on the prediction mode and the color component, and a value of 0 means intra-frame prediction / Y component, a value of 1 means intra-frame prediction / Cb component, and a value of 2 means intra-frame prediction / Cr component. Similarly, a matrixId of 3 means inter-frame prediction / Y component, a matrixId of 4 means inter-frame prediction / Cb component, and a matrixId of 5 means inter-frame prediction / Cr component. That is, in a set of quantization matrices of the same size, the quantization matrix corresponding to the intra-frame prediction is encoded first, and then the quantization matrix corresponding to the inter-frame prediction is encoded. In addition, in a set of quantization matrices corresponding to the same prediction mode, the quantization matrices are encoded in the order of Y component, Cb component, and Cr component.
[0087] Considering the encoding order of the quantization matrix above, FIG. 8A to FIG. 8L The encoding order of the quantization matrix is as follows.
[0088] First of all, Figure 8H , Figure 8I , Figure 8K and Figure 8L The quantization matrix of 2×2 pixels is encoded. Figure 8G , Figure 8B , Figure 8C , Figure 8J , Fig. 8E and Figure 8F Finally, the 4×4 pixel size quantization matrix is encoded. Fig. 8Aand Fig.8D The quantization matrix of 8×8 pixel size is encoded.
[0089] In short, the above encoding order involves encoding the quantization matrix in the following priority order: (i) quantization matrix size, (ii) intra / inter type, and (iii) component (luminance, chrominance) type. The quantization matrix size is the first priority order because the encoding is performed taking into account the correlation between quantization matrices.
[0090] Next, the process for encoding each quantization matrix will be described. First, the quantization matrix encoding unit 113 encodes the determined quantization matrix encoding mode. Fig.12 In the syntax table of , scaling_list_pred_mode_idx corresponds to the quantization matrix coding mode. In the present embodiment, the "quantization matrix reference mode" is used when its value is 0, the "inter-quantization matrix differential coding mode" is used when the value is 1, and the "intra-quantization matrix differential coding mode" is used when the value is 2. Since the quantization matrix coding mode is Golomb coding, as the value decreases, the amount of code associated with scaling_list_pred_mode_idx decreases. That is, in the present embodiment, the amount of code associated with the matrix reference mode is the smallest, which is advantageous because the amount of code generated is minimized in the presence of a large number of identical quantization matrices.
[0091] In the case of using the quantization matrix reference mode, the quantization matrix encoding unit 113 then encodes the quantization matrix reference index. Fig.12 The syntax table of corresponds to scaling_list_pred_matrix_id_delta. For example, Figure 8B The quantization matrix 801 of the most recently encoded Figure 8G The quantization matrix 806 is the same as Figure 8B When encoding the quantization matrix 801 of the most recently encoded quantization matrix, the quantization matrix encoding unit 113 selects the quantization matrix reference mode and encodes 0 indicating that the quantization matrix 801 is the most recently encoded quantization matrix as the quantization matrix reference index. This enables Figure 8G The same quantization matrix as the quantization matrix 806 is encoded. Note that in the case of using the second most recently encoded quantization matrix, the quantization matrix reference index will be "1".
[0092] On the other hand, in the case of using the inter-quantization matrix difference coding mode, similar to the case of using the quantization matrix reference mode, the quantization matrix coding unit 113 first encodes the quantization matrix reference index. Then, the quantization matrix coding unit 113 encodes the difference information, which is the difference between each element of the encoded reference quantization matrix defined by the quantization matrix reference index and the corresponding element of the quantization matrix to be encoded. This Fig.12 The syntax table of corresponds to scaling_list_pred_delta. For example, Figure 8C The elements of the quantization matrix 802 are related to the most recently encoded Figure 8B The corresponding elements of the quantization matrix 801 are similar (in this example, the absolute value of the difference between the corresponding elements is less than or equal to a predetermined value (exemplified as "1" or less)). Figure 8C When encoding the quantization matrix 802 of the most recently encoded quantization matrix, the quantization matrix encoding unit 113 selects the inter-quantization matrix difference encoding mode and first encodes 0 indicating that the quantization matrix 802 is the most recently encoded quantization matrix as the quantization matrix reference index. Note that in the case of encoding the difference with each element of the second most recently encoded quantization matrix, the quantization matrix reference index will be "1". Subsequently, the quantization matrix encoding unit 113 calculates the reference quantization matrix Figure 8B The elements of Figure 8C The difference between the corresponding elements of and calculate Fig. 10A The two-dimensional difference matrix 1000 is shown. Then, the quantization matrix encoding unit 113 scans the difference values of the two-dimensional difference matrix 1000 and arranges the difference values in a one-dimensional matrix. 9A to 9C The scanning method used when the two-dimensional matrix is arranged one-dimensionally in this embodiment is shown, wherein Fig.9A , Fig. 9B and Fig. 9C They correspond to 8×8 quantization matrix, 4×4 quantization matrix and 2×2 quantization matrix respectively. Since the 4×4 quantization matrix is used as the object here, the quantization matrix encoding unit 113 uses Fig. 9B The scanning method shown will Fig. 10A The two-dimensional difference matrix 1000 is arranged in Fig. 10B Then, the quantization matrix encoding unit 113 performs Fig. 10B The difference values of the matrix 1001 are encoded. Note that Fig. 10B The absolute value of each difference value in is "1" or less, but there may be a case where the absolute value is greater than 1. By further obtaining the difference between adjacent difference values in consideration of this case, the amount of code can be further reduced.
[0093] In addition, when using the intra-quantization matrix difference encoding mode, the quantization matrix encoding unit 113 uses 9A to 9C One of them scans each element of the quantization matrix to be encoded, calculates the difference between two consecutive elements, and arranges these differences in a one-dimensional matrix. Then, the quantization matrix encoding unit 113 encodes each difference value arranged in the one-dimensional matrix as difference information. Fig.12 The syntax table of corresponds to scaling_list_delta_coef. For example, since Figure 8G The quantization matrix 806 of is the first of the quantization matrices of 4×4 pixel size to be encoded, so there is no quantization matrix to be referenced using the above-mentioned quantization matrix reference mode or inter-quantization matrix difference encoding mode. Figure 8G When encoding the quantization matrix 806 of , the quantization matrix encoding unit 113 must select the intra-quantization matrix differential encoding mode. Therefore, the quantization matrix encoding unit 113 uses Fig. 9B scanning Figure 8G The quantization matrix encoding unit 113 arranges the calculated differences in a one-dimensional matrix and obtains Fig. 10C The one-dimensional difference matrix 1002 is shown. Here, for example, by Fig. 9B Scanning method shown Figure 8G A quantization matrix of 4×4 pixel size is used, thereby scanning the element 13 located immediately below the first element 6 located in the upper left corner, and calculating +7 as the difference. In addition, when encoding the first element of the quantization matrix (6 in the present embodiment), it is assumed that the difference from a predetermined initial value (e.g., 8) is calculated, but the present invention is of course not limited to this, and the difference from an arbitrary value or the value of the first element itself may be used. The initial value only needs to be the same as that of the decoding device. Then, the quantization matrix encoding unit 113 performs Fig. 10C The difference values are encoded.
[0094] Here, the above-mentioned process for encoding the quantization matrix will be further explained. Fig.12 The fifth "scaling_list_pred_mode_id×[sizeId][matrixId]" from the top is a parameter indicating the encoding mode of the quantization matrix that depends on sizeId and matrixId. When calculating this parameter, the quantization matrix ( FIG. 8A to FIG. 8L One of them) is encoded, and the encoding mode that minimizes the amount of generated code is determined. Then, information specifying the determined encoding mode (0 to 2 mentioned above) is set as its parameter.
[0095] The quantization matrix encoding unit 113 generates quantization matrix encoded data as the encoding result of each quantization matrix generated using the quantization matrix encoding mode in the above manner. The quantization matrix encoding unit 113 of this embodiment uses Fig.11A The coding table shown is used to encode each element of the one-dimensional difference matrix and generate quantization matrix encoded data. Note that the coding table is not limited to this, and for example, Fig. 11B The coding table shown in FIG. 114 shows a coding table of a quantization matrix encoding unit 113. In this way, the quantization matrix encoding unit 113 outputs the generated quantization matrix encoding data to the downstream integrated encoding unit 111.
[0096] Back to Figure 1 , the integrated encoding unit 111 integrates the quantization matrix encoding data into the header information required when encoding the image data.
[0097] The image data of one frame input from the input terminal 101 is supplied to the block division unit 102 .
[0098] The block division unit 102 divides the input image data of one frame into a plurality of basic blocks, and outputs the image data in units of basic blocks to the prediction unit 104. In this embodiment, the image data in units of basic blocks of 8×8 pixels is supplied to the prediction unit 104.
[0099] The prediction unit 104 performs prediction processing on the image data in units of basic blocks input from the block division unit 102. Specifically, the prediction unit 104 determines sub-block division for dividing the basic block into smaller sub-blocks, and further determines a prediction mode such as intra prediction or inter prediction in units of sub-blocks. Intra prediction involves generating predicted pixels of the block to be encoded using already encoded pixels spatially located around the block to be encoded, and also generates an intra prediction mode indicating an intra prediction method such as horizontal prediction, vertical prediction, or DC prediction. Inter prediction involves generating predicted pixels of the block to be encoded using already encoded pixels of a frame that is temporally different from the block to be encoded, and also generates motion information indicating a reference frame or a motion vector, etc.
[0100] Will refer to 7A to 7F To illustrate the sub-block segmentation method. 7A to 7F The bold frames of blocks 700 to 705 are 8×8 pixels in size, which is the same as the basic block. Each rectangle within the bold frame represents a sub-block. Figure 7B , represents an example of conventional square sub-block partitioning, where a basic block 701 of 8×8 pixels is partitioned into four sub-blocks of 4×4 pixels. FIG. 7C to FIG. 7F An example showing rectangular sub-block segmentation. Figure 7C The basic block 702 is shown to be divided into two sub-blocks of 4×8 pixels in size (vertically long). Fig.7D It is shown that the basic block 703 is divided into two sub-blocks of 8×4 pixels in size (longer in the horizontal direction). Fig. 7E and Figure 7F It is shown that the basic blocks 704 and 705 are divided into three rectangular sub-blocks in a ratio of 1:2:1, although the division method is different between the two. In this way, not only square but also rectangular sub-blocks are used for encoding.
[0101] From using a basic block of 8×8 pixel size that is not split into sub-blocks Fig. 7A and the basic block is split into four Figure 7B The present embodiment is described in one aspect, but the sub-block segmentation method is not limited thereto. Fig. 7E and Figure 7F Ternary tree partitioning such as Figure 7C and Fig.7D The binary tree partitioning shown in the figure is also used in addition to Fig. 7A and Figure 7B In the case of sub-block division other than , a quantization matrix corresponding to the sub-block to be used is generated in the quantization matrix holding unit 103. In addition, the generated quantization matrix is encoded by the quantization matrix encoding unit 113.
[0102] The prediction unit 104 generates predicted image data according to the determined prediction mode and the coding area stored in the frame memory 108, further calculates a prediction error in units of pixels according to the predicted image data corresponding to the subblock of interest in the input image data, and outputs the calculated error to the transform / quantization unit 105. In addition, the prediction unit 104 outputs information such as subblock division and prediction mode to the encoding unit 110 and the image reconstruction unit 107 as prediction information.
[0103] The transform / quantization unit 105 performs orthogonal transform and quantization on the prediction error input from the prediction unit 104, and generates a residual coefficient. Specifically, the transform / quantization unit 105 first performs an orthogonal transform process depending on the size of the sub-block on the prediction error, and generates an orthogonal transform coefficient. Then, the transform / quantization unit 105 quantizes the orthogonal transform coefficient using the quantization matrix stored in the quantization matrix holding unit 103 according to the prediction mode and the color component, and generates a residual coefficient. In this embodiment, when sub-block division is not performed and the intra-frame prediction mode is used, the orthogonal transform coefficient of the Y component is used. Fig. 8A The quantization matrix is used for the orthogonal transform coefficients of the Cb component. Figure 8B The quantization matrix, and the orthogonal transform coefficients of the Cr component are used Figure 8C Similarly, without performing sub-block segmentation and using the inter-frame prediction mode instead, the orthogonal transform coefficients of the Y component are used. Fig.8D The quantization matrix is used for the orthogonal transform coefficients of the Cb component. Fig. 8E The quantization matrix, and the orthogonal transform coefficients of the Cr component are used Figure 8F On the other hand, when performing Figure 7B In the case of sub-block partitioning and using intra-frame prediction mode, the orthogonal transform coefficient of the Y component is used Figure 8G The quantization matrix is used for the orthogonal transform coefficients of the Cb component. Figure 8H The quantization matrix, and the orthogonal transform coefficients of the Cr component are used Figure 8I In addition, similarly, when performing Figure 7B In the case of sub-block partitioning and using inter-frame prediction mode instead, the orthogonal transform coefficients for the Y component are used Figure 8J The quantization matrix is used for the orthogonal transform coefficients of the Cb component. Figure 8K The quantization matrix, and the orthogonal transform coefficients of the Cr component are used Figure 8L However, the quantization matrix used is not limited thereto. The generated residual coefficient and chroma integration information are output to the encoding unit 110 and the inverse quantization / inverse transformation unit 106.
[0104] The inverse quantization / inverse transform unit 106 reconstructs the orthogonal transform coefficient by inverse quantizing the residual coefficient input from the transform / quantization unit 105, using the corresponding quantization matrix stored in the quantization matrix holding unit 103. The inverse quantization / inverse transform unit 106 further performs inverse orthogonal transform on the reconstructed orthogonal transform coefficient to reconstruct the prediction error. In the inverse quantization process, similar to the transform / quantization unit 105, a quantization matrix corresponding to the size and color component of the sub-block to be encoded is used. Specifically, the inverse quantization / inverse transform unit 106 performs inverse quantization using the same quantization matrix used by the transform / quantization unit 105. That is, in the case where sub-block division is not performed and the intra-frame prediction mode is used, the orthogonal transform coefficient of the Y component is used. Fig. 8A The quantization matrix is used for the orthogonal transform coefficients of the Cb component. Figure 8B The quantization matrix, and the orthogonal transform coefficients of the Cr component are used Figure 8C Similarly, without performing sub-block segmentation and using the inter-frame prediction mode instead, the orthogonal transform coefficients of the Y component are used. Fig.8D The quantization matrix is used for the orthogonal transform coefficients of the Cb component. Fig. 8E The quantization matrix, and the orthogonal transform coefficients of the Cr component are used Figure 8F On the other hand, when performing Figure 7B When the sub-block is partitioned and the intra prediction mode is used, the orthogonal transform coefficient of the Y component is used Figure 8G The quantization matrix is used for the orthogonal transform coefficients of the Cb component. Figure 8H The quantization matrix, and the orthogonal transform coefficients of the Cr component are used Figure 8I In addition, similarly, when performing Figure 7B In the case of sub-block partitioning and using inter-frame prediction mode instead, the orthogonal transform coefficients for the Y component are used Figure 8J The quantization matrix is used for the orthogonal transform coefficients of the Cb component. Figure 8K The quantization matrix, and the orthogonal transform coefficients of the Cr component are used Figure 8L The quantization matrix of .
[0105] The prediction error reconstructed by performing inverse orthogonal transform on the orthogonal transform coefficient thus reconstructed is output to the image reconstruction unit 107 .
[0106] The image reconstruction unit 107 reconstructs a predicted image by appropriately referring to the frame memory 108 based on the prediction information input from the prediction unit 104. Then, the image reconstruction unit 107 generates reconstructed image data in units of sub-blocks based on the reconstructed predicted image and the prediction error of the sub-block reconstructed by the inverse quantization / inverse transform unit 106, and stores the generated reconstructed image data in the frame memory 108.
[0107] The loop filtering unit 109 reads out the reconstructed image data from the frame memory 108 and performs loop filtering such as deblocking filtering, etc. Then, the loop filtering unit 109 stores the filtered image data in the frame memory 108 again.
[0108] The encoding unit 110 performs entropy encoding on the residual coefficient and chroma integration information in units of subblocks generated by the transform / quantization unit 105 and the prediction information input from the prediction unit 104, and generates encoded data. Although no specific entropy encoding method is specified, methods such as Golomb encoding, arithmetic encoding, and Huffman encoding can be used. The encoding unit 110 outputs the generated encoded data to the integration encoding unit 111.
[0109] The integration coding unit 111 multiplexes the coded data input from the coding unit 110 and the coded data of the header described above to form a bit stream. Then, the integration coding unit 111 outputs the formed bit stream from the output terminal 112 to the outside (to a storage medium, a network, etc.).
[0110] Fig. 6Ais an example of the data structure of the bit stream output in this embodiment. The sequence header includes the encoded data of the quantization matrix and is composed of the encoded result of the quantization matrix. However, the location where the encoded data of the quantization matrix is stored is not limited to this, and of course, the following configuration can also be adopted: the encoded data is arranged in a location such as Figure 6B In the picture header or the header spanning multiple pictures as shown. In addition, in the case of changing the quantization matrix in a single sequence, it is also possible to update by newly encoding the quantization matrix. At this time, it is possible to cover all quantization matrices, and the following configuration can also be adopted: a part of the quantization matrix is changed by specifying the size of the quantization matrix corresponding to the quantization matrix to be covered, the prediction mode and the color component.
[0111] Figure 3 1 is a flowchart showing encoding processing of one frame by the control unit 150 in the image encoding device of the embodiment.
[0112] First, before encoding an image, in step S301, the control unit 150 controls the quantization matrix holding unit 103 to generate and hold a two-dimensional quantization matrix. The quantization matrix holding unit 103 of this embodiment is applicable to blocks of 8×8 to 2×2 pixel sizes, and generates and holds FIG. 8A to FIG. 8L The quantization matrices corresponding to the respective color components and prediction modes are shown.
[0113] In step S302, the control unit 150 controls the quantization matrix encoding unit 113 to encode the quantization matrix generated and maintained in step S301. The specific operation of the quantization matrix encoding unit 113 has been described here, so it will be omitted. In this embodiment, the control unit 150 controls the quantization matrix encoding unit 113 to encode the quantization matrix generated and maintained in step S301. FIG. 8A to FIG. 8L The quantization matrices 801 to 811 shown are based on Fig.12 The syntax table is encoded and quantization matrix encoding data is generated.
[0114] In step S303 , the control unit 150 controls the integrated encoding unit 111 to encode and output header information required when encoding image data together with the generated quantization matrix encoded data.
[0115] In step S304 , the control unit 150 controls the block division unit 102 to divide the input image in frame units into basic block units.
[0116] In step S305, the control unit 150 controls the prediction unit 104 to divide the image data in units of basic blocks generated in step S304 into sub-blocks, perform prediction processing on each sub-block, and generate predicted image data and prediction information such as sub-block division information and prediction mode, etc. In addition, the control unit 150 controls the prediction unit 104 to calculate a prediction error based on the input image data of the sub-block and the predicted image data.
[0117] In step S306, the control unit 150 controls the transform / quantization unit 105 to perform an orthogonal transform on the prediction error calculated in step S305 and generate an orthogonal transform coefficient. In addition, the control unit 150 controls the transform / quantization unit 105 to perform quantization using the quantization matrix generated and held in step S301 and generate a residual coefficient. In this embodiment, the prediction mode and the color component are used according to the sub-block size. FIG. 8A to FIG. 8L The quantization matrix of .
[0118] In step S307, the control unit 150 controls the inverse quantization / inverse transform unit 106 to inversely quantize the residual coefficient generated in step S306 using the quantization matrix generated and held in step S301 and reconstruct the orthogonal transform coefficient. In this step, the inverse quantization process is performed using the same quantization matrix as that used in step S306. Then, the reconstructed orthogonal transform coefficient is inversely orthogonally transformed and the prediction error is reconstructed.
[0119] In step S308 , the control unit 150 controls the image reconstruction unit 107 to reconstruct a predicted image based on the prediction information generated in step S305 , reconstruct image data according to the reconstructed predicted image and the prediction error generated in step S307 , and store the reconstructed image data in the frame memory 108 .
[0120] In step S309, the control unit 150 controls the encoding unit 110 to encode the prediction information generated in step S305 and the residual coefficient generated in step S306 and generate encoded data. In addition, the encoding unit 110 outputs the generated encoded data to the integrated encoding unit 111. The integrated encoding unit 111 positions the encoded data from the encoding unit 110 after the previously generated header and outputs the data thus obtained.
[0121] In step S310, the control unit 150 determines whether all basic blocks in the focus frame have been encoded. If it is determined that the encoding is completed, the control unit 150 proceeds to step S311, and if it is determined that there are any basic blocks that have not been encoded, the control unit 150 returns the process to step S304 and continues encoding the next basic block.
[0122] In step S311 , the control unit 150 controls the loop filtering unit 109 to loop-filter the image data reconstructed in step S308 and generate a filtered image, and ends the process.
[0123] In the above-described configuration and operation, in particular, by determining the quantization matrix encoding mode that minimizes the amount of generated code of the quantization matrix and encoding it in step S302, the amount of generated code of the quantization matrix can be suppressed.
[0124] Note that in the present embodiment, only quantization matrices of the same size can be referenced in the quantization matrix reference mode and the inter-quantization matrix difference encoding mode, but a configuration in which quantization matrices of different sizes can be referenced using enlargement and reduction can also be adopted. Figure 8G When encoding the quantization matrix 806, the following configuration may also be adopted: refer to and amplify Figure 8H The quantization matrix 807 of the first encoding is obtained, and the difference between the two quantization matrices is encoded as difference information. Thus, the amount of data of the quantization matrix initially encoded in each size can be further reduced.
[0125] In addition, in this embodiment, Fig.12 As shown in the syntax table of , the configuration of encoding the quantization matrix in ascending order of size is adopted, but the configuration of encoding the quantization matrix in descending order of size can also be adopted. Therefore, especially in the above-mentioned case where references can be made between quantization matrices of different sizes, the amount of data of the quantization matrix can be further reduced.
[0126] Furthermore, in the present embodiment, the reference object of the quantization matrix is limited to the encoded quantization matrix, but the following configuration may be adopted: the reference object is such as FIG. 13A to FIG. 13C The flat quantization matrix used as a reference when no quantization matrix is used is shown in the figure etc. Thus, particularly when the quantization matrix to be encoded is close to the flat quantization matrix, the amount of data of the quantization matrix can be further reduced.
[0127] Figure 2 The following is a block diagram of a configuration of an image decoding device for decoding coded image data generated by an image coding device. Figure 2 To explain the configuration and operation related to the decoding process.
[0128] The image decoding device has a control unit 250 that manages the overall control of the device. The control unit 250 has a CPU, a ROM for storing programs executed by the CPU, and a RAM used as a work area of the CPU. In addition, the image decoding device has an input terminal 201, a separation decoding unit 202, a decoding unit 203, an inverse quantization / inverse transformation unit 204, an image reconstruction unit 205, a frame memory 206, a loop filtering unit 207, an output terminal 208, and a quantization matrix decoding unit 209.
[0129] The input terminal 201 inputs a coded bit stream, wherein the input source is, for example, a storage medium storing the coded bit stream, but the coded bit stream may be input from a network, and the type of the coded bit stream is not limited.
[0130] The separation decoding unit 202 separates information related to the decoding process and coded data related to the coefficients from the bit stream, and decodes the coded data in the header of the bit stream. The separation decoding unit 202 of this embodiment separates the quantization matrix coded data and outputs the separated quantization matrix coded data to the quantization matrix decoding unit 209. In addition, the separation decoding unit 202 outputs the coded data of the image to the decoding unit 203. That is, the separation decoding unit 202 performs Figure 1 The inverse operation of the integrated coding unit 111.
[0131] The quantization matrix decoding unit 209 reconstructs and holds the quantization matrix by decoding the quantization matrix encoded data supplied from the separation decoding unit 202 .
[0132] The decoding unit 203 decodes the encoded data of the image output from the separation decoding unit 202, and reconstructs the residual coefficient and prediction information in units of subblocks.
[0133] and Figure 1 Similar to the inverse quantization / inverse transform unit 106, the inverse quantization / inverse transform unit 204 reconstructs the prediction error by inverse quantizing the residual coefficients of the subblock of interest using the reconstructed quantization matrix, obtaining the inverse quantized coefficients, and performing an inverse orthogonal transform.
[0134] The image reconstruction unit 205 generates predicted image data based on the input prediction information by appropriately referring to the frame memory 206. Then, the image reconstruction unit 205 generates reconstructed image data of the subblock of interest based on the predicted image data and the prediction error reconstructed by the inverse quantization / inverse transform unit 204, and stores the generated reconstructed image data in the frame memory 206.
[0135] and Figure 1Similar to the loop filtering unit 109 , the loop filtering unit 207 performs loop filtering such as deblocking filtering on the reconstructed image data stored in the frame memory 206 , and stores the filtered image data in the frame memory 206 again.
[0136] The output terminal 208 sequentially outputs the frame images stored in the frame memory 206 to the outside. The output destination is usually a display device, but may be other devices.
[0137] The operation related to decoding an image using the image decoding apparatus of the above-described embodiment will now be described in more detail. In this embodiment, a coded bit stream is input in units of frames.
[0138] exist Figure 2 2, a bit stream of one frame input from the input terminal 201 is supplied to the separation decoding unit 202. The separation decoding unit 202 separates information related to the decoding process and coded data related to the coefficients from the bit stream, and decodes the coded data present in the header of the bit stream. The separation decoding unit 202 then supplies the quantization matrix coded data included in the header to the quantization matrix decoding unit 209, and supplies the coded data of the image data to the decoding unit 203. Specifically, the separation decoding unit 202 first separates the information related to the decoding process and the coded data related to the coefficients from the bit stream, and decodes the coded data present in the header of the bit stream. Fig. 6A The quantization matrix encoding data is extracted from the sequence header of the bit stream shown in FIG. 1 and the extracted quantization matrix encoding data is output to the quantization matrix decoding unit 209. In this embodiment, the quantization matrix encoding data is extracted and outputted. FIG. 8A to FIG. 8L Next, the quantization matrix coded data corresponding to the quantization matrix shown is extracted in units of basic blocks of the picture data and output to the decoding unit 203.
[0139] The quantization matrix decoding unit 209 first decodes the input quantization matrix encoded data, and based on Fig.12 The syntax table shown in FIG. 1 is used to reconstruct the encoding results of each quantization matrix generated at the encoding side. Fig. 6A As shown, the encoding result of each quantization matrix is composed of a quantization matrix encoding mode and data corresponding to the quantization matrix encoding mode.
[0140] For example, if the quantization matrix coding mode (i.e. Fig.12If the scaling_list_pred_mode_idx in the syntax table of is 0, this means that the quantization matrix is encoded in the quantization matrix reference mode. In this case, the quantization matrix encoding result consists of the quantization matrix encoding mode (0) and the quantization matrix reference index. For example, when decoding the 4×4 quantization matrix used for intra prediction and Cb component, the quantization matrix decoding unit 209 first reconstructs the quantization matrix encoding mode = 0 (which means the quantization matrix reference mode). Then, the quantization matrix decoding unit 209 reconstructs the quantization matrix reference index. In the case where the quantization matrix reference index indicates that the quantization matrix to be referenced is the most recently decoded quantization matrix, the quantization matrix reference index will be 0. If the quantization matrix reference index is 1, the second most recently decoded quantization matrix is referenced. As a result of this information, the quantization matrix 209 knows that the 4×4 quantization matrix used for intra prediction and Cb component is different from the one used for intra prediction and Y component. Figure 8G The quantization matrix 806 is the same as that of Figure 8B Quantization matrix 801.
[0141] In addition, if the quantization matrix coding mode (i.e., Fig.12 ) is 1, this means that the quantization matrix is encoded in the inter-quantization matrix difference coding mode. In this case, the quantization matrix encoding result consists of the quantization matrix encoding mode (1), the quantization matrix reference index, and the difference information. For example, when decoding the 4×4 quantization matrix used for intra-frame prediction and Cr components, the quantization matrix decoding unit 209 first reconstructs the quantization matrix encoding mode = 1 (which means the inter-quantization matrix difference coding mode). Then, the quantization matrix decoding unit 209 reconstructs the quantization matrix reference index. In the case where the quantization matrix reference index indicates that the quantization matrix to be referenced is the most recently decoded quantization matrix, the quantization matrix reference index will be 0. If the quantization matrix reference index is 1, the second most recently decoded quantization matrix is referenced. Subsequently, the quantization matrix decoding unit 209 performs a differential analysis on the difference information (i.e., Fig.12 The number of elements in the quantization matrix is decoded by the scaling_list_pred_delta in the syntax table of Fig. 10B The one-dimensional difference matrix 1001 is shown. The quantization matrix decoding unit 209 uses Fig. 9B The scanning method is used to arrange the Fig. 10B The one-dimensional difference matrix 1001 is obtained Fig. 10A The two-dimensional difference matrix 1000 is shown. Then, the quantization matrix decoding unit 209 converts Fig. 10A The differences between the two-dimensional difference matrix 1000 and Figure 8B The elements of the quantization matrix 801 to be referenced in are added, and the Figure 8C quantization matrix 802.
[0142] On the other hand, if the quantization matrix coding mode (i.e. Fig.12 If the scaling_list_pred_mode_idx in the syntax table of is 2, this means that the quantization matrix is encoded in the intra-quantization matrix differential coding mode. In this case, the quantization matrix encoding result is composed of the quantization matrix encoding mode (2) and the difference information. For example, when decoding the 4×4 quantization matrix used for intra-frame prediction and Y component, the quantization matrix decoding unit 209 first reconstructs the quantization matrix encoding mode = 2 (which means the intra-quantization matrix differential coding mode). Then, the quantization matrix decoding unit 209 reconstructs the difference information (i.e., Fig.12 The number of elements in the quantization matrix is decoded by the scaling_list_delta_coef in the syntax table of Fig. 10C The one-dimensional difference matrix 1002 is shown. Fig. 10C Each difference in the one-dimensional difference matrix 1002 is added to the corresponding immediately preceding element, using Fig. 9B The quantization matrix decoding unit 209 is two-dimensionally arranged by a scanning method, and the quantization matrix decoding unit 209 reconstructs Figure 8G Quantization matrix 806.
[0143] Note that in this embodiment, when decoding the difference information, Fig.11A , but you can also use Fig. 11B In short, the same coding table as that on the encoding side can be used. Then, the quantization matrix decoding unit 209 holds the quantization matrices 800 to 811 thus reconstructed. Here, the operation is opposite to that using the quantization matrix encoding unit 113 on the encoding side.
[0144] The decoding unit 203 decodes the coded data supplied from the separation decoding unit 202, reconstructs the prediction information, and further reconstructs the residual coefficient. First, the decoding unit 203 reconstructs the prediction information and obtains the prediction mode used in the sub-block. The decoding unit 203 outputs the reconstructed residual coefficient to the inverse quantization / inverse transform unit 204, and outputs the reconstructed prediction information to the image reconstruction unit 205.
[0145] The inverse quantization / inverse transform unit 204 uses the quantization matrix reconstructed by the quantization matrix decoding unit 209 to inverse quantize the input residual coefficient to generate an orthogonal transform coefficient, and further performs inverse orthogonal transform to reconstruct the prediction error. Similar to the inverse quantization / inverse transform unit 106 on the encoding side, the inverse quantization / inverse transform unit 204 performs inverse quantization using a quantization matrix corresponding to the size and color component of the decoded sub-block. That is, in the case where sub-block division is not performed and the intra-frame prediction mode is used, the orthogonal transform coefficient of the Y component is used. Fig. 8A The quantization matrix is used for the orthogonal transform coefficients of the Cb component. Figure 8B The quantization matrix, and the orthogonal transform coefficients of the Cr component are used Figure 8C Similarly, without performing sub-block segmentation and using the inter-frame prediction mode instead, the orthogonal transform coefficients of the Y component are used. Fig.8D The quantization matrix is used for the orthogonal transform coefficients of the Cb component. Fig. 8E The quantization matrix, and the orthogonal transform coefficients of the Cr component are used Figure 8F On the other hand, when performing Figure 7B When the sub-block is partitioned and the intra prediction mode is used, the orthogonal transform coefficient of the Y component is used Figure 8G The quantization matrix is used for the orthogonal transform coefficients of the Cb component. Figure 8H The quantization matrix, and the orthogonal transform coefficients of the Cr component are used Figure 8I In addition, similarly, when performing Figure 7B In the case of sub-block partitioning and using inter-frame prediction mode instead, the orthogonal transform coefficients for the Y component are used Figure 8J The quantization matrix is used for the orthogonal transform coefficients of the Cb component. Figure 8K The quantization matrix, and the orthogonal transform coefficients of the Cr component are used Figure 8L The quantization matrix of .
[0146] The prediction error reconstructed by inversely orthogonally transforming the orthogonal transform coefficients thus reconstructed is output to the image reconstruction unit 205. However, the quantization matrix used is not limited thereto, and may be the same as the quantization matrix used by the transform / quantization unit 105 and the inverse quantization / inverse transform unit 106 on the encoding side.
[0147] The image reconstruction unit 205 reconstructs the predicted image by appropriately referring to the frame memory 206 based on the prediction information input from the decoding unit 203. For the image reconstruction unit 205 of this embodiment, similar to the prediction unit 104 on the encoding side, intra-frame prediction and inter-frame prediction are used. Since the specific prediction processing is similar to the prediction unit 104 on the encoding side, its description will be omitted. The image reconstruction unit 205 reconstructs image data based on the predicted image and the prediction error input from the inverse quantization / inverse transformation unit 204, and stores the reconstructed image data in the frame memory 206. The stored image data is used for reference during prediction.
[0148] Similar to the loop filtering unit 109 on the encoding side, the loop filtering unit 207 reads out the reconstructed image from the frame memory 206 and performs loop filtering such as deblocking filtering, etc. Then, the loop filtering unit 207 stores the filtered image in the frame memory 206 again.
[0149] The reconstructed image stored in the frame memory 206 is finally output from the output terminal 208 to the outside (usually to a display device).
[0150] Figure 4 : is a flowchart showing the decoding process using the control unit 250 in the image decoding device according to the present embodiment.
[0151] First, in step S401, the control unit 250 controls the separation decoding unit 202 to separate the information related to the decoding process and the coded data related to the coefficient from the bit stream, and decodes the coded data of the header. More specifically, the separation decoding unit 202 supplies the coded data of the quantization matrix to the quantization matrix decoding unit 209, and supplies the coded data of the image to the decoding unit 203.
[0152] In step S402, the control unit 250 controls the quantization matrix decoding unit 209 to Fig.12 The syntax table decodes the quantization matrix coded data reconstructed in step S401, and reconstructs and maintains FIG. 8A to FIG. 8L The quantization matrices 801 to 811 are shown. The specific operation of the quantization matrix decoding unit 209 mentioned here has been explained and will be omitted.
[0153] In step S403 , the control unit 250 controls the decoding unit 203 to decode the encoded data separated in step S401 , reconstruct the prediction information, and reconstruct the residual coefficient.
[0154] In step S404, the control unit 250 controls the inverse quantization / inverse transform unit 204 to inverse quantize the residual coefficient using the quantization matrix reconstructed in step S402 and generate an orthogonal transform coefficient. The inverse quantization / inverse transform unit 204 further performs an inverse orthogonal transform and reconstructs the prediction error. In this embodiment, the quantization matrix used in the inverse quantization process is determined according to the characteristics of the sub-block to be decoded (such as color components and size, etc.). That is, the inverse quantization / inverse transform unit 204 uses the quantization matrix according to the characteristics of the sub-block (such as size, prediction mode, and color components, etc.). FIG. 8A to FIG. 8L However, the quantization matrix used is not limited to this and may be the same as the quantization matrix used on the encoding side.
[0155] In step S405, the control unit 250 controls the image reconstruction unit 205 to reconstruct an image based on the prediction information generated in step S403. Specifically, the image reconstruction unit 205 reconstructs the predicted image based on the prediction information, referring to the frame memory 206. At this time, similar to step S305 on the encoding side, the image reconstruction unit 205 uses intra-frame prediction and inter-frame prediction. Then, the image reconstruction unit 205 reconstructs image data based on the reconstructed predicted image and the prediction error generated in step S404, and stores the reconstructed image data in the frame memory 206.
[0156] In step S406, the control unit 250 determines whether the decoding of all basic blocks in the focus frame has been completed, and if the decoding is completed, it enters step S407, and if there is any basic block that has not been decoded, the processing returns to step S403 to set the next basic block as the decoding object.
[0157] In step S407 , the control unit 250 controls the loop filtering unit 207 to loop-filter the image data reconstructed in step S405 and generate a filtered image, and ends the process.
[0158] Due to the above-described configuration and operation, it is possible to decode the encoded bit stream generated by the above-described image encoding device (ie, the bit stream encoded in the appropriate quantization matrix encoding mode and with the amount of generated codes of the quantization matrix suppressed).
[0159] Note that in the present embodiment, only quantization matrices of the same size can be referenced in the quantization matrix reference mode and the inter-quantization matrix difference encoding mode, but a configuration may also be adopted in which quantization matrices of different sizes can be referenced using enlargement and reduction. Figure 8G When decoding the quantization matrix 806, the following configuration may also be adopted: refer to and amplify Figure 8HThe difference between the two quantization matrices is decoded as difference information, and the quantization matrix is reconstructed. Thus, a bit stream in which the data amount of the quantization matrix initially encoded in each size has been further reduced can also be decoded.
[0160] In addition, in this embodiment, Fig.12 As shown in the syntax table of , the configuration of decoding the quantization matrix in ascending order of size is adopted, but the configuration of decoding the quantization matrix in descending order of size may also be adopted. Thus, in particular in the case where reference between quantization matrices of different sizes can be made as described above, a bit stream in which the data amount of the quantization matrix has been further reduced can be decoded.
[0161] In addition, in the present embodiment, the reference object of the quantization matrix is limited to the encoded quantization matrix, but the following configuration may be adopted: the reference object is such as FIG. 13A to FIG. 13C The flat quantization matrix used as a reference when no quantization matrix is used is shown in the figure. Thus, particularly when the quantization matrix to be decoded is close to the flat quantization matrix, a bit stream in which the data amount of the quantization matrix is further reduced can be decoded.
[0162] Second embodiment
[0163] The processing units included in the image encoding device and the image decoding device of the first embodiment are described as being configured by hardware. However, the processing performed in each processing unit shown in the figure may be configured by a computer program.
[0164] Figure 5 : is a block diagram showing a configuration example of computer hardware applicable to the image encoding device and decoding device according to the above-described embodiments.
[0165] The CPU 501 uses the computer programs and data stored in the RAM 502 and the RAM 503 to perform overall control of the computer and execute the various processes described above as being performed by the image processing apparatus according to the above-described embodiment. That is, the CPU 501 functions as Figure 1 and Figure 2 The individual processing units are shown.
[0166] The RAM 502 has an area for temporarily storing data acquired from the outside via an external storage device 506 and an I / F (interface) 507. In addition, the RAM 502 is also used as a work area used when the CPU 501 performs various processes. For example, the RAM 502 can be allocated as a frame memory, or various other areas can be appropriately provided.
[0167] ROM 503 stores setting data and a boot program of the computer. Operation unit 504 is composed of a keyboard and a mouse, etc., and can be operated by a user of the computer to input various instructions to CPU 501. Display unit 505 displays the result of processing by CPU 501. Display unit 505 is composed of, for example, a liquid crystal display.
[0168] The external storage device 506 is a large-capacity information storage device represented by a hard disk drive device. The external storage device 506 stores information for the CPU 501 to implement Figure 1 and Figure 2 The functions of each unit shown are implemented by an OS (operating system) and a computer program (application program). In addition, each image data to be processed can be stored in the external storage device 506.
[0169] The computer program and data stored in the external storage device 506 are appropriately loaded into the RAM 502 according to the control of the CPU 501, and are processed by the CPU 501. A network such as a LAN and the Internet and other devices such as a projection device and a display device can be connected to the I / F 507, and the computer can acquire and send various information via the I / F 507. Reference numeral 508 denotes a bus that connects the above-mentioned units.
[0170] In the above configuration, when the device is powered on, the CPU 501 executes the boot program stored in the ROM 503, and loads the OS stored in the external storage device 506 to the RAM 502 and executes the loaded OS. Then, the CPU 501 loads the application program related to encoding or decoding from the external storage device 506 to the RAM 502 under the control of the OS, and executes the loaded application program. As a result, the CPU 501 functions as Figure 1 or Figure 2 The individual processing units in the device are used as an image encoding device or an image decoding device.
[0171] Here, we will refer to Fig.14 The flowchart of the second embodiment is used to illustrate the processing for encoding the quantization matrix, that is, Figure 3 The processing corresponding to step S302.
[0172] Note that the order of the encoded quantization matrices is the same as that of the first embodiment, and is initially in the order of quantization matrices 807→808→810→811 of 2×2 pixel size, followed by the order of quantization matrices 806→801→802→809→804→806 of 4×4 pixel size, and finally in the order of quantization matrices 800→803 of 8×8 pixel size.
[0173] Furthermore, the following variables sizeId and matrixId have the same meanings as those shown in the first embodiment.
[0174] In other words, the variable sizeId is a variable indicating the size of the quantization matrix, and when sizeId is "1", this indicates that the size of the quantization matrix to be encoded is 2×2 pixels. In addition, when sizeId is "2", this indicates that the size of the quantization matrix to be encoded is 4×4 pixels. In addition, when sizeId is "3", this indicates that the size of the quantization matrix to be encoded is 8×8 pixels. In addition, the variable matrixId indicates the order of quantization matrices of the same size. The quantization matrix to be encoded is identified by sizeId and matrixId.
[0175] For example, when sizeId=2 and matrixId=0, this indicates that the original quantization matrix 806 of 4×4 pixel size is the quantization matrix to be encoded.
[0176] First, in step S1401, the CPU 501 sets the size of the quantization matrix to be encoded to 2×2 pixels by setting the variable sizeId to “1” as an initial value. Next, in step S1402, the CPU 501 sets matrixId to “0” as an initial value. As a result, in the case where step S1402 is initially executed, Figure 8H The quantization matrix 807 is set as the quantization matrix of interest to be encoded.
[0177] In step S1403, the CPU 501 determines whether matrixId is "0". If matrixId is not "0", there is an encoded quantization matrix of the same size as the quantization matrix of interest. Therefore, the CPU 501 advances the process to step S1404 and performs a search process in the quantization matrix reference mode. In other words, the CPU 501 performs a search process as to whether there is an encoded quantization matrix that is the same as the quantization matrix of interest.
[0178] In step S1405, the CPU 501 determines whether an encoded quantization matrix identical to the quantization matrix of interest has been found. If it is determined that such a quantization matrix has been found, the CPU 501 causes the processing to enter step S1406, and outputs "0" and the value of the specified matching quantization matrix as the encoded data of the quantization matrix of interest. Setting "0" at the beginning of the encoded data is equivalent to setting the parameter scaling_list_pred_mode_idx to "0". In addition, regarding the "value of the specified matching quantization matrix", the number of quantization matrices of 4×4 pixel size "6" is greater than the number of quantization matrices of other sizes, so the number of bits allocated to specifying the value of the quantization matrix only needs to be "3". On the other hand, if it is determined that such a quantization matrix has not been found, the CPU 501 causes the processing to enter step S1407.
[0179] If it is determined in step S1403 that matrixId is "0", the CPU 501 advances the processing to step S1409.
[0180] In this step S1407, the CPU 501 performs the inter-quantization matrix differential encoding mode on the quantization matrix of interest. Then, in step S1408, the CPU 501 obtains "1" and the encoded data generated in the inter-quantization matrix differential encoding mode. Setting "1" at the beginning of the encoded data is equivalent to setting the parameter scaling_list_pred_mode_idx to "1". It is also possible to adopt a configuration in which information indicating the order of the quantization matrix that is the same as the quantization matrix of interest is output after the value "1" set at the beginning of the encoded data, and then the differential encoded data is output.
[0181] Next, in step S1409, the CPU 501 performs the intra-quantization matrix differential encoding mode on the quantization matrix of interest. Then, in step S1410, the CPU 501 obtains "2" and the encoded data generated in the inter-quantization matrix differential encoding mode. Setting "2" at the beginning of the encoded data is equivalent to setting the return value of scaling_list_pred_mode_idx to "2".
[0182] Then, in step S1411, the CPU 501 outputs the smaller of the encoded data generated in step S1408 and step S1410 as the encoded data of the quantization matrix of interest. Note that in the case of branching from step S1403 to step S1409, the encoded data generated in step S1409 in the inter-quantization matrix difference encoding mode is output.
[0183] In step S1412, the CPU 501 increases the variable matrixId by "1". Then, in step S1413, the CPU 501 determines whether all quantization matrices of the same size as the quantization matrix of interest have been encoded based on the variables sizeId and matrixId. If the determination result of step S1413 is "No", the CPU 501 advances the processing to step S1403 and repeats the above processing.
[0184] Furthermore, if the judgment result of step S1413 is "yes", the CPU 501 increases sizeId by "1". Then, in step S1415, the CPU 501 judges whether the quantization matrices of all sizes have been encoded based on the variables sizeId and matrixId. If the judgment result of step S1415 is "no", there are quantization matrices of sizes that have not been encoded, so the CPU returns the process to step S1402. Furthermore, if the judgment result of step S1415 is "yes", the process ends ( Figure 3 Step S302).
[0185] Note that in the case where the processing proceeds to step S1406, it is ensured that the encoded data generated in the quantization matrix reference mode will be reduced, so comparison with encoded data generated in other modes is omitted, but such comparison may of course be performed.
[0186] Other embodiments
[0187] The present invention can be implemented by the following process: a program for implementing one or more functions of the above-mentioned embodiment is supplied to a system or device via a network or storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be implemented by a circuit (e.g., ASIC) for implementing one or more functions.
[0188] Industrial Applicability
[0189] The present invention is used in an encoding device / decoding device for encoding / decoding a still image / moving image. In particular, the present invention is applicable to an encoding method and a decoding method using a quantization matrix.
[0190] The present invention is not limited to the above-described embodiments, and various changes and modifications may be made without departing from the spirit and scope of the present invention. Therefore, the following claims are attached to disclose the scope of the present invention.
[0191] This application claims priority from Japanese Patent Application No. 2019-170810 filed on September 19, 2019, and the entire contents of which are incorporated herein by reference.
Claims
1. An image encoding device, comprising: a quantization unit configured to quantize transform coefficients in a block within the image using a first quantization matrix; as well as an encoding unit configured to encode the first quantization matrix, in, (a) in a case where the first quantization matrix is encoded using the first mode, the encoding unit encodes information indicating that the first quantization matrix is the same as a second quantization matrix to be referenced into a scaling list data syntax structure, and encodes information for specifying the second quantization matrix into the scaling list data syntax structure, (b) when the first quantization matrix is encoded using the second mode, the encoding unit encodes information for specifying the second quantization matrix into the scaling list data syntax structure, and encodes information for obtaining differences between elements of the first quantization matrix and elements of the second quantization matrix into the scaling list data syntax structure, and (c) when the first quantization matrix is encoded using the third mode, the encoding unit encodes a difference between an element in the first quantization matrix and another element, wherein the amount of code indicating the use of the first mode is smaller than the amount of code indicating the use of the second mode, wherein the amount of code indicating the use of the first mode is not greater than the amount of code indicating the use of the third mode, The size of the second quantization matrix can only be 2×2, 4×4, or 8×8, and In which, when the size of the first quantization matrix to be encoded is 8×8, the size of the second quantization matrix is 8×8.
2. The image encoding device according to claim 1, wherein: Types of components of the block include a luminance component and two chrominance components, and a transform coefficient of each component is quantized using a first quantization matrix corresponding to the component.
3. The image encoding device according to claim 1, in, The second quantization matrix is an already encoded quantization matrix.
4. The image encoding device according to claim 1, in, A default quantization matrix, in which the values of all elements of each element are 16, can be used as the second quantization matrix.
5. An image decoding device, comprising: A decoding unit configured to decode the first quantization matrix; as well as an inverse quantization unit, configured to inverse quantize the residual coefficients using the first quantization matrix, The decoding unit decodes first information indicating a mode for decoding the first quantization matrix from a scaling list data syntax structure in a bit stream, in, (a) in a case where the first information decoded by the decoding unit indicates that a first mode is used, the decoding unit decodes second information for specifying a second quantization matrix to be referenced from the scaling list data syntax structure, and decodes the first quantization matrix by determining that the first quantization matrix is the same as the second quantization matrix specified based on the second information, (b) in a case where the first information decoded by the decoding unit indicates that the second mode is used, the decoding unit decodes the second information for specifying the second quantization matrix from the scaling list data syntax structure, and decodes the first quantization matrix by using a difference between an element of the first quantization matrix and an element of the second quantization matrix specified based on the second information, and (c) in a case where the first information decoded by the decoding unit indicates that the third mode is used, the decoding unit decodes the first quantization matrix by using a difference between an element in the first quantization matrix and another element, wherein the amount of code indicating that the first information uses the first mode is smaller than the amount of code indicating that the first information uses the second mode, wherein the amount of code indicating that the first information uses the first mode is not greater than the amount of code indicating that the first information uses the third mode, The size of the second quantization matrix can only be 2×2, 4×4, or 8×8, and Wherein, when the size of the first quantization matrix to be decoded is 8×8, the size of the second quantization matrix is 8×8.
6. The image decoding device according to claim 5, in, The second quantization matrix is a quantization matrix that has been decoded.
7. The image decoding device according to claim 5, in, A default quantization matrix, in which the values of all elements of each element are 16, can be used as the second quantization matrix.
8. A coding method comprising: quantizing transform coefficients in a block within the image using a first quantization matrix; as well as encoding the first quantization matrix, in, (a) in a case where the first quantization matrix is encoded using a first mode, the encoding encodes information indicating that the first quantization matrix is the same as a second quantization matrix to be referenced into a scaling list data syntax structure, and encodes information for specifying the second quantization matrix into the scaling list data syntax structure, (b) when the first quantization matrix is encoded using a second mode, the encoding encodes information for specifying the second quantization matrix into the scaling list data syntax structure, and encodes information for obtaining differences between elements of the first quantization matrix and elements of the second quantization matrix into the scaling list data syntax structure, and (c) when the first quantization matrix is encoded using the third mode, the encoding encodes a difference between an element in the first quantization matrix and another element, wherein the amount of code indicating the use of the first mode is smaller than the amount of code indicating the use of the second mode, wherein the amount of code indicating the use of the first mode is not greater than the amount of code indicating the use of the third mode, The size of the second quantization matrix can only be 2×2, 4×4, or 8×8, and In which, when the size of the first quantization matrix to be encoded is 8×8, the size of the second quantization matrix is 8×8.
9. A decoding method, comprising: Decoding a first quantization matrix; as well as Inverse quantization is performed on the residual coefficients using the first quantization matrix, wherein the decoding decodes first information indicating a mode for decoding the first quantization matrix from a scaling list data syntax structure in a bitstream, in, (a) in the case where the first information decoded in the decoding indicates that the first mode is used, the decoding decodes second information for specifying a second quantization matrix to be referenced from the scaling list data syntax structure, and decodes the first quantization matrix by determining that the first quantization matrix is the same as the second quantization matrix specified based on the second information, (b) in the case where the first information decoded in the decoding indicates that the second mode is used, the decoding decodes the second information for specifying the second quantization matrix from the scaling list data syntax structure, and decodes the first quantization matrix by using the difference between the elements of the first quantization matrix and the elements of the second quantization matrix specified based on the second information, and (c) in the decoding, when the first information decoded indicates that the third mode is used, the decoding decodes the first quantization matrix by using a difference between an element in the first quantization matrix and another element, wherein the amount of code indicating that the first information uses the first mode is smaller than the amount of code indicating that the first information uses the second mode, wherein the amount of code indicating that the first information uses the first mode is not greater than the amount of code indicating that the first information uses the third mode, The size of the second quantization matrix can only be 2×2, 4×4, or 8×8, and Wherein, when the size of the first quantization matrix to be decoded is 8×8, the size of the second quantization matrix is 8×8.
Citation Information
Patent Citations
Image encoder, image encoding method, program, image decoder, image decoding method and program
JP2013038758A
Game system and program
JP2019170810A
