Image encoding device, image decoding device, image encoding method, and image decoding method
By using prediction components and quantization matrix to process mixed intra-inter prediction images in image encoding, the problem that quantization matrix in VVC cannot support mixed intra-inter prediction is solved, and the mixed intra-inter prediction is realized Effective quantitative control of errors improves image quality.
Patent Information
- Application Number
- CN202280093799.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-22
- Filing Date
- 2022-12-21
- Publication Date
- 2024-11-12
AI Technical Summary
The quantization matrix in VVC cannot support mixed intra-inter prediction, resulting in the intra-inter prediction that the frequency component cannot be quantized and image quality cannot be improved.
The prediction component is used to generate a prediction image, and partial areas of the image are processed through intra prediction and inter prediction, prediction errors are generated, and frequency transformation is performed through the transformation component, and the transformation coefficient is quantized using a quantization matrix to encode the quantization coefficient.
It provides a quantization matrix suitable for mixed intra-inter prediction, which can effectively control quantization errors and improve image quality.
Smart Images

Figure CN118947117A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to image encoding technology and image decoding technology. Background Art
[0002] As a coding method used for compressed recording of moving images, a VVC (Versatile Video Coding) coding method (hereinafter referred to as VVC) is known. In VVC, in order to improve coding efficiency, a basic block including a maximum of 128×128 pixels is divided into sub-blocks having not only a conventional square shape but also a rectangular shape.
[0003] In addition, in VVC, the following matrix is used, which is called a quantization matrix and is configured to weight the coefficients after orthogonal transformation (hereinafter referred to as orthogonal transformation coefficients) according to the frequency components. The data of high-frequency components whose degradation is not obvious to human vision is reduced, thereby increasing the compression efficiency while maintaining the image quality. Patent document 1 discloses a technology for encoding such a quantization matrix.
[0004] In recent years, in JVET (Joint Video Experts Group) obtained by standardizing VVC, a technology for achieving higher compression efficiency than VVC has been studied. In order to improve coding efficiency, in addition to conventional intra-prediction and inter-prediction, a new prediction method in which intra-prediction pixels and inter-prediction pixels are mixed in the same sub-block (hereinafter referred to as hybrid intra-inter prediction) has been studied.
[0005] Prior art literature
[0006] Patent Literature
[0007] Patent Document 1: Japanese Patent Application Laid-Open No. 2013-38758 Summary of the invention
[0008] Problem that the invention aims to solve
[0009] The quantization matrix in VVC assumes a prediction method such as conventional intra prediction or inter prediction, and cannot support hybrid intra-inter prediction as a new prediction method. For this reason, quantization control according to frequency components cannot be performed for errors in hybrid intra-inter prediction, and image quality cannot be improved. The present invention provides a technique for quantizing using an appropriate quantization matrix for hybrid intra-inter prediction.
[0010] Solutions for solving problems
[0011] One aspect of the present invention includes: a prediction unit for generating a predicted image and deriving a prediction error, in which an intra-frame prediction image obtained by intra-frame prediction is applied to a partial area in a coding object block included in an image, and an inter-frame prediction image obtained by inter-frame prediction is applied to other areas in the coding object block that are different from the partial area, and each prediction error in the prediction error is a difference between the coding object block and the generated predicted image; a transform unit for deriving a transform coefficient by frequency transforming the prediction error derived by the prediction unit; a quantization unit for deriving a quantization coefficient by quantizing the transform coefficient derived by the transform unit using a quantization matrix; and a first encoding unit for encoding the quantization coefficient derived by the quantization unit.
[0012] Effects of the Invention
[0013] According to the present invention, a technique for quantizing using an appropriate quantization matrix for hybrid intra-inter prediction may be provided.
[0014] Other features and advantages of the present invention will be apparent from the following description in conjunction with the accompanying drawings. Note that throughout the accompanying drawings, the same reference numerals represent the same or similar components. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.
[0016] Figure 1 is a block diagram showing an example of a functional structure of an image encoding device;
[0017] Figure 2 is a block diagram showing an example of a functional structure of an image decoding device;
[0018] Figure 3 is a flow chart of an encoding process using an image encoding device;
[0019] Figure 4 is a flowchart of a decoding process using an image decoding device;
[0020] Figure 5 is a block diagram showing an example of a hardware structure of a computer device;
[0021] Fig. 6A is a diagram showing an example of the structure of a bit stream;
[0022] Figure 6B is a diagram showing an example of the structure of a bit stream;
[0023] Fig. 7Ais a diagram showing an example of a method of dividing a basic block 700 into sub-blocks;
[0024] Figure 7B is a diagram showing an example of a method of dividing a basic block 700 into sub-blocks;
[0025] Figure 7C is a diagram showing an example of a method of dividing a basic block 700 into sub-blocks;
[0026] Fig.7D is a diagram showing an example of a method of dividing a basic block 700 into sub-blocks;
[0027] Fig. 7E is a diagram showing an example of a method of dividing a basic block 700 into sub-blocks;
[0028] Figure 7F is a diagram showing an example of a method of dividing a basic block 700 into sub-blocks;
[0029] Fig. 8A is a diagram showing an example of a quantization matrix;
[0030] Figure 8B is a diagram showing an example of a quantization matrix;
[0031] Figure 8C is a diagram showing an example of a quantization matrix;
[0032] Fig. 9 is a diagram showing a reference order of values of elements in a quantization matrix;
[0033] Fig. 10A is a diagram showing an example of a one-dimensional array;
[0034] Fig. 10B is a diagram showing an example of a one-dimensional array;
[0035] Fig. 10C is a diagram showing an example of a one-dimensional array;
[0036] Fig.11A is a diagram showing an example of a coding table;
[0037] Fig. 11B is a diagram showing an example of a coding table;
[0038] Fig. 12A is a diagram for explaining hybrid intra-inter prediction;
[0039] Fig. 12B is a diagram for explaining hybrid intra-inter prediction;
[0040] Fig. 12C is a diagram for explaining hybrid intra-inter prediction;
[0041] Fig.12D is a diagram for explaining hybrid intra-inter prediction;
[0042] Fig.12E is a diagram for explaining hybrid intra-inter prediction;
[0043] Fig.12F is a diagram for explaining hybrid intra-inter prediction;
[0044] Figure 12G is a diagram for explaining hybrid intra-inter prediction; and
[0045] Fig.12H It is a diagram for explaining hybrid intra-inter prediction. DETAILED DESCRIPTION
[0046] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. Multiple features are described in the embodiments, but are not limited to inventions requiring all such features, and multiple such features may be appropriately combined. In addition, in the accompanying drawings, the same reference numerals are given to the same or similar structures, and redundant descriptions thereof are omitted.
[0047] [First embodiment]
[0048] The image encoding device according to the present embodiment obtains a predicted image by applying an intra-frame prediction image obtained by intra-frame prediction to a partial area of a coding object block included in an image, and applying an inter-frame prediction image obtained by inter-frame prediction to other areas different from the partial area of the block. The image encoding device encodes a quantization coefficient obtained by quantizing an orthogonal transformation coefficient of a difference between the block and the predicted image using a quantization matrix (first encoding).
[0049] First, refer to Figure 1 , to illustrate an example of the functional structure of the image encoding device according to the present embodiment. The control unit 150 controls the operation of the entire image encoding device. The segmentation unit 102 segments the input image into a plurality of basic blocks and outputs each basic block obtained by the segmentation. Note that the input image may be an image of each frame of a moving image (for example, an image of each frame of a moving image at 30 frames per second), or may be a still image captured regularly or irregularly. In addition, the segmentation unit 102 can obtain the input image from any device. For example, the input image can be obtained from a camera device such as a video camera, can be obtained from a device that holds a plurality of images, or can be obtained from a memory accessible to the self-device.
[0050] The holding unit 103 holds a quantization matrix corresponding to each prediction process in a plurality of prediction processes. In the present embodiment, the holding unit 103 holds a quantization matrix corresponding to an intra-frame prediction as an intra-frame prediction, a quantization matrix corresponding to an inter-frame prediction as an inter-frame prediction, and a quantization matrix corresponding to the above-mentioned hybrid intra-inter prediction. Note that each quantization matrix held by the holding unit 103 may be a quantization matrix having a default element value, or may be a quantization matrix generated by the control unit 150 according to a user operation. Alternatively, each quantization matrix held by the holding unit 103 may be a quantization matrix generated by the control unit 150 according to characteristics of an input image (such as the amount or frequency of edges included in the input image).
[0051] The prediction unit 104 divides the basic block into a plurality of sub-blocks in units of the basic block. The prediction unit 104 obtains a predicted image for each sub-block by one of intra prediction, inter prediction, and hybrid intra-inter prediction, and obtains the difference between the sub-block and the predicted image as a prediction error. In addition, the prediction unit 104 generates information required for prediction (such as information indicating a basic block division method, a prediction mode indicating prediction for obtaining a predicted image for each sub-block, and a motion vector, etc.) as prediction information.
[0052] The transform / quantization unit 105 generates transform coefficients for each sub-block by performing an orthogonal transform (frequency transform) on the prediction errors of each sub-block obtained by the prediction unit 104, obtains from the holding unit 103 a quantization matrix corresponding to the prediction (intra-frame prediction, inter-frame prediction, or mixed intra-frame-inter-frame prediction) performed by the prediction unit 104 to obtain a predicted image of the sub-block, and uses the obtained quantization matrix to quantize the transform coefficients, thereby generating quantization coefficients for the sub-blocks (quantization results of the transform coefficients).
[0053] The inverse quantization / inverse transform unit 106 uses the quantization matrix used by the transform / quantization unit 105 to generate quantization coefficients to inverse quantize the quantization coefficients of each sub-block generated by the transform / quantization unit 105, thereby generating transform coefficients, and performs an inverse orthogonal transform on the transform coefficients, thereby generating (reproducing) prediction errors.
[0054] The image reproduction unit 107 generates a predicted image from the image stored in the frame memory 108 based on the prediction information generated by the prediction unit 104, and reproduces an image from the predicted image and the prediction error generated by the inverse quantization / inverse transform unit 106. Then, the image reproduction unit 107 stores the reproduced image in the frame memory 108. The image stored in the frame memory 108 is an image that is referenced when the prediction unit 104 predicts an image of a current frame or a next frame.
[0055] The loop filtering unit 109 performs loop filtering processing such as deblocking filtering or sample adaptive offset on the image stored in the frame memory 108 .
[0056] The encoding unit 110 encodes the quantization coefficient generated by the transform / quantization unit 105 and the prediction information generated by the prediction unit 104, thereby generating encoded data (code data).
[0057] The encoding unit 113 encodes the quantization matrix (including at least the quantization matrix used for quantization by the transform / quantization unit 105 ) held by the holding unit 103 , thereby generating encoded data (code data).
[0058] The integrated encoding unit 111 generates header code data using the encoded data generated by the encoding unit 113 , generates a bit stream including the encoded data generated by the encoding unit 110 and the header code data, and outputs the bit stream.
[0059] Note that the output destination of the bit stream is not limited to a specific output destination. For example, the bit stream may be output to a memory provided in the image encoding device, may be output to an external device via a network to which the image encoding device is connected, or may be sent to the outside for broadcasting.
[0060] Next, the operation of the image encoding device according to the present embodiment will be described. First, encoding of an input image will be described. The division unit 102 divides the input image into a plurality of basic blocks and outputs each basic block obtained by the division.
[0061] The prediction unit 104 divides a basic block into a plurality of sub-blocks in units of a basic block. 7A to 7F An example of a method of dividing the basic block 700 into sub-blocks is shown.
[0062] Fig. 7A A basic block 700 of 8 pixels×8 pixels (=sub-block) is shown which is not divided into sub-blocks. Figure 7B An example of conventional square sub-block partitioning is shown, and a basic block 700 of 8 pixels×8 pixels is partitioned into four sub-blocks each including 4 pixels×4 pixels (quadtree partitioning).
[0063] FIG. 7C to FIG. 7F An example of the type of rectangular sub-block partitioning is shown. Figure 7C In FIG. 7 , a basic block 700 of 8 pixels×8 pixels is divided into two sub-blocks each including 4 pixels (horizontally)×8 pixels (vertically) (binary tree division). Fig.7D , a basic block 700 of 8 pixels×8 pixels is divided into two sub-blocks each including 8 pixels (horizontal direction)×4 pixels (vertical direction) (binary tree division).
[0064] exist Fig. 7E In FIG. 7 , the basic block 700 of 8 pixels×8 pixels is divided into sub-blocks of 2 pixels (horizontally)×8 pixels (vertically), sub-blocks of 4 pixels (horizontally)×8 pixels (vertically), and sub-blocks of 2 pixels (horizontally)×8 pixels (vertically). That is, in Fig. 7E , the basic block 700 is partitioned into sub-blocks in a ratio of 1:2:1 with respect to the width (length in the horizontal direction) (ternary tree partitioning).
[0065] exist Figure 7F In FIG. 7 , the basic block 700 of 8 pixels×8 pixels is divided into sub-blocks of 8 pixels (horizontally)×2 pixels (vertically), sub-blocks of 8 pixels (horizontally)×4 pixels (vertically), and sub-blocks of 8 pixels (horizontally)×2 pixels (vertically). That is, in Figure 7F , the basic block 700 is partitioned into sub-blocks (ternary tree partitioning) in a ratio of 1:2:1 with respect to height (length in the vertical direction).
[0066] As described above, in the present embodiment, not only square sub-blocks but also rectangular sub-blocks are used for encoding processing. In the present embodiment, prediction information including information indicating a basic block division method is generated. Note that the division method shown in FIG. 7 is merely an example, and the method of dividing a basic block into sub-blocks is not limited to the division method shown in FIG. 7.
[0067] The prediction unit 104 determines the prediction (prediction mode) to be performed for each sub-block. For each sub-block, the prediction unit 104 generates a predicted image based on the prediction mode determined for the sub-block and the coded pixels, and obtains the difference between the sub-block and the predicted image as a prediction error. In addition, the prediction unit 104 generates "information required for prediction" (such as information indicating the basic block division method, the prediction mode of each sub-block, and the motion vector, etc.) as prediction information.
[0068] Here, the prediction used in the present embodiment will be re-explained. In the present embodiment, three types of prediction (prediction modes) including intra prediction, inter prediction, and hybrid intra-inter prediction are used.
[0069] In intra prediction, predicted pixels of the encoding object block are generated using already encoded pixels spatially located around the encoding object block (subblock in this embodiment). In other words, in intra prediction, predicted pixels of the encoding object block are generated using already encoded pixels in a frame including the encoding object block. For a subblock subjected to intra prediction, information indicating an intra prediction method such as horizontal prediction, vertical prediction, or DC prediction is generated as "information required for prediction".
[0070] In inter-frame prediction, predicted pixels of the encoding target block (sub-block in this embodiment) are generated using already encoded pixels in a frame different (in time) from the frame to which the encoding target block (sub-block in this embodiment) belongs. For the sub-block subjected to inter-frame prediction, motion information indicating a frame to be referenced or a motion vector is generated as "information required for prediction".
[0071] In hybrid intra-inter prediction, first, the encoding object block (in this embodiment, the sub-block) is divided by line segments in an oblique direction, thereby dividing the encoding object block into two divided areas. As the predicted pixels of one divided area, the "predicted pixels obtained for the one divided area by intra prediction for the encoding object block" are obtained. In addition, as the predicted pixels of the other divided areas, the "predicted pixels obtained for the other divided areas by inter prediction for the encoding object block" are obtained. That is, the predicted pixels of one divided area of the predicted image obtained by hybrid intra-inter prediction for the encoding object block are "predicted pixels obtained for the one divided area by intra prediction for the encoding object block". In addition, the predicted pixels of the other divided areas of the predicted image obtained by hybrid intra-inter prediction for the encoding object block are "predicted pixels obtained for the other divided areas by inter prediction for the encoding object block". Figure 12 shows an example of the division of the encoding object block in hybrid intra-inter prediction.
[0072] like Fig. 12A As shown, it is assumed that the encoding object block 1200 is divided by a line segment passing through the upper left vertex and the lower right vertex of the encoding object block 1200, so that the encoding object block 1200 is divided into a divided area 1200a and a divided area 1200b. FIG. 12A to FIG. 12D In this case, the prediction unit 104 performs mixed intra-frame and inter-frame prediction processing on the encoding target block 1200. At this time, the prediction unit 104 performs intra-frame prediction on the encoding target block 1200, thereby generating an intra-frame prediction image 1201 ( Fig. 12BHere, the intra-frame prediction image 1201 includes a region 1201a located at the same position as the segmented region 1200a and a region 1201b located at the same position as the segmented region 1200b. The prediction unit 104 obtains, as "prediction pixels of the segmented region 1200a", the intra-frame prediction pixels included in the intra-frame prediction image 1201, which belong to the region 1201a located at the same position as the segmented region 1200a. In addition, the prediction unit 104 performs inter-frame prediction on the encoding target block 1200, thereby generating an inter-frame prediction image 1202 ( Fig. 12C ). Here, the inter-frame prediction image 1202 includes a region 1202a located at the same position as the segmented region 1200a and a region 1202b located at the same position as the segmented region 1200b. The prediction unit 104 obtains, from among the pixels (inter-frame prediction pixels) included in the inter-frame prediction image 1202, the inter-frame prediction pixels belonging to the region 1202b located at the same position as the segmented region 1200b as "prediction pixels of the segmented region 1200b". Then, the prediction unit 104 generates a prediction image 1203 ( Fig.12D ).
[0073] In the above manner, the prediction unit 104 generates the intra-frame prediction image 1201 ( Fig. 12B ), and further generates an inter-frame prediction image 1202 ( Fig. 12C ). Then, the prediction unit 104 arranges the intra-frame prediction pixels at the positions of the coordinates (x, y) included in the region 1201a corresponding to the segmented region 1200a, among the intra-frame prediction pixels in the intra-frame prediction image 1201, at the same coordinates (x, y) of the prediction image 1203. In addition, the prediction unit 104 arranges the inter-frame prediction pixels at the positions of the coordinates (x, y) included in the region 1202b corresponding to the segmented region 1200b, among the inter-frame prediction pixels in the inter-frame prediction image 1202, at the same coordinates (x, y) of the prediction image 1203. Thus, Fig.12D The predicted image 1203 is shown.
[0074] In addition, here, we will refer to FIG. 12E to FIG. 12H The processing of the mixed intra-frame-inter-frame prediction by the prediction unit 104 for the encoding target block 1200 is further described. Fig.12EAs shown, it is assumed that the encoding target block 1200 is divided into a segmented area 1200c and a segmented area 1200d by a line segment passing through the midpoint between the upper left corner vertex and the lower left corner vertex of the encoding target block 1200. At this time, the prediction unit 104 performs intra-frame prediction for the encoding target block 1200, thereby generating an intra-frame prediction image 1201 ( Fig.12F Here, the intra-frame prediction image 1201 includes a region 1201c located at the same position as the segmented region 1200c, and a region 1201d located at the same position as the segmented region 1200d. The prediction unit 104 obtains, from among the pixels (intra-frame prediction pixels) included in the intra-frame prediction image 1201, the intra-frame prediction pixels belonging to the region 1201c located at the same position as the segmented region 1200c as "prediction pixels of the segmented region 1200c". In addition, the prediction unit 104 performs inter-frame prediction on the encoding target block 1200, thereby generating an inter-frame prediction image 1202 ( Figure 12G ). Here, the inter-frame prediction image 1202 includes a region 1202c located at the same position as the segmented region 1200c, and a region 1202d located at the same position as the segmented region 1200d. The prediction unit 104 obtains, from among the pixels (inter-frame prediction pixels) included in the inter-frame prediction image 1202, the inter-frame prediction pixels belonging to the region 1202d located at the same position as the segmented region 1200d as "prediction pixels of the segmented region 1200d". Then, the prediction unit 104 generates a prediction image 1203 ( Fig.12H ).
[0075] In the above manner, the prediction unit 104 generates the intra-frame prediction image 1201 ( Fig.12F ), and further generates an inter-frame prediction image 1202 ( Figure 12G ). Then, the prediction unit 104 arranges the intra-frame prediction pixels at the position of the coordinates (x, y) included in the region 1201c corresponding to the divided region 1200c among the intra-frame prediction pixels in the intra-frame prediction image 1201 at the same coordinates (x, y) of the prediction image 1203. In addition, the prediction unit 104 arranges the inter-frame prediction pixels at the position of the coordinates (x, y) included in the region 1202d corresponding to the divided region 1200d among the inter-frame prediction pixels in the inter-frame prediction image 1202 at the same coordinates (x, y) of the prediction image 1203. Thus, Fig.12D The predicted image 1203 is shown.
[0076] For a sub-block that has undergone hybrid intra-inter prediction, information indicating the intra prediction method, motion information indicating a frame to be referenced or a motion vector, and information defining a segmented area (for example, information defining the above-mentioned line segment) are generated as "information required for prediction".
[0077] The prediction unit 104 decides the prediction mode of the subblock of interest by the following processing. The prediction unit 104 generates a difference image between the subblock of interest and the predicted image generated by intra prediction for the subblock of interest. In addition, the prediction unit 104 generates a difference image between the subblock of interest and the predicted image generated by inter prediction for the subblock of interest. In addition, the prediction unit 104 generates a difference image between the subblock of interest and the predicted image generated by hybrid intra-inter prediction for the subblock of interest. Note that the pixel value at the pixel position (x, y) in the difference image C between the image A and the image B is the difference between the pixel value AA at the pixel position (x, y) in the image A and the pixel value BB at the pixel position (x, y) in the image B (such as the absolute value of the difference between AA and BB or the square value of the difference between AA and BB). The prediction unit 104 specifies the predicted image in which the sum of the pixel values of all pixels in the difference image is the smallest, and decides the prediction performed for the subblock of interest to obtain the predicted image as the "prediction mode of the subblock of interest". Note that the method of deciding the prediction mode of the focused subblock is not limited to the above method.
[0078] Then, the prediction unit 104 obtains, for each sub-block, a predicted image generated by the prediction mode determined for the sub-block as a "prediction image of the sub-block", and generates a prediction error from the sub-block and the predicted image. In addition, the prediction unit 104 generates prediction information for each sub-block, the prediction information including the prediction mode determined for the sub-block and the "information required for prediction" generated for the sub-block.
[0079] The transform / quantization unit 105 performs an orthogonal transform process on the prediction error of the sub-block corresponding to the size of the prediction error for each sub-block, thereby generating an orthogonal transform coefficient. Then, the transform / quantization unit 105 obtains a quantization matrix corresponding to the prediction mode of the sub-block from the quantization matrix held by the holding unit 103 for each sub-block, and quantizes the orthogonal transform coefficient of the sub-block using the obtained quantization matrix, thereby generating a quantization coefficient.
[0080] For example, assuming that the holding unit 103 holds Fig. 8A, 8×8 elements (the values of all 64 elements are quantization step values) as a quantization matrix to be used for quantizing the orthogonal transform coefficients of a sub-block having 8 pixels×8 pixels when intra prediction is performed on the sub-block. In addition, for example, it is assumed that the holding unit 103 holds Figure 8B 8×8 elements (the values of all 64 elements are quantization step values) as a quantization matrix to be used for quantizing the orthogonal transform coefficients of a sub-block having 8 pixels×8 pixels when inter-frame prediction is performed on the sub-block. In addition, for example, it is assumed that the holding unit 103 holds Figure 8C The quantization matrix with 8 elements × 8 elements (the values of all 64 elements are quantization step values) illustrated in the figure is used as the quantization matrix to be used for quantizing the orthogonal transform coefficients of a sub-block with 8 pixels × 8 pixels when mixed intra-frame-inter-frame prediction is performed on the sub-block.
[0081] In this case, the transform / quantization unit 105 uses Fig. 8A The quantization matrix used for intra prediction shown is used to quantize the orthogonal transform coefficient of "the prediction error obtained by intra prediction for the sub-block of 8 pixels×8 pixels".
[0082] In addition, the transform / quantization unit 105 uses Figure 8B The quantization matrix used for inter-frame prediction shown in FIG. 1 quantizes the orthogonal transform coefficient of “the prediction error obtained by the inter-frame prediction for the sub-block of 8 pixels×8 pixels”.
[0083] In addition, the transform / quantization unit 105 uses Figure 8C The quantization matrix used for the hybrid intra-inter prediction shown is used to quantize the orthogonal transform coefficients of the prediction error obtained by the hybrid intra-inter prediction for the sub-block of 8 pixels×8 pixels.
[0084] The inverse quantization / inverse transform unit 106 uses the quantization matrix used by the transform / quantization unit 105 to quantize the sub-blocks to inverse quantize the quantization coefficients of each sub-block generated by the transform / quantization unit 105, thereby generating transform coefficients, and performs an inverse orthogonal transform on the transform coefficients, thereby generating (reproducing) prediction errors.
[0085] The image reproduction unit 107 generates a predicted image from the image stored in the frame memory 108 based on the prediction information generated by the prediction unit 104, and reproduces the image of the subblock by adding the predicted image and the prediction error generated (reproduced) by the inverse quantization / inverse transform unit 106. Then, the image reproduction unit 107 stores the reproduced image in the frame memory 108.
[0086] The loop filtering unit 109 performs a loop filtering process such as a deblocking filter or a sample adaptive offset on the image stored in the frame memory 108 , and stores the image subjected to the loop filtering process in the frame memory 108 .
[0087] The encoding unit 110 performs entropy encoding of the quantization coefficient of the sub-block generated by the transform / quantization unit 105 and the prediction information of the sub-block generated by the prediction unit 104 for each sub-block, thereby generating encoded data. Note that the entropy encoding method is not particularly specified, and Golomb encoding, arithmetic encoding, Huffman encoding, or the like may be used.
[0088] Next, the encoding of the quantization matrix will be described. The quantization matrix held by the holding unit 103 is generated according to the size of the sub-block to be encoded or the prediction mode. For example, as shown in FIG. 7 , in the case where a size such as 8 pixels × 8 pixels, 4 pixels × 4 pixels, 8 pixels × 4 pixels, 4 pixels × 8 pixels, 8 pixels × 2 pixels, or 2 pixels × 8 pixels is adopted as the size of the sub-block obtained by segmentation, the quantization matrix of the adopted size is registered in the holding unit 103. The quantization matrix is prepared for each of the intra prediction, the inter prediction, and the mixed intra-inter prediction, and is registered in the holding unit 103.
[0089] The method of generating the quantization matrix according to the size of the subblock or the prediction mode is not limited to the specific generation method as described above, and the method of managing the quantization matrix in the holding unit 103 is not limited to the specific management method.
[0090] In the present embodiment, as shown in FIG8 , the quantization matrix held by the holding unit 103 is held in a two-dimensional shape, but the elements in the quantization matrix are not limited to these. In addition, depending on the size of the sub-block or whether the encoding object is a luminance block or a color difference block, a plurality of quantization matrices may be held corresponding to the same prediction method. Generally, since the quantization matrix implements quantization processing according to human visual characteristics, as shown in FIG8 , the elements used for the DC component corresponding to the upper left corner portion of the quantization matrix are small, and the elements used for the AC component corresponding to the lower right portion are large.
[0091] The encoding unit 113 reads out the quantization matrix (including at least the quantization matrix used for quantization by the transform / quantization unit 105) held by the holding unit 103, and encodes the read quantization matrix. For example, the encoding unit 113 encodes the quantization matrix of interest by the following process.
[0092] The encoding unit 113 refers to the values of the elements in the quantization matrix of interest as a two-dimensional array in a predetermined order, and generates a one-dimensional array in which the difference values between the values of the currently referenced elements and the values of the immediately previously referenced elements are arranged. For example, if Figure 8CThe quantization matrix shown is the focus quantization matrix, then Fig. 9 As shown, the encoding unit 113 refers to the values of the elements from the value of the element at the upper left corner of the quantization matrix of interest to the value of the element at the lower right corner in the order indicated by the arrows.
[0093] In this case, since the value of the first referenced element is "8" and there is no value of the immediately previously referenced element, the encoding unit 113 outputs a predetermined value or a value obtained by a certain method as an output value. For example, the encoding unit 113 may output the value "8" of the currently referenced element as an output value, or output a value obtained by subtracting a predetermined value from the value "8" of the element as an output value, and the output value may not be a value determined by a specific method.
[0094] Since the value of the next referenced element is "11" and the value of the immediately preceding referenced element is "8", the encoding unit 113 outputs a differential value "+3" obtained by subtracting the value "8" of the immediately preceding referenced element from the value "11" of the currently referenced element as an output value. In this way, the encoding unit 113 refers to the values of the elements in the quantization matrix in a predetermined order, obtains the output value and outputs the output value, and generates a one-dimensional array in which the output values are arranged in the output order.
[0095] Fig. 10A It is shown that through this process Fig. 8A The quantization matrix shown generates a one-dimensional array. Fig. 10B It is shown that through this process Figure 8B The quantization matrix shown generates a one-dimensional array. Fig. 10C It is shown that through this process Figure 8C The quantization matrix shown generates a one-dimensional array. In FIG10, "8" is set as the predetermined value.
[0096] Then, the encoding unit 113 encodes the one-dimensional array generated for the quantization matrix of interest. For example, the encoding unit 113 refers to Fig.11A The coding table illustrated in FIG. 1 is used to generate a bit string as coded data by replacing each element value in the one-dimensional array with the corresponding binary code. Note that the coding table is not limited to Fig.11A The coding table shown, and for example, can be used Fig. 11B The coding table shown in the example.
[0097] Return to reference Figure 1 , the integrated coding unit 111 integrates the "header information required for image coding" with the coded data generated by the coding unit 113, and generates header code data using the coded data integrated with the header information. Then, the integrated coding unit 111 generates a bit stream by multiplexing the coded data generated by the coding unit 110 and the header code data, and outputs the bit stream.
[0098] Fig. 6A An example of the structure of the bit stream generated by the integrated coding unit 111 is shown. The sequence header includes the encoded data of the quantization matrix, and includes the encoded data of each element. However, the location where these are encoded is not limited to this, and these can be encoded to the picture header or other headers. When the quantization matrix is changed in a sequence, the quantization matrix can be updated by newly encoding the quantization matrix. At this time, all quantization matrices can be rewritten, or a part of the quantization matrix can be changed by specifying the prediction mode for the quantization matrix corresponding to the quantization matrix to be rewritten.
[0099] Reference Figure 3 The flowchart of FIG. 1 is used to illustrate the encoding process using the above-mentioned image encoding device. Figure 3 The processing of the flowchart is a coding process for one input image. Therefore, when coding the images of each frame of a moving image or a plurality of images captured regularly or irregularly, the processing of steps S304 to S311 is repeated for each image.
[0100] At the beginning according to Figure 3 Before the processing of the flowchart of , the quantization matrix corresponding to the intra prediction, the quantization matrix corresponding to the inter prediction, and the quantization matrix corresponding to the mixed intra-inter prediction have been registered in the holding unit 103. Note that, as described above, all the quantization matrices held by the holding unit 103 are quantization matrices according to the sizes of the sub-blocks obtained by division.
[0101] In step S302 , the encoding unit 113 reads out the quantization matrix (including at least the quantization matrix used for quantization by the transform / quantization unit 105 ) held by the holding unit 103 , and encodes the read quantization matrix, thereby generating encoded data.
[0102] In step S303, the integration coding unit 111 generates “header information required for image coding.” Then, the integration coding unit 111 integrates the “header information required for image coding” with the coded data generated by the coding unit 113 in step S302, and generates header code data using the coded data integrated with the header information.
[0103] In step S304, the segmentation unit 102 segments the input image into a plurality of basic blocks and outputs each of the segmented basic blocks. The prediction unit 104 segments the basic block into a plurality of sub-blocks in units of the basic block.
[0104] In step S305, the prediction unit 104 selects one of the unselected sub-blocks in the sub-blocks of the input image as a selected sub-block, and determines a prediction mode for the selected sub-block. The prediction unit 104 performs prediction on the selected sub-block according to the determined prediction mode, and obtains a predicted image, a prediction error, and prediction information for the selected sub-block.
[0105] In step S306, the transform / quantization unit 105 performs an orthogonal transform process corresponding to the size of the prediction error for the selected sub-block acquired in step S305, thereby generating an orthogonal transform coefficient. Then, the transform / quantization unit 105 acquires a quantization matrix corresponding to the prediction mode of the selected sub-block from among the quantization matrices held by the holding unit 103, and quantizes the orthogonal transform coefficient of the sub-block using the acquired quantization matrix, thereby acquiring a quantization coefficient.
[0106] In step S307, the inverse quantization / inverse transform unit 106 performs inverse quantization on the quantization coefficient of the selected subblock acquired in step S306 using the quantization matrix used by the transform / quantization unit 105 to quantize the selected subblock, thereby generating a transform coefficient. Then, the inverse quantization / inverse transform unit 106 performs an inverse orthogonal transform on the generated transform coefficient, thereby generating (reproducing) a prediction error.
[0107] In step S308, the image reproduction unit 107 generates a predicted image from the image stored in the frame memory 108 based on the prediction information acquired in step S305, and reproduces the image of the sub-block by adding the predicted image and the prediction error generated in step S307. Then, the image reproduction unit 107 stores the reproduced image in the frame memory 108.
[0108] In step S309 , the encoding unit 110 performs entropy encoding of the quantization coefficient acquired in step S306 and the prediction information acquired in step S305 , thereby generating encoded data.
[0109] The integration encoding unit 111 generates a bit stream by multiplexing the header code data generated in step S303 and the encoded data generated by the encoding unit 110 in step S309 , and outputs the bit stream.
[0110] In step S310, the control unit 150 determines whether all sub-blocks of the input image are selected as selection sub-blocks. As a result of this determination, if all sub-blocks of the input image are selected as selection sub-blocks, the process proceeds to step S311. On the other hand, if at least one sub-block that has not been selected as a selection sub-block remains among the sub-blocks of the input image, the process returns to step S305.
[0111] In step S311, the loop filtering unit 109 performs loop filtering processing on the image stored in the frame memory 108 (the image of the selected subblock reproduced in step S308). Then, the loop filtering unit 109 stores the image subjected to the loop filtering processing in the frame memory 108.
[0112] Through this process, since the transform coefficients of the subblocks subjected to the hybrid intra-inter prediction can be quantized using a quantization matrix corresponding to the hybrid intra-inter prediction, it is possible to control quantization for each frequency component and improve image quality.
[0113] <Modification>
[0114] In the first embodiment, quantization matrices are prepared for intra prediction, inter prediction, and mixed intra-inter prediction, respectively, and the quantization matrices corresponding to the respective predictions are encoded. However, some of these quantization matrices may be shared.
[0115] For example, in order to quantize the orthogonal transform coefficients of the prediction error obtained based on the hybrid intra-frame inter prediction, the quantization matrix corresponding to the hybrid intra-frame inter prediction may not be used, but the quantization matrix corresponding to the intra prediction may be used. That is, for example, in order to quantize the orthogonal transform coefficients of the prediction error obtained based on the hybrid intra-frame inter prediction, the quantization matrix corresponding to the intra prediction may not be used. Figure 8C The quantization matrix used for the hybrid intra-inter prediction shown in FIG. Fig. 8A The quantization matrix used for intra prediction shown in FIG. 1 is a quantization matrix used for intra prediction. In this case, the encoding of the quantization matrix corresponding to the hybrid intra-inter prediction can be omitted. This makes it possible to reduce the amount of encoded data of the quantization matrix included in the bit stream and reduce image quality degradation caused by errors of intra prediction such as block distortion.
[0116] In addition, in order to quantize the orthogonal transform coefficients of the prediction error obtained based on the hybrid intra-frame inter prediction, the quantization matrix corresponding to the hybrid intra-frame inter prediction may be used instead of the quantization matrix corresponding to the inter prediction. That is, for example, in order to quantize the orthogonal transform coefficients of the prediction error obtained based on the hybrid intra-frame inter prediction, the quantization matrix corresponding to the inter prediction may be used instead of the quantization matrix corresponding to the inter prediction. Figure 8C The quantization matrix used for the hybrid intra-inter prediction shown in FIG. Figure 8B The quantization matrix used for inter prediction shown in FIG. 1 is a quantization matrix used for inter prediction. In this case, the encoding of the quantization matrix corresponding to the hybrid intra-inter prediction can be omitted. This makes it possible to reduce the amount of encoded data of the quantization matrix included in the bit stream and reduce image quality degradation caused by errors of inter prediction such as jitter motion.
[0117] In addition, the quantization matrix to be used for the subblock can be determined based on the size of the area of "prediction pixels obtained by intra prediction" and the area of "prediction pixels obtained by inter prediction" in the predicted image of the subblock on which the hybrid intra-inter prediction is performed.
[0118] For example, Fig.12E As shown, it is assumed that the sub-block 1200 is divided into a divided area 1200c and a divided area 1200d. It is assumed that "predicted pixels obtained by intra prediction" are obtained as predicted pixels of the divided area 1200c, and "predicted pixels obtained by inter prediction" are obtained as predicted pixels of the divided area 1200d. It is also assumed that the size (area (number of pixels)) S1 of the divided area 1200c: the size (area (number of pixels)) S2 of the divided area 1200d = 1:3.
[0119] In this case, the size of the partitioned region 1200d to which the inter-frame prediction is applied in the sub-block 1200 is larger than the size of the partitioned region 1200c to which the intra-frame prediction is applied. Therefore, the transform / quantization unit 105 applies a quantization matrix corresponding to the inter-frame prediction (for example, Figure 8B The quantization matrix shown in FIG. 12 is used to quantize the transform coefficients of sub-block 1200.
[0120] Note that if the size of the partitioned region 1200d to which the inter prediction is applied in the sub-block 1200 is smaller than the size of the partitioned region 1200c to which the intra prediction is applied, the transform / quantization unit 105 applies a quantization matrix corresponding to the intra prediction (for example, Fig. 8A The quantization matrix shown in FIG. 12 is used to quantize the transform coefficients of sub-block 1200.
[0121] This makes it possible to omit encoding of a quantization matrix corresponding to hybrid intra-inter prediction while reducing image quality degradation of a segmented region having a larger size. Thus, the amount of encoded data of the quantization matrix included in the bitstream can be reduced.
[0122] In addition, a quantization matrix obtained by synthesizing the "quantization matrix corresponding to intra prediction" and the "quantization matrix corresponding to inter prediction" according to the ratio of S1 to S2 can be generated as a quantization matrix corresponding to hybrid intra-inter prediction. For example, the transform / quantization unit 105 can generate a quantization matrix corresponding to hybrid intra-inter prediction using equation (1).
[0123] QM[x][y]={w×QMinter[x][y]+(1-w)×QMintra[x][y]})...(1)
[0124] Here, QM[x][y] indicates the value (quantization step value) of the element at the coordinate (x, y) in the quantization matrix corresponding to the hybrid intra-inter prediction. QMinter[x][y] indicates the value (quantization step value) of the element at the coordinate (x, y) in the quantization matrix corresponding to the inter prediction. QMintra[x][y] indicates the value (quantization step value) of the element at the coordinate (x, y) in the quantization matrix corresponding to the intra prediction. In addition, w has a value not less than 0 and not more than 1, which indicates the proportion of the area using inter prediction in the sub-block, and w=S2 / (S1+S2). Since the quantization matrix corresponding to the hybrid intra-inter prediction can be generated as needed and does not need to be created in advance, the encoding of the quantization matrix can be omitted. Therefore, the amount of encoded data of the quantization matrix included in the bitstream can be reduced. It is also possible to perform appropriate quantization control according to the ratio of the size of the area using intra prediction and inter prediction, and improve image quality.
[0125] Furthermore, in the first embodiment, a quantization matrix to be applied to a subblock to which hybrid intra-inter prediction is applied is uniquely determined. However, a quantization matrix may be selected by introducing an identifier.
[0126] Various methods are used to select a quantization matrix to be applied to a subblock to which hybrid intra-frame prediction is applied from a quantization matrix corresponding to intra-frame prediction, a quantization matrix corresponding to inter-frame prediction, and a quantization matrix corresponding to hybrid intra-frame prediction. For example, the control unit 150 can select a quantization matrix according to a user operation.
[0127] An identifier for specifying a quantization matrix selected as a quantization matrix to be applied to a subblock to which hybrid intra-inter prediction is applied is stored in the bit stream.
[0128] For example, in Figure 6B In, a quantization matrix encoding method information code is newly introduced as an identifier, thereby selecting a quantization matrix to be applied to a sub-block to which hybrid intra-frame-inter prediction is applied. For example, if the quantization matrix encoding method information code indicates 0, it indicates that the quantization matrix corresponding to the intra-frame prediction is applied to the sub-block to which the hybrid intra-frame-inter prediction is applied. In addition, if the quantization matrix encoding method information code indicates 1, it indicates that the quantization matrix corresponding to the inter-frame prediction is applied to the sub-block to which the hybrid intra-frame-inter prediction is applied. On the other hand, if the quantization matrix encoding method information code indicates 2, it indicates that the quantization matrix corresponding to the hybrid intra-frame-inter prediction is applied to the sub-block to which the hybrid intra-frame-inter prediction is applied.
[0129] This makes it possible to selectively achieve a reduction in the amount of encoded data of a quantization matrix included in a bit stream and unique quantization control for a subblock to which hybrid intra-inter prediction is applied.
[0130] Furthermore, in the first embodiment, a predicted image is generated, which includes predicted pixels (first predicted pixels) for one segmented area obtained by segmenting the sub-block, and predicted pixels (second predicted pixels) for other segmented areas. However, the predicted image generation method is not limited to this generation method. For example, in order to improve the image quality of an area (boundary area) near the boundary between one segmented area and other segmented areas, a third predicted pixel calculated by weighted averaging the first predicted pixel and the second predicted pixel included in the boundary area can be used as the predicted pixel of the boundary area. In this case, the predicted pixel value in the corresponding area corresponding to one segmented area in the predicted image is the first predicted pixel, and the predicted pixel value in the corresponding area corresponding to the other segmented area in the predicted image is the second predicted pixel. The predicted pixel value in the corresponding area corresponding to the above-mentioned boundary area in the predicted image is the third predicted pixel. This can suppress the degradation of image quality in the boundary area between segmented areas using different predictions and improve image quality.
[0131] In addition, in the first embodiment, three types of prediction including intra prediction, inter prediction and hybrid intra-inter prediction are described as examples, but the type and number of predictions are not limited to this example. For example, combined inter-intra prediction (CIIP) adopted in VVC can be used. Combined inter-intra prediction is a prediction of pixels in the entire encoding object block calculated by weighted averaging predicted pixels using intra prediction and predicted pixels using inter prediction. In this case, the quantization matrix used for the sub-block using hybrid intra-inter prediction can be shared as the quantization matrix used for the sub-block using combined inter-intra prediction. This makes it possible to apply quantization using a quantization matrix with the same quantization control characteristic to the following sub-blocks, for which a prediction having the common feature of using both predicted pixels using intra prediction and predicted pixels using inter prediction in the same sub-block is used. In addition, the code amount of the quantization matrix corresponding to the new prediction method can also be reduced.
[0132] Furthermore, in the first embodiment, the encoding object is an input image. However, the encoding object is not limited to an image. For example, a two-dimensional data array as feature quantity data used in machine learning such as object recognition can be encoded like an input image, and a bit stream can be generated and outputted thereby. This can efficiently encode feature quantity data used in machine learning.
[0133] [Second embodiment]
[0134] The image decoding device according to the present embodiment decodes the quantization coefficient for the decoding object block from the bit stream, derives the transform coefficient from the quantization coefficient using the quantization matrix, and performs inverse frequency transform of the transform coefficient, thereby deriving the prediction error for the decoding object block. Then, the image decoding device generates a prediction image by applying an intra-frame prediction image obtained by intra-frame prediction to a partial area in the decoding object block and applying an inter-frame prediction image obtained by inter-frame prediction to other areas in the decoding object block different from the partial area, and decodes the decoding object block using the generated prediction image and the prediction error.
[0135] In this embodiment, an image decoding device for decoding a bit stream encoded by the image encoding device according to the first embodiment will be described. Figure 2 An example of the functional structure of the image decoding device according to the present embodiment is explained with reference to the block diagram.
[0136] The control unit 250 controls the operation of the entire image decoding device. The separation decoding unit 202 acquires a bit stream encoded by the image encoding device according to the first embodiment. The bit stream acquisition form is not limited to a specific acquisition form. For example, the bit stream output from the image encoding device according to the first embodiment can be acquired via a network, or can be acquired from a memory that stores the bit stream at one end. Then, the separation decoding unit 202 separates information related to the decoding process or encoded data related to the coefficients from the acquired bit stream, and decodes the encoded data in the header of the bit stream. In the present embodiment, the separation decoding unit 202 separates the encoded data of the quantization matrix from the bit stream, and supplies the encoded data to the decoding unit 209. In addition, the separation decoding unit 202 separates the encoded data of the input image from the bit stream, and supplies the encoded data to the decoding unit 203. That is, the separation decoding unit 202 performs Figure 1 The operation of the integrated encoding unit 111 is shown to be the reverse operation.
[0137] The decoding unit 209 decodes the coded data supplied from the separation decoding unit 202, thereby reproducing the quantization matrix. The decoding unit 203 decodes the coded data supplied from the separation decoding unit 202, thereby reproducing the quantization coefficient and the prediction information.
[0138] The inverse quantization / inverse transform unit 204 performs the same operation as the inverse quantization / inverse transform unit 106 provided in the image encoding device according to the first embodiment. The inverse quantization / inverse transform unit 204 selects one of the quantization matrices decoded by the decoding unit 209, and inversely quantizes the quantization coefficients using the selected quantization matrix, thereby reproducing the transform coefficients. The inverse quantization / inverse transform unit 204 inversely orthogonally transforms the reproduced transform coefficients, thereby reproducing the prediction error.
[0139] The image reproduction unit 205 generates a predicted image by referring to the image stored in the frame memory 206 based on the prediction information decoded by the decoding unit 203. Then, the image reproduction unit 205 generates a reproduced image by adding the prediction error obtained by the inverse quantization / inverse transform unit 204 to the generated predicted image, and stores the generated reproduced image in the frame memory 206.
[0140] The loop filtering unit 207 performs loop filtering processing such as deblocking filtering or sample adaptive offset on the reproduced image stored in the frame memory 206. The reproduced image stored in the frame memory 206 is appropriately output by the control unit 250. The output destination of the reproduced image is not limited to a specific output destination. For example, the reproduced image can be displayed on a display screen of a display device such as a display, or the reproduced image can be output to a projection device such as a projector.
[0141] Next, the operation (bit stream decoding process) of the image decoding device having the above structure will be described. The separation decoding unit 202 obtains the bit stream generated by the image encoding device, separates information related to the decoding process or encoded data related to the coefficients from the bit stream, and decodes the encoded data present in the header of the bit stream. Fig. 6A The encoding data of the quantization matrix is extracted from the sequence header of the bit stream shown in FIG. 1 , and the extracted encoding data is supplied to the decoding unit 209. Furthermore, the separation and decoding unit 202 supplies the encoding data of each sub-block of the picture data to the decoding unit 203.
[0142] The decoding unit 209 decodes the encoded data of the quantization matrix supplied from the separation decoding unit 202, thereby reproducing a one-dimensional array. More specifically, the decoding unit 209 refers to Fig.11A or Fig. 11B , and generates a one-dimensional array in which the binary codes in the quantization matrix coded data are decoded into differential values and the differential values are arranged. FIG. 8A to FIG. 8C When the quantization matrix shown in the encoding data is decoded, the reproduction FIG. 10A to FIG. 10C In this embodiment, as in the first embodiment, the one-dimensional array is used Fig.11A (or Fig. 11B ) is used for decoding. The coding table is not limited to this, and other coding tables may be used as long as the same coding table as the first embodiment is used.
[0143] In addition, the decoding unit 209 reproduces each element value of the quantization matrix from each difference value of the reproduced one-dimensional array. A process opposite to the process performed by the encoding unit 113 for generating the one-dimensional array from the quantization matrix is performed. That is, the value of the element at the beginning of the one-dimensional array is the element value at the upper left corner of the quantization matrix. The value obtained by adding the value of the element at the beginning of the one-dimensional array to the value of the second element starting from the beginning of the one-dimensional array is the second element value in the above-mentioned "predetermined order". The value obtained by adding the value of the (n - 1)-th (2 < n ≤ N: N is the number of elements of the one-dimensional array) element starting from the beginning of the one-dimensional array to the value of the n-th element starting from the beginning of the one-dimensional array is the n-th element value in the above-mentioned "predetermined order". For example, the decoding unit 209 uses Fig. 9 the order shown to reproduce from FIG. 10A to FIG. 10C the one-dimensional array shown FIG. 8A to FIG. 8C the quantization matrix shown.
[0144] The decoding unit 203 decodes the encoded data of the input image supplied from the separation decoding unit 202, thereby decoding the quantization coefficients and the prediction information.
[0145] The inverse quantization / inverse transformation unit 204 designates the "prediction mode corresponding to the quantization coefficient to be decoded" included in the prediction information decoded by the decoding unit 203, and selects the quantization matrix corresponding to the designated prediction mode from the quantization matrix reproduced by the decoding unit 209. Then, the inverse quantization / inverse transformation unit 204 inverse-quantizes the quantization coefficients using the selected quantization matrix, thereby reproducing the transform coefficients. The inverse quantization / inverse transformation unit 204 reproduces the prediction error by performing an inverse orthogonal transformation on the reproduced transform coefficients, and supplies the reproduced prediction error to the image reproduction unit 205.
[0146] The image reproduction unit 205 generates a predicted image by referring to the image stored in the frame memory 206 based on the prediction information decoded by the decoding unit 203. In the present embodiment, as in the prediction unit 104 according to the first embodiment, three types of prediction including intra-frame prediction, inter-frame prediction, and hybrid intra-frame / inter-frame prediction are used. The detailed prediction process is the same as the prediction process of the prediction unit 104 described in the first embodiment, and its description will be omitted. Then, the image reproduction unit 205 generates a reproduced image by adding the prediction error obtained by the inverse quantization / inverse transformation unit 204 to the generated predicted image, and stores the generated reproduced image in the frame memory 206. The reproduced image stored in the frame memory 206 is a prediction reference candidate to be referred to when decoding other sub-blocks.
[0147] The loop filtering unit 207 operates as the loop filtering unit 109 described above, and performs loop filtering processing such as deblocking filtering or sample adaptive offset on the reproduced image stored in the frame memory 206. The reproduced image stored in the frame memory 206 is output by the control unit 250 as appropriate.
[0148] Reference Figure 4 The decoding process of the image decoding device according to the present embodiment is explained with reference to the flowchart of FIG. 401. In step S401, the separation decoding unit 202 acquires the encoded bit stream. Then, the separation decoding unit 202 separates the encoded data of the quantization matrix from the acquired bit stream, and supplies the encoded data to the decoding unit 209. In addition, the separation decoding unit 202 separates the encoded data of the input image from the bit stream, and supplies the encoded data to the decoding unit 203.
[0149] In step S402, decoding section 209 decodes the coded data supplied from separation and decoding section 202 to reproduce the quantization matrix. In step S403, decoding section 203 decodes the coded data supplied from separation and decoding section 202 to reproduce the quantization coefficients and prediction information of the decoding target subblock.
[0150] In step S404, the inverse quantization / inverse transform unit 204 specifies the "prediction mode corresponding to the quantization coefficient of the decoding target subblock" included in the prediction information decoded by the decoding unit 203. The inverse quantization / inverse transform unit 204 selects a quantization matrix corresponding to the specified prediction mode from among the quantization matrices reproduced by the decoding unit 209. For example, if the prediction mode specified for the decoding target subblock is intra prediction, FIG. 8A to FIG. 8C In the quantization matrix shown, select Fig. 8A In addition, if the prediction mode specified for the decoding target subblock is inter-frame prediction, select Figure 8B In addition, if the prediction mode specified for the decoding target subblock is hybrid intra-inter prediction, select Figure 8C The quantization matrix used for the hybrid intra-inter prediction shown in FIG. Then, the inverse quantization / inverse transform unit 204 inversely quantizes the quantization coefficient of the decoding target subblock using the selected quantization matrix, thereby reproducing the transform coefficient. The inverse quantization / inverse transform unit 204 reproduces the prediction error of the decoding target subblock by performing inverse orthogonal transform on the reproduced transform coefficient, and supplies the reproduced prediction error to the image reproduction unit 205.
[0151] In step S405, the image reproduction unit 205 generates a predicted image of the decoding target subblock by referring to the image stored in the frame memory 206 based on the prediction information decoded by the decoding unit 203. Then, the image reproduction unit 205 generates a reproduced image of the decoding target subblock by adding the prediction error of the decoding target subblock obtained by the inverse quantization / inverse transform unit 204 to the generated predicted image, and stores the generated reproduced image in the frame memory 206.
[0152] In step S406, the control unit 250 determines whether the processing of steps S403 to S405 has been performed for all sub-blocks. As a result of this determination, if the processing of steps S403 to S405 has been performed for all sub-blocks, the process proceeds to step S407. On the other hand, if a sub-block that has not been processed by steps S403 to S405 still remains, the process returns to step S403 to perform the processing of steps S403 to S405 for the sub-block.
[0153] In step S407 , the loop filtering unit 207 performs loop filtering processing such as deblocking filtering or sample adaptive offset on the reproduced image generated in step S405 and stored in the frame memory 206 .
[0154] Through this processing, even for the subblock using hybrid intra-inter prediction generated in the first embodiment, it is possible to control quantization for each frequency component and decode a bit stream with improved image quality.
[0155] <Modification>
[0156] In the second embodiment, quantization matrices are prepared for intra prediction, inter prediction, and mixed intra-inter prediction, respectively, and the quantization matrices corresponding to each prediction are decoded. However, some of these quantization matrices may be shared.
[0157] For example, in order to inversely quantize the quantized coefficients of the orthogonal transform coefficients of the prediction error obtained based on the hybrid intra-frame inter prediction, the quantization matrix corresponding to the intra prediction may be decoded and used instead of the quantization matrix corresponding to the hybrid intra-frame inter prediction. That is, for example, in order to inversely quantize the quantized coefficients of the orthogonal transform coefficients of the prediction error obtained based on the hybrid intra-frame inter prediction, the quantization matrix corresponding to the intra prediction may be decoded and used instead of the quantization matrix corresponding to the hybrid intra-frame inter prediction. Fig. 8A Quantization matrix used for intra prediction shown in FIG. 1 . In this case, decoding of the quantization matrix corresponding to the hybrid intra-inter prediction can be omitted. That is, a bit stream in which the amount of encoded data of the quantization matrix included in the bit stream is reduced can be decoded, and a decoded image in which image quality degradation caused by errors of intra prediction such as block distortion is reduced can be obtained.
[0158] In addition, in order to inversely quantize the quantized coefficients of the orthogonal transform coefficients of the prediction error obtained based on the hybrid intra-frame inter prediction, the quantization matrix corresponding to the inter prediction may be decoded and used instead of the quantization matrix corresponding to the hybrid intra-frame inter prediction. That is, for example, in order to inversely quantize the quantized coefficients of the orthogonal transform coefficients of the prediction error obtained based on the hybrid intra-frame inter prediction, the quantization matrix corresponding to the inter prediction may be used. Figure 8B Quantization matrix used for inter prediction shown in FIG. 1 . In this case, decoding of the quantization matrix corresponding to the hybrid intra-inter prediction can be omitted. That is, a bit stream in which the amount of encoded data of the quantization matrix included in the bit stream is reduced can be decoded, and a decoded image in which image quality degradation caused by an error of inter prediction such as jitter motion is reduced can be obtained.
[0159] In addition, the quantization matrix to be used for inverse quantization of the sub-block can be determined based on the size of the area of "prediction pixels obtained by intra prediction" and the area of "prediction pixels obtained by inter prediction" in the predicted image of the sub-block on which the hybrid intra-inter prediction is performed.
[0160] For example, Fig.12E As shown, it is assumed that the sub-block 1200 is divided into a divided area 1200c and a divided area 1200d. It is assumed that "predicted pixels obtained by intra prediction" are obtained as predicted pixels of the divided area 1200c, and "predicted pixels obtained by inter prediction" are obtained as predicted pixels of the divided area 1200d. It is also assumed that the size (area (number of pixels)) S1 of the divided area 1200c: the size (area (number of pixels)) S2 of the divided area 1200d = 1:3.
[0161] In this case, the size of the partitioned area 1200d to which the inter prediction is applied is larger than the size of the partitioned area 1200c to which the intra prediction is applied in the subblock 1200. Therefore, the inverse quantization / inverse transform unit 204 applies a quantization matrix corresponding to the inter prediction to inversely quantize the quantization coefficients of the subblock 1200.
[0162] Note that if the size of the partitioned area 1200d to which inter-frame prediction is applied in the sub-block 1200 is smaller than the size of the partitioned area 1200c to which intra-frame prediction is applied, the inverse quantization / inverse transform unit 204 applies the quantization matrix corresponding to the intra-frame prediction to inverse quantize the quantization coefficients of the sub-block 1200.
[0163] This makes it possible to omit the decoding of the quantization matrix corresponding to the hybrid intra-inter prediction while reducing the image quality degradation of the segmented regions having a larger size. This makes it possible to decode a bitstream in which the amount of encoded data of the quantization matrix included in the bitstream is reduced.
[0164] In addition, a quantization matrix obtained by synthesizing the "quantization matrix corresponding to intra prediction" and the "quantization matrix corresponding to inter prediction" according to the ratio of S1 to S2 can be generated as a quantization matrix corresponding to hybrid intra-inter prediction. For example, the inverse quantization / inverse transform unit 204 can generate a quantization matrix corresponding to hybrid intra-inter prediction using the above-mentioned formula (1).
[0165] Since a quantization matrix corresponding to the hybrid intra-inter prediction can be generated as needed, the encoding of the quantization matrix can be omitted. This makes it possible to decode a bit stream in which the amount of encoded data of the quantization matrix included in the bit stream is reduced. It is also possible to perform appropriate quantization control according to the ratio of the sizes of the regions using intra prediction and inter prediction, and decode a bit stream with improved image quality.
[0166] In addition, in the second embodiment, the quantization matrix to be applied to the sub-block to which the hybrid intra-frame inter prediction is applied is uniquely determined. However, as in the first embodiment, the quantization matrix can be selected by introducing an identifier. Therefore, it is possible to decode a bit stream in which a reduction in the amount of encoded data of the quantization matrix included in the bit stream and unique quantization control for the sub-block to which the hybrid intra-frame inter prediction is applied are selectively achieved.
[0167] Furthermore, in the second embodiment, a predicted image including predicted pixels (first predicted pixels) for one segmented area obtained by segmenting the sub-blocks, and predicted pixels (second predicted pixels) for other segmented areas is decoded. However, the predicted image to be decoded is not limited to the predicted image. For example, as in the modified example of the first embodiment, a predicted image can be generated in which a third predicted pixel calculated by weighted averaging the first predicted pixel and the second predicted pixel included in an area near the boundary between one segmented area and the other segmented area (boundary area) is used as a predicted pixel of the boundary area. In this case, in the predicted image to be decoded, as in the first embodiment, the predicted pixel value in the corresponding area corresponding to one segmented area in the predicted image is the first predicted pixel, and the predicted pixel value in the corresponding area corresponding to the other segmented area in the predicted image is the second predicted pixel. The predicted pixel value in the corresponding area corresponding to the above-mentioned boundary area in the predicted image is the third predicted pixel. This can suppress the degradation of image quality in the boundary area between segmented areas using different predictions, and decode a bit stream with improved image quality.
[0168] In addition, in the second embodiment, three types of prediction including intra prediction, inter prediction and hybrid intra-inter prediction are described as examples, but the type and number of predictions are not limited to this example. For example, combined inter-intra prediction (CIIP) adopted in VVC can be used. In this case, the quantization matrix used for the sub-block using hybrid intra-inter prediction can be shared as the quantization matrix used for the sub-block using combined inter-intra prediction. This makes it possible to decode a bitstream in which quantization using a quantization matrix with the same quantization control characteristics is applied to a sub-block using a prediction method having the common feature of using both predicted pixels using intra prediction and predicted pixels using inter prediction in the same sub-block. In addition, a bitstream with a reduced code amount of the quantization matrix corresponding to the new prediction method can also be decoded.
[0169] Furthermore, in the second embodiment, an input image as an encoding object is decoded from a bit stream. However, the decoding object is not limited to an image. For example, a two-dimensional data array obtained by encoding a two-dimensional array of feature quantity data used in machine learning such as object recognition, as an input image, can be decoded from a bit stream including encoded data. This makes it possible to decode a bit stream that efficiently encodes feature quantity data used in machine learning.
[0170] [Third embodiment]
[0171] Figure 1 and Figure 2 The functional units shown may be realized by hardware, or the functional units except the holding unit 103 and the frame memories 108 and 206 may be realized by software (computer program).
[0172] In the former case, the hardware may be a circuit incorporated in a device that performs encoding or decoding of images (such as a camera device, etc.), or may be a circuit incorporated in a device that performs encoding or decoding of images supplied from an external device (such as a camera device or a server device, etc.).
[0173] In the latter case, the computer program may be stored in a memory of a device that performs encoding or decoding of images (such as a camera device, etc.), or in a memory accessible from a device that performs encoding or decoding of images supplied from an external device (such as a camera device or a server device, etc.). A device (computer device) that can read out a computer program from a memory and execute the computer program may be applied to the above-mentioned image encoding device or the above-mentioned image decoding device. Figure 5 An example of a hardware structure of a computer device is described with a block diagram of FIG.
[0174] The CPU 501 executes various processes using computer programs and data stored in the RAM 502 or the ROM 503. Thus, the CPU 501 controls the operation of the entire computer device, and executes or controls various processes described as processes executed by the image encoding device or the image decoding device in the above-described embodiments and modifications.
[0175] The RAM 502 has an area configured to store computer programs and data loaded from an external storage device 506, and an area configured to store data acquired from the outside via an I / F (interface) 507. The RAM 502 also has a work area (frame memory, etc.) used when the CPU 501 performs various processes. Thus, the RAM 502 can provide various areas as appropriate.
[0176] The ROM 503 stores setting data of the computer device, computer programs and data associated with activation of the computer device, computer programs and data associated with basic operations of the computer device, and the like.
[0177] The operation unit 504 is a user interface such as a keyboard, a mouse, or a touch panel, and a user can input various instructions to the CPU 501 by operating the operation unit 504 .
[0178] The display unit 505 includes a liquid crystal screen or a touch panel screen, and displays the processing result of the CPU 501 as an image or characters, etc. Note that the display unit 505 may be a projection device such as a projector that projects images or characters.
[0179] The external storage device 506 is a large-capacity information storage device such as a hard disk drive device. In the external storage device 506, an OS (operating system) and a computer program and data for causing the CPU 501 to execute the above-mentioned various processes explained as the processes performed by the image encoding device or the image decoding device are stored. Information (encoding table, etc.) treated as known information in the above description is also stored in the external storage device 506. Encoding object data (such as an input image or a two-dimensional data array, etc.) can be stored in the external storage device 506.
[0180] Computer programs and data stored in the external storage device 506 are appropriately loaded into the RAM 502 according to the control of the CPU 501 and processed by the CPU 501. Note that the above-mentioned holding unit 103 and frame memories 108 and 206 can be implemented using the RAM 502, ROM 503, or the external storage device 506 or the like.
[0181] A network such as a LAN or the Internet or other devices such as a projection device or a display device can be connected to the I / F 507 , and the computer device can acquire or send various information via the I / F 507 .
[0182] The CPU 501 , the RAM 502 , the ROM 503 , the operation unit 504 , the display unit 505 , the external storage device 506 , and the I / F 507 are all connected to a system bus 508 .
[0183] In the above configuration, when the computer device is powered on, the CPU 501 executes the boot program stored in the ROM 503, loads the OS stored in the external storage device 506 into the RAM 502, and starts the OS. As a result, the computer device can communicate via the I / F 507. Under the control of the OS, the CPU 501 loads the application associated with the code from the external storage device 506 into the RAM 502 and executes the application, thereby serving as Figure 1 The functional units shown in FIG. 1 are excluding the holding unit 103 and the frame memory 108. That is, the computer device functions as the above-mentioned image encoding device. On the other hand, under the control of the OS, the CPU 501 loads an application associated with decoding from the external storage device 506 into the RAM 502 and executes the application, thereby functioning as Figure 2 The functional units shown are (except the frame memory 206). That is, the computer device functions as the above-mentioned image decoding device.
[0184] Note that in this embodiment, Figure 5 The computer device of the structure shown can be applied to the description of the image encoding device or the image decoding device. However, the hardware structure of the computer device applicable to the image encoding device or the image decoding device is not limited to Figure 5 In addition, the hardware structure of the computer device applied to the image encoding device and the hardware structure of the computer device applied to the image decoding device may be the same or different.
[0185] The numerical values, processing timing, processing order, processing subject, and data (information) transmission destination / transmission source / storage location used in the above-mentioned embodiments and modifications are merely examples for detailed description and are not intended to be limited to these examples.
[0186] A part or all of the above-described embodiments or modified examples may be appropriately combined and used. In addition, a part or all of the above-described embodiments or modified examples may be selectively used.
[0187] (Other embodiments)
[0188] The present invention can be implemented by the following process: a program for implementing one or more functions of the above-mentioned embodiment is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be implemented by a circuit (e.g., ASIC) for implementing one or more functions.
[0189] The present invention is not limited to the above-described embodiments, and various changes and modifications can be made within the spirit and scope of the present invention. Therefore, the following claims are made to disclose the scope of the present invention.
[0190] This application claims the benefit of Japanese Patent Application No. 2022-046034, filed on March 22, 2022, which is hereby incorporated by reference herein.
Claims
1. An image encoding device, It is characterized in that include: A prediction unit configured to generate a prediction image in which an intra-frame prediction image obtained by intra-frame prediction is applied to a partial region in a coding target block included in an image, and an inter-frame prediction image obtained by inter-frame prediction is applied to another region in the coding target block different from the partial region, and to derive prediction errors, each of which is a difference between the coding target block and the generated prediction image; a transform unit configured to derive transform coefficients by frequency transforming the prediction error derived by the prediction unit; quantizing means for deriving quantized coefficients by quantizing the transform coefficients derived by the transform means using a quantization matrix; as well as The first encoding component is used to encode the quantization coefficients derived by the quantization component.
2. The image encoding device according to claim 1, It is characterized in that Also includes: A second encoding component, used for encoding the quantization matrix; as well as A generating means for generating a bit stream and outputting the bit stream, the bit stream including a result of encoding by the first encoding means and a result of encoding by the second encoding means.
3. The image encoding device according to claim 1 or 2, It is characterized in that The quantization section quantizes the transform coefficients using at least one of a preset quantization matrix for intra prediction and a preset quantization matrix for inter prediction.
4. The image encoding device according to claim 3, It is characterized in that The quantization section quantizes the transform coefficients using a quantization matrix generated based on both the quantization matrix used for the intra prediction and the quantization matrix used for the inter prediction.
5. The image encoding device according to claim 4, It is characterized in that The quantization means performs the quantization using a quantization matrix obtained by synthesizing a quantization matrix used for the intra prediction and a quantization matrix used for the inter prediction according to a ratio between a size of the partial region and a size of the other region.
6. The image encoding device according to claim 3, It is characterized in that The quantization section quantizes the transform coefficients using a quantization matrix used for the intra prediction instead of a quantization matrix used for the inter prediction.
7. The image encoding device according to claim 3, It is characterized in that The quantization section quantizes the transform coefficients using a quantization matrix used for the inter prediction instead of a quantization matrix used for the intra prediction.
8. The image encoding device according to claim 3, It is characterized in that When the size of the partial region in the encoding target block is larger than the size of the other region in the encoding target block, the quantization unit performs the quantization using a quantization matrix used for the intra prediction.
9. The image encoding device according to claim 3, It is characterized in that When the size of the other region in the encoding target block is larger than the size of the partial region in the encoding target block, the quantization unit performs the quantization using a quantization matrix used for the inter prediction.
10. The image encoding device according to any one of claims 1 to 9, It is characterized in that The prediction component generates the predicted image using the following predicted pixels: predicted pixels located at positions corresponding to the partial area among predicted pixels included in an intra-prediction image generated using intra-prediction for the encoding target block, and The predicted pixels located at the position corresponding to the other region are predicted pixels included in the inter-frame prediction image generated using the inter-frame prediction for the encoding target block.
11. The image encoding device according to any one of claims 1 to 10, It is characterized in that The partial region is one of two regions generated by dividing the encoding target block using a line segment, and The other region is the other region of the two regions.
12. An image decoding device, It is characterized in that include: A decoding unit for decoding a quantized coefficient for a decoding target block from a bit stream; an inverse quantization component for deriving transform coefficients from the quantized coefficients using a quantization matrix; an inverse transform unit configured to derive a prediction error for the decoding target block by performing an inverse frequency transform on the transform coefficients; as well as A prediction component is used to generate a prediction image, and use the generated prediction image and the prediction error to decode the decoding object block, in which the intra-frame prediction image obtained by intra-frame prediction is applied to a partial area in the decoding object block, and the inter-frame prediction image obtained by inter-frame prediction is applied to other areas of the decoding object block different from the partial area.
13. A method for encoding an image, It is characterized in that include: A prediction step for generating a prediction image and deriving a prediction error, wherein an intra-frame prediction image obtained by intra-frame prediction is applied to a partial region in a coding target block included in an image, and an inter-frame prediction image obtained by inter-frame prediction is applied to another region in the coding target block different from the partial region, each of the prediction errors being a difference between the coding target block and the generated prediction image; a transform step for deriving transform coefficients by frequency transforming the prediction error derived in the predicting step; a quantization step for deriving quantized coefficients by quantizing the transformed coefficients derived in the transforming step using a quantization matrix; as well as The encoding step is used to encode the quantization coefficients derived in the quantization step.
14. An image decoding method, It is characterized in that include: A decoding step for decoding a quantized coefficient for a decoding target block from a bit stream; an inverse quantization step for deriving transform coefficients from said quantized coefficients using a quantization matrix; an inverse transform step for deriving a prediction error for the decoding target block by performing an inverse frequency transform on the transform coefficients; as well as A prediction step for generating a prediction image, and decoding the decoding object block using the generated prediction image and the prediction error, in which the intra-frame prediction image obtained by intra-frame prediction is applied to a partial area in the decoding object block, and the inter-frame prediction image obtained by inter-frame prediction is applied to other areas of the decoding object block different from the partial area.
15. A computer program configured to cause a computer to function as each component of the image encoding device according to any one of claims 1 to 11.
16. A computer program configured to cause a computer to function as each component of the image decoding device according to claim 12.
