Image encoding device, image decoding device, image encoding method, and image decoding method

Abstract

To provide a technique to allow deblocking filter processing designed for intra-inter mixed prediction.SOLUTION: When at least one of a first block and a second block is a block applying a prediction mode for deriving a predicted pixel of a block to be encoded by using pixels in an image including the block, an image encoding device determines the intensity of deblocking filter processing performed on the boundary between the first block and the second block as a first intensity. When at least one of the first block and the second block is a block applying a prediction mode for deriving a predicted pixel by using pixels in an image including a block to be encoded for one area in the block, and deriving a predicted pixel by using pixels in the other image different from the image including the block for the other area different from one area in the block, the image encoding device determines the intensity as an intensity based on the first intensity.SELECTED DRAWING: Figure 13

Description

[Technical Field]

[0001] The present invention relates to image encoding and decoding techniques. [Background technology]

[0002] The Versatile Video Coding (VVC) coding method (hereafter referred to as VVC) is known as a coding method for compressing and recording moving images. In VVC, basic blocks of up to 128 x 128 pixels are divided into rectangular sub-blocks in addition to the conventional square sub-blocks to improve coding efficiency.

[0003] Furthermore, VVC uses a matrix called a quantization matrix, which weights coefficients after orthogonal transformation (hereinafter referred to as orthogonal transform coefficients) according to frequency components. By further reducing the data of high-frequency components, degradation of which is less noticeable to the human eye, it is possible to improve compression efficiency while maintaining image quality. Patent Document 1 discloses a technique for encoding such a quantization matrix.

[0004] Furthermore, in VVC, adaptive deblocking filter processing is performed on the block boundaries of a reconstructed image obtained by adding a signal after inverse quantization and inverse transform processing to a predicted image, thereby suppressing block distortion that is easily noticeable to the human eye and preventing image quality degradation from propagating to the predicted image. Patent Document 2 discloses technology related to such a deblocking filter.

[0005] In recent years, the Joint Video Experts Team (JVET), which standardized VVC, has been studying technologies to achieve compression efficiency that exceeds that of VVC.To improve coding efficiency, in addition to conventional intra-prediction and inter-prediction, a new prediction method that mixes intra-predicted and inter-predicted pixels within the same subblock (hereinafter referred to as mixed intra-inter prediction) is being studied. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-38758 [Patent Document 2] Special Publication No. 2014-507863 Summary of the Invention [Problem to be solved by the invention]

[0007] The deblocking filter in VVC is based on conventional prediction methods such as intra prediction and inter prediction, and is not compatible with the new intra / inter mixed prediction method. Therefore, this invention provides technology that enables deblocking filter processing that is compatible with intra / inter mixed prediction. [Means for solving the problem]

[0008] One aspect of the present invention includes an encoding means for encoding an image by performing a prediction process on a block-by-block basis, a determination means for determining the strength of a deblocking filter process to be performed on a boundary between a first block in the image and a second block adjacent to the first block in the image, and a processing means for performing a deblocking filter process on the boundary in accordance with the strength determined by the determination means, wherein the encoding means selects, as the prediction process, a first prediction mode for deriving predicted pixels of the block to be encoded using pixels in an image including the block to be encoded, a second prediction mode for deriving predicted pixels of the block to be encoded using pixels in another image different from the image including the block to be encoded, and a third prediction mode for deriving predicted pixels of the block to be encoded using pixels in the image including the block to be encoded. a third prediction mode in which, for a partial area in the block, predicted pixels are derived using pixels in an image including the block to be coded, and for another area different from the partial area in the block to be coded, predicted pixels are derived using pixels in another image different from the image including the block to be coded; and the determination means is characterized in that, if at least one of the first block and the second block is a block to which the first prediction mode is applied, it determines the intensity to be a first intensity, and if at least one of the first block and the second block is a block to which the third prediction mode is applied, it determines the intensity to be an intensity based on the first intensity. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide a technique for enabling deblocking filtering processing compatible with mixed intra / inter prediction. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing an example of the functional configuration of an image encoding device. [Figure 2] FIG. 1 is a block diagram showing an example of the functional configuration of an image decoding device. [Figure 3] 10 is a flowchart of an encoding process performed by an image encoding device. [Figure 4]10 is a flowchart of a decoding process in an image decoding device. [Figure 5] FIG. 1 is a block diagram showing an example of the hardware configuration of a computer device. [Figure 6] FIG. 10 is a diagram showing an example of the configuration of a bit stream. [Figure 7] FIG. 7 is a diagram showing an example of a method for dividing a basic block 700 into sub-blocks. [Figure 8] FIG. 10 is a diagram showing an example of a quantization matrix. [Figure 9] FIG. 10 is a diagram showing the order in which values ​​of elements in a quantization matrix are referenced. [Figure 10] FIG. 10 is a diagram showing an example of a one-dimensional array. [Figure 11] FIG. 10 is a diagram showing an example of an encoding table. [Figure 12] FIG. 10 is a diagram for explaining mixed intra-inter prediction. [Figure 13] FIG. 1 is a block diagram showing an example of the functional configuration of an image encoding device. [Figure 14] FIG. 2 is a diagram illustrating a group of pixels at the boundary between sub-blocks P and Q. [Figure 15] 10 is a flowchart of an encoding process performed by an image encoding device. [Figure 16] FIG. 1 is a block diagram showing an example of the functional configuration of an image decoding device. [Figure 17] 10 is a flowchart of a decoding process in an image decoding device. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] [First embodiment] The image encoding device according to this embodiment applies an intra-predicted image obtained by intra-prediction to a portion of a block to be encoded included in an image, and applies an inter-predicted image obtained by inter-prediction to a portion of the block that is different from the portion of the block, thereby obtaining a predicted image.The image encoding device then encodes quantized coefficients obtained by quantizing, using a quantization matrix, the orthogonal transform coefficients of the difference between the block and the predicted image (first encoding).

[0013] First, an example of the functional configuration of the image encoding device according to this embodiment will be described using the block diagram in FIG. 1. A control unit 150 controls the operation of the entire image encoding device. A division unit 102 divides an input image into a plurality of basic blocks and outputs each of the divided basic blocks. The input images may be images of each frame constituting a moving image (e.g., images of each frame in a moving image at 30 frames per second), or may be still images captured periodically or irregularly. The division unit 102 may acquire the input images from any device, such as from an imaging device such as a video camera, from a device storing multiple images, or from a memory accessible to the device itself.

[0014] The storage unit 103 stores quantization matrices corresponding to each of a plurality of prediction processes. In this embodiment, the storage unit 103 stores a quantization matrix corresponding to intra-prediction, which is an intra-frame prediction, a quantization matrix corresponding to inter-prediction, which is an inter-frame prediction, and a quantization matrix corresponding to the above-mentioned mixed intra-inter prediction. Each quantization matrix stored by the storage unit 103 may be a quantization matrix having default element values, or may be a quantization matrix generated by the control unit 150 in response to a user operation. Each quantization matrix stored by the storage unit 103 may also be a quantization matrix generated by the control unit 150 in response to characteristics of the input image (such as the amount of edges and frequencies contained in the input image).

[0015] The prediction unit 104 divides each basic block into a plurality of sub-blocks. Then, for each sub-block, the prediction unit 104 obtains a predicted image using intra prediction, inter prediction, or mixed intra-inter prediction, and calculates the difference between the sub-block and the predicted image as a prediction error. The prediction unit 104 also generates prediction information, including information indicating a division method for the basic block, a prediction mode indicating a prediction for obtaining a predicted image of the sub-block, and information required for prediction, such as a motion vector.

[0016] The transform / quantization unit 105 generates orthogonal transform coefficients for each sub-block by performing an orthogonal transform (frequency transform) on the prediction errors of the sub-blocks obtained by the prediction unit 104, obtains from the storage unit 103 a quantization matrix corresponding to the prediction (intra prediction, inter prediction, or mixed intra-inter prediction) performed by the prediction unit 104 to obtain a predicted image of the sub-block, and quantizes the orthogonal transform coefficients using the obtained quantization matrix to generate quantized coefficients for the sub-block (the quantization results of the orthogonal transform coefficients).

[0017] The inverse quantization and inverse transform unit 106 generates orthogonal transform coefficients by inverse quantizing the quantized coefficients of each sub-block generated by the transform and quantization unit 105 using the quantization matrix used by the transform and quantization unit 105 to generate the quantized coefficients, and then performs inverse orthogonal transform on the orthogonal transform coefficients to generate (reproduce) prediction errors.

[0018] The image reproduction unit 107 generates a predicted image from the image stored in the frame memory 108 based on the prediction information generated by the prediction unit 104, and reproduces an image from the predicted image and the prediction error generated by the inverse quantization and inverse transform unit 106. The image reproduction unit 107 then stores the reproduced image in the frame memory 108. The image stored in the frame memory 108 is used as a reference image when the prediction unit 104 makes a prediction for an image of the current frame or the next frame.

[0019] The in-loop filter unit 109 performs in-loop filtering such as deblocking filtering and sample adaptive offset on the image stored in the frame memory 108 .

[0020] The encoding unit 110 encodes the quantization coefficients generated by the transform / quantization unit 105 and the prediction information generated by the prediction unit 104 to generate encoded data (coded data).

[0021] The encoding unit 113 encodes the quantization matrices (including at least the quantization matrices used in quantization by the transform / quantization unit 105) held in the holding unit 103 to generate encoded data (code data).

[0022] The integrated encoding unit 111 generates header code data using the encoded data generated by the encoding unit 113, and generates and outputs a bit stream including the encoded data generated by the encoding unit 110 and the header code data.

[0023] The output destination of the bitstream is not limited to a specific destination, and for example, the bitstream may be output to a memory included in the image encoding device, may be output to an external device via a network to which the image encoding device is connected, or may be transmitted to an external device for broadcasting.

[0024] Next, the operation of the image encoding device according to this embodiment will be described. First, encoding of an input image will be described. The division unit 102 divides the input image into a plurality of basic blocks and outputs each of the divided basic blocks.

[0025] The prediction unit 104 divides each basic block into a plurality of sub-blocks. Figures 7(a) to 7(f) show an example of a method for dividing a basic block 700 into sub-blocks.

[0026] Figure 7(a) shows an 8x8 pixel basic block 700 (=subblock) that is not divided into subblocks. Figure 7(b) shows an example of conventional square subblock division, in which the 8x8 pixel basic block 700 is divided into four 4x4 pixel subblocks (quadtree division).

[0027] Figures 7(c) to (f) show examples of rectangular sub-block division. In Figure 7(c), an 8 pixel x 8 pixel basic block 700 is divided into two 4 pixel (horizontal) x 8 pixel (vertical) sub-blocks (binary tree division). In Figure 7(d), an 8 pixel x 8 pixel basic block 700 is divided into two 8 pixel (horizontal) x 4 pixel (vertical) sub-blocks (binary tree division).

[0028] In Fig. 7(e), an 8 pixel x 8 pixel basic block 700 is divided into three sub-blocks: a 2 pixel (horizontal) x 8 pixel (vertical) sub-block, a 4 pixel (horizontal) x 8 pixel (vertical) sub-block, and a 2 pixel (horizontal) x 8 pixel (vertical) sub-block. In other words, in Fig. 7(e), the basic block 700 is divided into sub-blocks with a width (horizontal length) ratio of 1:2:1 (ternary tree division).

[0029] In Fig. 7(f), an 8 pixel x 8 pixel basic block 700 is divided into three sub-blocks: an 8 pixel (horizontal) x 2 pixel (vertical) sub-block, an 8 pixel (horizontal) x 4 pixel (vertical) sub-block, and an 8 pixel (horizontal) x 2 pixel (vertical) sub-block. In other words, in Fig. 7(f), the basic block 700 is divided into sub-blocks with a height (vertical length) ratio of 1:2:1 (ternary tree division).

[0030] In this way, in this embodiment, encoding processing is performed using not only square but also rectangular sub-blocks. Then, in this embodiment, prediction information including information indicating a division method of such basic blocks is generated. Note that the division method shown in FIG. 7 is merely an example, and the method of dividing basic blocks into sub-blocks is not limited to the division method shown in FIG. 7.

[0031] The prediction unit 104 then determines a prediction (prediction mode) to be performed for each sub-block. The prediction unit 104 then generates a predicted image for each sub-block based on the prediction mode determined for that sub-block and the coded pixels, and calculates the difference between that sub-block and the predicted image as a prediction error. The prediction unit 104 also generates "information necessary for prediction," such as information indicating a division method for basic blocks, prediction modes for sub-blocks, and motion vectors, as prediction information.

[0032] Here, the prediction used in this embodiment will be explained again. In this embodiment, three types of prediction (prediction modes) are used: intra prediction, inter prediction, and mixed intra-inter prediction.

[0033] In intra prediction (first prediction mode), predicted pixels of a block to be coded (sub-block in this embodiment) are generated using coded pixels located spatially around the block. In other words, in intra prediction, predicted pixels (predicted image) of the block to be coded are generated using coded pixels in a frame (image) including the block to be coded. For a sub-block on which intra prediction has been performed, information indicating an intra prediction method such as horizontal prediction, vertical prediction, or DC prediction is generated as "information necessary for prediction."

[0034] In inter prediction (second prediction mode), predicted pixels of a current block (sub-block in this embodiment) are generated using coded pixels of a frame (another image) that is different (in terms of time) from the frame (image) to which the current block belongs. For a sub-block that has undergone inter prediction, motion information indicating a reference frame, a motion vector, etc. is generated as "information necessary for prediction."

[0035] In mixed intra-inter prediction (third prediction mode), a current block (sub-block in this embodiment) is first divided along a diagonal line into two partitioned regions. Then, predicted pixels for one of the partitioned regions are obtained as predicted pixels for the current block by intra prediction. Similarly, predicted pixels for the other partitioned region are obtained as predicted pixels for the other partitioned region by inter prediction. In other words, the predicted pixels for one partitioned region of a predicted image obtained by mixed intra-inter prediction for the current block are predicted pixels for the current block by intra prediction. Similarly, the predicted pixels for the other partitioned region of a predicted image obtained by mixed intra-inter prediction for the current block are predicted pixels for the other partitioned region by inter prediction. An example of dividing a current block in mixed intra-inter prediction is shown in FIG. 12.

[0036] As shown in FIG. 12(a), suppose that a current block 1200 to be coded is divided into divided regions 1200a and 1200b by a line segment passing through the upper left vertex and the lower right vertex of the current block 1200. The mixed intra-inter prediction process performed by the prediction unit 104 on the current block 1200 in this case will be described with reference to FIGS. 12(a) to 12(d). At this time, the prediction unit 104 performs intra prediction on the current block 1200 to generate an intra-predicted image 1201 (FIG. 12(b)). The intra-predicted image 1201 includes a region 1201a that is located at the same position as the divided region 1200a and a region 1201b that is located at the same position as the divided region 1200b. The prediction unit 104 then determines, among the pixels (intra-predicted pixels) included in the intra-predicted image 1201, the intra-predicted pixels that belong to a region 1201a that is in the same position as the divided region 1200a as the "predicted pixels of the divided region 1200a." The prediction unit 104 also generates an inter-predicted image 1202 ( FIG. 12(c) ) by performing inter prediction on the current block 1200. Here, the inter-predicted image 1202 includes a region 1202a that is in the same position as the divided region 1200a and a region 1202b that is in the same position as the divided region 1200b. The prediction unit 104 then determines, among the pixels (inter-predicted pixels) included in the inter-predicted image 1202, the inter-predicted pixels that belong to a region 1202b that is in the same position as the divided region 1200b as the "predicted pixels of the divided region 1200b." Then, the prediction unit 104 generates a predicted image 1203 (Figure 12(d)) consisting of intra-predicted pixels included in area 1201a, which are "predicted pixels for division area 1200a," and inter-predicted pixels included in area 1202b, which are "predicted pixels for division area 1200b."

[0037] As described above, the prediction unit 104 generates an intra-predicted image 1201 ( FIG. 12(b) ) by intra-prediction for the current block 1200 to be coded, and further generates an inter-predicted image 1202 ( FIG. 12(c) ) by inter-prediction for the current block 1200 to be coded. Then, the prediction unit 104 arranges, among the intra-predicted pixels in the intra-predicted image 1201, intra-predicted pixels at a position with coordinates (x, y) included in the region 1201a corresponding to the divided region 1200a, at the same coordinates (x, y) in the predicted image 1203. Furthermore, among the inter-predicted pixels in the inter-predicted image 1202, the prediction unit 104 arranges, among the inter-predicted pixels in the inter-predicted image 1202, inter-predicted pixels at a position with coordinates (x, y) included in the region 1202b corresponding to the divided region 1200b, at the same coordinates (x, y) in the predicted image 1203. In this way, the predicted image 1203 shown in FIG. 12(d) is generated.

[0038] 12(e) to 12(h), the intra-inter mixed prediction process performed by the prediction unit 104 on the current block 1200 will be described. In this example, as shown in FIG. 12(e), it is assumed that the current block 1200 is divided into divided regions 1200c and 1200d by dividing the current block 1200 along a line segment passing through the midpoint between the upper left corner and the lower left corner of the current block 1200 and the upper right corner of the current block 1200. In this case, the prediction unit 104 performs intra prediction on the current block 1200 to generate an intra-predicted image 1201 (FIG. 12(f)). The intra-predicted image 1201 includes a region 1201c that is located at the same position as the region 1200c and a region 1201d that is located at the same position as the region 1200d. Then, the prediction unit 104 determines, among the pixels (intra-predicted pixels) included in the intra-predicted image 1201, the intra-predicted pixels belonging to a region 1201c that is in the same position as the divided region 1200c as the "predicted pixels of the divided region 1200c." The prediction unit 104 also generates an inter-predicted image 1202 ( FIG. 12(g) ) by performing inter prediction on the current block 1200. Here, the inter-predicted image 1202 includes a region 1202c that is in the same position as the divided region 1200c and a region 1202d that is in the same position as the divided region 1200d. Then, the prediction unit 104 determines, among the pixels (inter-predicted pixels) included in the inter-predicted image 1202, the inter-predicted pixels that belong to a region 1202d that is in the same position as the divided region 1200d as the "predicted pixels of the divided region 1200d." Then, the prediction unit 104 generates a predicted image 1203 (Figure 12(h)) consisting of intra-predicted pixels included in area 1201c, which is the "predicted pixels of division area 1200c," and inter-predicted pixels included in area 1202d, which is the "predicted pixels of division area 1200d."

[0039] As described above, the prediction unit 104 generates an intra-predicted image 1201 (FIG. 12(f)) by intra-prediction of the current block 1200 to be coded, and further generates an inter-predicted image 1202 (FIG. 12(g)) by inter-prediction of the current block 1200 to be coded. Then, the prediction unit 104 arranges, among the intra-predicted pixels in the intra-predicted image 1201, intra-predicted pixels located at coordinates (x, y) included in an area 1201c corresponding to the divided area 1200c, at the same coordinates (x, y) in the predicted image 1203. Furthermore, the prediction unit 104 arranges, among the inter-predicted pixels in the inter-predicted image 1202, inter-predicted pixels located at coordinates (x, y) included in an area 1202d corresponding to the divided area 1200d, at the same coordinates (x, y) in the predicted image 1203. By performing the above steps, the predicted image 1203 shown in FIG. 12(d) is generated.

[0040] For sub-blocks that have undergone mixed intra-inter prediction, information indicating the intra prediction method, motion information indicating the reference frame and motion vector, information specifying the division area (for example, information specifying the line segment mentioned above), etc. are generated as "information necessary for prediction."

[0041] The prediction unit 104 determines the prediction mode of the sub-block of interest by the following process: The prediction unit 104 generates a difference image between the sub-block of interest and a predicted image generated by intra-prediction for the sub-block of interest. The prediction unit 104 also generates a difference image between the sub-block of interest and a predicted image generated by inter-prediction for the sub-block of interest. The prediction unit 104 also generates a difference image between the sub-block of interest and a predicted image generated by mixed intra-inter prediction for the sub-block of interest. Note that the pixel value at pixel position (x, y) in difference image C between images A and B is the difference between pixel value AA at pixel position (x, y) in image A and pixel value BB at pixel position (x, y) in image B (such as the absolute value of the difference between AA and BB or the squared value of the difference between AA and BB). The prediction unit 104 then identifies a predicted image with the smallest total value of pixel values ​​of all pixels in the difference image, and determines the prediction performed on the target sub-block to obtain the predicted image as the “prediction mode of the target sub-block.” Note that the method for determining the prediction mode of the target sub-block is not limited to the above method.

[0042] Then, for each sub-block, the prediction unit 104 sets the predicted image generated in the prediction mode determined for that sub-block as the "predicted image of that sub-block," and generates a prediction error from that sub-block and the predicted image. Furthermore, the prediction unit 104 generates prediction information for each sub-block, including the prediction mode determined for that sub-block and the "information necessary for prediction" generated for that sub-block.

[0043] The transform / quantization unit 105 performs an orthogonal transform process on the prediction error of each sub-block according to the size of the prediction error to generate orthogonal transform coefficients. Then, for each sub-block, the transform / quantization unit 105 obtains a quantization matrix corresponding to the prediction mode of the sub-block from the quantization matrices stored in the storage unit 103, and quantizes the orthogonal transform coefficients of the sub-block using the obtained quantization matrix to generate quantized coefficients.

[0044] For example, it is assumed that the storage unit 103 stores an 8×8 quantization matrix (where the values ​​of the 64 elements are all quantization step values) illustrated in FIG. 8(a) as a quantization matrix used to quantize orthogonal transform coefficients of prediction errors obtained when intra-prediction is performed on an 8×8 pixel sub-block. Also, it is assumed that the storage unit 103 stores an 8×8 quantization matrix (where the values ​​of the 64 elements are all quantization step values) illustrated in FIG. 8(b) as a quantization matrix used to quantize orthogonal transform coefficients of prediction errors obtained when inter-prediction is performed on an 8×8 pixel sub-block. Also, it is assumed that the storage unit 103 stores an 8×8 quantization matrix (where the values ​​of the 64 elements are all quantization step values) illustrated in FIG. 8(c) as a quantization matrix used to quantize orthogonal transform coefficients of prediction errors obtained when mixed intra-inter prediction is performed on an 8×8 pixel sub-block.

[0045] In this case, the transform / quantization unit 105 quantizes the orthogonal transform coefficients of the "prediction error obtained by intra prediction for an 8 pixel x 8 pixel sub-block" using the quantization matrix for intra prediction in FIG. 8(a).

[0046] Furthermore, the transform / quantization unit 105 quantizes the orthogonal transform coefficients of the "prediction error obtained by inter prediction for the 8 pixel x 8 pixel sub-block" using the quantization matrix for inter prediction in FIG. 8(b).

[0047] The transform / quantization unit 105 also quantizes the orthogonal transform coefficients of the "prediction error obtained by mixed intra / inter prediction for an 8 pixel x 8 pixel sub-block" using the quantization matrix for mixed intra / inter prediction in Figure 8(c).

[0048] The inverse quantization and inverse transform unit 106 generates orthogonal transform coefficients by inverse quantizing the quantized coefficients of each sub-block generated by the transform and quantization unit 105 using the quantization matrix used by the transform and quantization unit 105 in quantizing the sub-block, and then performs inverse orthogonal transform on the orthogonal transform coefficients to generate (reproduce) prediction errors.

[0049] The image reproduction unit 107 generates a predicted image from the image stored in the frame memory 108 based on the prediction information generated by the prediction unit 104, and regenerates an image of the sub-block by adding the predicted image to the prediction error generated (regenerated) by the inverse quantization and inverse transform unit 106. The image reproduction unit 107 then stores the regenerated image in the frame memory 108.

[0050] The in-loop filter unit 109 performs in-loop filtering such as deblocking filtering and sample adaptive offset on the image stored in the frame memory 108, and stores the image after the in-loop filtering in the frame memory 108.

[0051] The encoding unit 110 generates encoded data for each sub-block by entropy encoding the quantization coefficients of the sub-block generated by the transform / quantization unit 105 and the prediction information of the sub-block generated by the prediction unit 104. Note that the entropy encoding method is not particularly specified, but Golomb encoding, arithmetic encoding, Huffman encoding, etc. can be used.

[0052] Next, the coding of quantization matrices will be described. The quantization matrices stored in the storage unit 103 are generated according to the size and prediction mode of the sub-blocks to be coded. For example, as shown in FIG. 7, when sizes such as 8 pixels x 8 pixels, 4 pixels x 4 pixels, 8 pixels x 4 pixels, 4 pixels x 8 pixels, 8 pixels x 2 pixels, and 2 pixels x 8 pixels are adopted as the size of the sub-blocks to be divided, a quantization matrix of the adopted size is registered in the storage unit 103. Furthermore, quantization matrices are prepared for each of intra prediction, inter prediction, and mixed intra-inter prediction, and are registered in the storage unit 103.

[0053] The method of generating such a quantization matrix according to the size of the sub-block and the prediction mode is not limited to the specific generation method as described above, and the method of managing such a quantization matrix in the storage unit 103 is not limited to a specific management method.

[0054] In this embodiment, the quantization matrices stored in storage unit 103 are assumed to be stored in a two-dimensional form as shown in Fig. 8, but the elements in the quantization matrices are not limited to this. It is also possible to store multiple quantization matrices for the same prediction method depending on the size of the sub-block or whether the encoding target is a luminance block or a chrominance block. Generally, since a quantization matrix realizes quantization processing in accordance with human visual characteristics, as shown in Fig. 8, the elements for DC components corresponding to the upper left corner of the quantization matrix are small, and the elements for AC components corresponding to the lower right corner are large.

[0055] The encoding unit 113 reads out the quantization matrices (including at least the quantization matrices used for quantization by the transform / quantization unit 105) stored in the storage unit 103, and encodes the read out quantization matrices. For example, the encoding unit 113 encodes the quantization matrix of interest by the following process.

[0056] The encoding unit 113 references the value of each element in the quantization matrix of interest, which is a two-dimensional array, in a specified order, and generates a one-dimensional array in which the difference values ​​between the value of the element currently being referenced and the value of the element referenced immediately before are arranged. For example, if the quantization matrix of interest is the quantization matrix of interest in (c) of Fig. 8, the encoding unit 113 references the values ​​of each element from the value of the element in the upper left corner to the value of the element in the lower right corner of the quantization matrix of interest in the order indicated by the arrows, as shown in Fig. 9.

[0057] In this case, the value of the first referenced element is "8" and the value of the element referenced immediately before does not exist, so the encoding unit 113 outputs a predetermined value or a value determined by some method as the output value. For example, the encoding unit 113 may output the value "8" of the currently referenced element as the output value, or may output a value obtained by subtracting a predetermined value from the value "8" of the element as the output value, or the output value does not have to be a value determined by a specific method.

[0058] Because the value of the next element to be referenced is "11" and the value of the element referenced immediately before is "8", encoding unit 113 outputs the difference value "+3" obtained by subtracting the value of the element referenced immediately before, "8", from the value of the currently referenced element, "11". In this way, encoding unit 113 references the values ​​of each element in the quantization matrix in a specified order to determine and output an output value, and generates a one-dimensional array in which the output values ​​are arranged in output order.

[0059] FIG. 10(a) shows a one-dimensional array generated from the quantization matrix of FIG. 8(a) by such processing. FIG. 10(b) shows a one-dimensional array generated from the quantization matrix of FIG. 8(b) by such processing. FIG. 10(c) shows a one-dimensional array generated from the quantization matrix of FIG. 8(c) by such processing. In FIG. 10, it is assumed that the above specified value is set to "8."

[0060] The encoding unit 113 then encodes the one-dimensional array generated for the quantization matrix of interest. For example, the encoding unit 113 references the encoding table illustrated in Fig. 11(a) and generates, as encoded data, a bit string in which each element value in the one-dimensional array is replaced with a corresponding binary code. Note that the encoding table is not limited to the encoding table illustrated in Fig. 11(a), and for example, the encoding table illustrated in Fig. 11(b) may also be used.

[0061] 1, the integrating coding unit 111 integrates "header information necessary for image coding" with the coded data generated by the coding unit 113, and generates header coded data using the coded data with the integrated header information. The integrating coding unit 111 then generates and outputs a bit stream in which the coded data generated by the coding unit 110 and the header coded data are multiplexed.

[0062] FIG. 6(a) shows an example of the structure of a bitstream generated by the integrated encoding unit 111. The sequence header contains coded data of the quantization matrix and includes coded data of each element. However, the coding position is not limited to this, and coding may be performed in a picture header or other header. Furthermore, when changing the quantization matrix within a single sequence, it is also possible to update the quantization matrix by newly coding it. In this case, all quantization matrices may be rewritten, or it is also possible to change part of them by specifying a prediction mode of the quantization matrix corresponding to the quantization matrix to be rewritten.

[0063] The encoding process performed by the image encoding device described above will be described with reference to the flowchart in Fig. 3. Note that the process according to the flowchart in Fig. 3 is an encoding process for one input image. Therefore, when encoding an image of each frame in a moving image or a plurality of images captured periodically or irregularly, the processes of steps S304 to S311 are repeated for each image.

[0064] 3, it is assumed that a quantization matrix corresponding to intra prediction, a quantization matrix corresponding to inter prediction, and a quantization matrix corresponding to mixed intra-inter prediction are already registered in the storage unit 103. As described above, all of the quantization matrices stored in the storage unit 103 are quantization matrices corresponding to the size of the sub-blocks to be divided.

[0065] In step S302, the encoding unit 113 reads the quantization matrices (including at least the quantization matrices used by the transform / quantization unit 105 for quantization) stored in the storage unit 103, and encodes the read quantization matrices to generate encoded data.

[0066] In step S303, the integrated encoding unit 111 generates "header information necessary for encoding the image." Then, the integrated encoding unit 111 integrates the "header information necessary for encoding the image" with the encoded data generated by the encoding unit 113 in step S302, and generates header encoded data using the encoded data with the integrated header information.

[0067] In step S304, the division unit 102 divides the input image into a plurality of basic blocks and outputs each of the divided basic blocks. The prediction unit 104 then divides each basic block into a plurality of sub-blocks.

[0068] In step S305, the prediction unit 104 selects one of the unselected sub-blocks in the input image as a selected sub-block, determines a prediction mode for the selected sub-block, and performs prediction on the selected sub-block according to the determined prediction mode to obtain a predicted image, prediction error, and prediction information for the selected sub-block.

[0069] In step S306, the transform / quantization unit 105 performs orthogonal transform processing corresponding to the size of the prediction error on the prediction error of the selected sub-block acquired in step S305 to generate orthogonal transform coefficients.The transform / quantization unit 105 then acquires a quantization matrix corresponding to the prediction mode of the selected sub-block from the quantization matrices stored in the storage unit 103, and quantizes the orthogonal transform coefficients of the sub-block using the acquired quantization matrix to acquire quantization coefficients.

[0070] In step S307, the inverse quantization and inverse transform unit 106 generates orthogonal transform coefficients by inverse quantizing the quantized coefficients of the selected sub-block acquired in step S306 using the quantization matrix used for quantization of the selected sub-block by the transform and quantization unit 105. The inverse quantization and inverse transform unit 106 then performs inverse orthogonal transform on the generated orthogonal transform coefficients to generate (reproduce) prediction errors.

[0071] In step S308, the image reproduction unit 107 generates a predicted image from the image stored in the frame memory 108 based on the prediction information acquired in step S305, and regenerates the image of the sub-block by adding the predicted image to the prediction error generated in step S307. The image reproduction unit 107 then stores the regenerated image in the frame memory 108.

[0072] In step S309, the encoding unit 110 performs entropy encoding on the quantized coefficients obtained in step S306 and the prediction information obtained in step S305 to generate encoded data.

[0073] The integrated encoding unit 111 then generates and outputs a bit stream by multiplexing the header encoded data generated in step S303 and the encoded data generated by the encoding unit 110 in step S309.

[0074] In step S310, control unit 150 determines whether all sub-blocks in the input image have been selected as selected sub-blocks. If the result of this determination is that all sub-blocks in the input image have been selected as selected sub-blocks, the process proceeds to step S311. On the other hand, if one or more sub-blocks in the input image remain that have not yet been selected as selected sub-blocks, the process proceeds to step S305.

[0075] In step S311, the in-loop filter unit 109 performs in-loop filtering on the image (the image of the selected sub-block reproduced in step S308) stored in the frame memory 108. Then, the in-loop filter unit 109 stores the image that has undergone the in-loop filtering in the frame memory 108.

[0076] By using this type of processing, the orthogonal transform coefficients of sub-blocks that have undergone intra- and inter-mixed prediction can be quantized using a quantization matrix that corresponds to intra- and inter-mixed prediction, thereby controlling quantization for each frequency component and improving image quality.

[0077] <Modification> In the first embodiment, separate quantization matrices are prepared for intra prediction, inter prediction, and mixed intra-inter prediction, and the quantization matrices corresponding to each prediction are coded. However, some of these matrices may be shared.

[0078] For example, a quantization matrix corresponding to intra prediction may be used to quantize the orthogonal transform coefficients of the prediction error obtained based on intra-inter mixed prediction, rather than a quantization matrix corresponding to intra-inter mixed prediction. That is, for example, to quantize the orthogonal transform coefficients of the prediction error obtained based on intra-inter mixed prediction, the quantization matrix for intra prediction shown in FIG. 8(a) may be used instead of the quantization matrix for intra-inter mixed prediction shown in FIG. 8(c). In this case, coding of the quantization matrix corresponding to intra-inter mixed prediction can be omitted. This reduces the amount of coded data of the quantization matrix included in the bitstream and alleviates image quality degradation caused by errors due to intra prediction, such as block distortion.

[0079] Furthermore, a quantization matrix corresponding to inter prediction may be used to quantize the orthogonal transform coefficients of the prediction error obtained based on intra-inter mixed prediction, rather than a quantization matrix corresponding to intra-inter mixed prediction. That is, for example, to quantize the orthogonal transform coefficients of the prediction error obtained based on intra-inter mixed prediction, the quantization matrix for inter prediction shown in FIG. 8(b) may be used instead of the quantization matrix for intra-inter mixed prediction shown in FIG. 8(c). In this case, coding of the quantization matrix corresponding to intra-inter mixed prediction can be omitted. This not only reduces the amount of coded data of the quantization matrix included in the bitstream, but also reduces image quality degradation caused by errors due to inter prediction, such as jerky motion.

[0080] In addition, in the predicted image of a sub-block obtained by performing mixed intra-inter prediction, the quantization matrix to be used for the sub-block may be determined depending on the respective sizes of the area of ​​"predicted pixels obtained by intra prediction" and the area of ​​"predicted pixels obtained by inter prediction."

[0081] For example, suppose that the sub-block 1200 is divided into a division region 1200c and a division region 1200d as shown in Figure 12(e). Then, suppose that "predicted pixels obtained by intra prediction" are obtained as predicted pixels for the division region 1200c, and "predicted pixels obtained by inter prediction" are obtained as predicted pixels for the division region 1200d. Also, suppose that the size (area (number of pixels)) S1 of the division region 1200c: the size (area (number of pixels)) S2 of the division region 1200d = 1:3.

[0082] In this case, the size of the divided region 1200d to which inter prediction is applied is larger than the size of the divided region 1200c to which intra prediction is applied in the sub-block 1200. Therefore, the transform / quantization unit 105 applies a quantization matrix corresponding to inter prediction (for example, the quantization matrix in FIG. 8(b)) to quantize the orthogonal transform coefficients of this sub-block 1200.

[0083] Note that if the size of the divided region 1200d to which inter prediction has been applied is smaller than the size of the divided region 1200c to which intra prediction has been applied in the sub-block 1200, the transform / quantization unit 105 applies a quantization matrix corresponding to intra prediction (e.g., the quantization matrix in FIG. 8(a)) to quantize the orthogonal transform coefficients of the sub-block 1200. This reduces image quality degradation in larger divided regions while omitting the coding of the quantization matrix corresponding to mixed intra / inter prediction. This makes it possible to reduce the amount of coded data of the quantization matrix to be included in the bitstream.

[0084] Alternatively, a quantization matrix corresponding to intra-prediction may be generated by combining a "quantization matrix corresponding to intra-prediction" and a "quantization matrix corresponding to inter-prediction" in accordance with the ratio of S1 to S2. For example, the transform / quantization unit 105 may generate a quantization matrix corresponding to intra-inter-mixed prediction using the following equation (1):

[0085] QM[x][y]={w×QMinter[x][y]+(1-w)×QMintra[x][y]})…(1) Here, QM[x][y] indicates the value of the element (quantization step value) at coordinates (x, y) in a quantization matrix corresponding to intra-inter mixed prediction. QMinter[x][y] indicates the value of the element (quantization step value) at coordinates (x, y) in a quantization matrix corresponding to inter prediction. QMintra[x][y] indicates the value of the element (quantization step value) at coordinates (x, y) in a quantization matrix corresponding to intra prediction. Furthermore, w is a value between 0 and 1, indicating the proportion of the area in the subblock where inter prediction is used, where w = S2 / (S1 + S2). In this way, a quantization matrix corresponding to intra-inter mixed prediction can be generated as needed and does not need to be created in advance, so encoding of the quantization matrix can be omitted. This reduces the amount of encoded data of the quantization matrix included in the bitstream. Furthermore, appropriate quantization control can be performed according to the ratio between the sizes of the areas where intra prediction and inter prediction are used, thereby improving image quality.

[0086] Furthermore, in the first embodiment, the quantization matrix to be applied to a sub-block to which mixed intra-inter prediction is applied is configured to be uniquely determined, but it may also be configured to be selectable by introducing an identifier.

[0087] There are various methods for selecting the quantization matrix to be applied to a sub-block to which intra-inter mixed prediction is applied from among a quantization matrix corresponding to intra prediction, a quantization matrix corresponding to inter prediction, and a quantization matrix corresponding to intra-inter mixed prediction. For example, the control unit 150 may select the quantization matrix in response to a user operation.

[0088] Then, an identifier for identifying the quantization matrix selected as the quantization matrix to be applied to the sub-block to which intra-inter mixed prediction is applied is stored in the bitstream.

[0089] For example, FIG. 6(b) shows an example in which a quantization matrix encoding method information code is newly introduced as the identifier to selectively select a quantization matrix to be applied to a sub-block to which intra-inter mixed prediction is applied. For example, when the quantization matrix encoding method information code indicates 0, this indicates that a quantization matrix corresponding to intra-prediction has been applied to a sub-block to which intra-inter mixed prediction is applied. When the quantization matrix encoding method information code indicates 1, this indicates that a quantization matrix corresponding to inter-prediction has been applied to a sub-block to which intra-inter mixed prediction is applied. On the other hand, when the quantization matrix encoding method information code indicates 2, this indicates that a quantization matrix corresponding to intra-inter mixed prediction has been applied to a sub-block to which intra-inter mixed prediction is applied.

[0090] This makes it possible to selectively reduce the amount of coded data for quantization matrices included in the bitstream and to achieve unique quantization control for sub-blocks to which mixed intra- and inter-prediction is applied.

[0091] In the first embodiment, a predicted image including predicted pixels (first predicted pixels) for one division area obtained by dividing a sub-block and predicted pixels (second predicted pixels) for the other division area is generated. However, the method for generating a predicted image is not limited to this. For example, to improve the image quality of a region near the boundary between one division area and the other division area (boundary area), a third predicted pixel calculated by a weighted average of the first predicted pixel and the second predicted pixel included in the boundary area may be used as the predicted pixel for the boundary area. In this case, the predicted pixel value of the corresponding region corresponding to the one division area in the predicted image is the first predicted pixel, and the predicted pixel value of the corresponding region corresponding to the other division area in the predicted image is the second predicted pixel. The predicted pixel value of the corresponding region corresponding to the boundary area in the predicted image is the third predicted pixel. This makes it possible to suppress deterioration in image quality in the boundary area between division areas in which different predictions are used, thereby improving image quality.

[0092] Furthermore, in the first embodiment, three types of prediction, i.e., intra prediction, inter prediction, and intra-inter hybrid prediction, have been described as examples. However, the types and number of predictions are not limited to these examples. For example, intra-inter hybrid prediction (CIIP) adopted in VVC may be used. Intra-inter hybrid prediction is a prediction in which pixels of the entire block to be coded are calculated by a weighted average of predicted pixels by intra prediction and predicted pixels by inter prediction. In this case, the quantization matrix used for sub-blocks using intra-inter hybrid prediction and the quantization matrix used for sub-blocks using intra-inter hybrid prediction may be made the same. This allows quantization using a quantization matrix with the same quantization control properties to be applied to sub-blocks using predictions that have in common the use of predicted pixels by intra prediction and predicted pixels by inter prediction within the same sub-block. Furthermore, it is possible to reduce the amount of code by the amount of the quantization matrix corresponding to the new prediction method.

[0093] In the first embodiment, the encoding target is an input image, but the encoding target is not limited to an image. For example, a two-dimensional data array, which is feature data used in machine learning such as object recognition, may be encoded in the same manner as the input image, and a bit stream may be generated and output. This makes it possible to efficiently encode feature data used in machine learning.

[0094] [Second embodiment] The image decoding device according to this embodiment decodes quantization coefficients for a block to be decoded from a bitstream, derives transform coefficients from the quantization coefficients using a quantization matrix, and performs inverse frequency transform on the transform coefficients to derive prediction errors for the block to be decoded. The image decoding device then generates a predicted image by applying an intra-predicted image obtained by intra-prediction to some regions of the block to be decoded, and applying an inter-predicted image obtained by inter-prediction to other regions different from the some regions of the block to be decoded, and decodes the block to be decoded using the generated predicted image and the prediction error.

[0095] In this embodiment, an image decoding device that decodes a bitstream coded by the image coding device according to Embodiment 1 will be described. First, an example of the functional configuration of the image decoding device according to this embodiment will be described with reference to the block diagram in FIG.

[0096] The control unit 250 controls the operation of the entire image decoding device. The separate decoding unit 202 acquires a bitstream encoded by the image encoding device according to the first embodiment. The bitstream may be acquired in any manner. For example, the bitstream output from the image encoding device according to the first embodiment may be acquired via a network, or from a memory in which the bitstream was once stored. The separate decoding unit 202 then separates the acquired bitstream into coded data related to information about the decoding process and coefficients, and decodes the coded data present in the header portion of the bitstream. In this embodiment, the separate decoding unit 202 separates coded data of a quantization matrix from the bitstream and supplies the coded data to the decoding unit 209. The separate decoding unit 202 also separates coded data of an input image from the bitstream and supplies the coded data to the decoding unit 203. In other words, the separate decoding unit 202 performs the reverse operation of the integrated coding unit 111 in FIG. 1.

[0097] The decoding unit 209 reproduces the quantization matrix by decoding the coded data supplied from the separate decoding unit 202. The decoding unit 203 decodes the coded data supplied from the separate decoding unit 202 to reproduce the quantization coefficients and prediction information.

[0098] The inverse quantization and inverse transform unit 204 performs the same operation as the inverse quantization and inverse transform unit 106 included in the image coding device according to the first embodiment. The inverse quantization and inverse transform unit 204 selects a quantization matrix corresponding to a prediction corresponding to a quantization coefficient to be decoded from the quantization matrices decoded by the decoding unit 209, and inverse quantizes the quantization coefficient using the selected quantization matrix to reproduce orthogonal transform coefficients. The inverse quantization and inverse transform unit 204 then performs inverse orthogonal transform on the reproduced orthogonal transform coefficients to reproduce prediction errors.

[0099] The image reproduction unit 205 generates a predicted image by referring to the image stored in the frame memory 206 based on the prediction information decoded by the decoding unit 203. The image reproduction unit 205 then generates a reproduced image by adding the prediction error obtained by the inverse quantization and inverse transform unit 204 to the generated predicted image, and stores the generated reproduced image in the frame memory 206.

[0100] The in-loop filter unit 207 performs in-loop filtering such as deblocking filtering and sample adaptive offset on the reconstructed image stored in the frame memory 206. The reconstructed image stored in the frame memory 206 is output as appropriate by the control unit 250. The output destination of the reconstructed image is not limited to a specific output destination, and the reconstructed image may be displayed on the display screen of a display device such as a monitor, or may be output to a projection device such as a projector.

[0101] Next, the operation (bitstream decoding process) of the image decoding device having the above configuration will be described. The separate decoding unit 202 obtains the bitstream generated by the image encoding device, separates from the bitstream coded data related to the decoding process and coefficients, and decodes the coded data present in the bitstream header. The separate decoding unit 202 extracts coded data of the quantization matrix from the bitstream sequence header of FIG. 6(a), and supplies the extracted coded data to the decoding unit 209. The separate decoding unit 202 also supplies coded data of picture data in sub-block units to the decoding unit 203.

[0102] The decoding unit 209 decodes the coded data of the quantization matrix supplied from the separation decoding unit 202 to reconstruct a one-dimensional array. More specifically, the decoding unit 209 references the coding tables exemplified in FIG. 11(a) or FIG. 11(b) to generate a one-dimensional array in which the binary codes in the coded data of the quantization matrix are decoded into differential values ​​and arranged. For example, when the decoding unit 209 decodes the coded data of the quantization matrix of FIG. 8(a) to FIG. 8(c), the one-dimensional arrays of FIG. 10(a) to FIG. 10(c) are reconstructed. In this embodiment, as in the first embodiment, decoding is performed using the coding table shown in FIG. 11(a) (or FIG. 11(b)). However, the coding table is not limited to this, and other coding tables may be used as long as they are the same as those in the first embodiment.

[0103] Furthermore, the decoding unit 209 reproduces each element value of the quantization matrix from each difference value of the reproduced one-dimensional array. This is to perform a process reverse to the process performed by the encoding unit 113 to generate a one-dimensional array from the quantization matrix. That is, the value of the element at the head of the one-dimensional array becomes the element value at the upper left corner of the quantization matrix. The value obtained by adding the value of the element at the head of the one-dimensional array to the value of the second element from the head of the one-dimensional array becomes the second element value in the above "prescribed order". The value obtained by adding the value of the (n - 1)-th element from the head of the one-dimensional array to the value of the n-th element (2 < n ≦ N: N is the number of elements of the one-dimensional array) from the head of the one-dimensional array becomes the n-th element value in the above "prescribed order". For example, the decoding unit 209 reproduces the quantization matrices of FIGS. 8(a) to (c) from the one-dimensional arrays of FIGS. 10(a) to (c) using the order shown in FIG. 9, respectively.

[0104] The decoding unit 203 decodes the quantization coefficients and prediction information by decoding the encoded data of the input image supplied from the separation decoding unit 202.

[0105] The inverse quantization / inverse transformation unit 204 identifies the "prediction mode corresponding to the quantization coefficient to be decoded" included in the prediction information decoded by the decoding unit 203, and selects the quantization matrix corresponding to the identified prediction mode from the quantization matrices reproduced by the decoding unit 209. Then, the inverse quantization / inverse transformation unit 204 inverse-quantizes the quantization coefficient using the selected quantization matrix to reproduce the orthogonal transformation coefficients. Then, the inverse quantization / inverse transformation unit 204 performs an inverse orthogonal transformation on the reproduced orthogonal transformation coefficients to reproduce the prediction error, and supplies the reproduced prediction error to the image reproduction unit 205.

[0106] The image reproduction unit 205 generates a predicted image by referencing an image stored in the frame memory 206 based on the prediction information decoded by the decoding unit 203. In this embodiment, three types of prediction are used: intra prediction, inter prediction, and mixed intra-inter prediction, as in the prediction unit 104 of the first embodiment. The specific prediction process is the same as that of the prediction unit 104 described in the first embodiment, and therefore description thereof will be omitted. The image reproduction unit 205 then generates a reproduced image by adding the prediction error obtained by the inverse quantization and inverse transform unit 204 to the generated predicted image, and stores the generated reproduced image in the frame memory 206. The reproduced image stored in the frame memory 206 becomes a prediction reference candidate to be referenced when decoding other sub-blocks.

[0107] The in-loop filter unit 207 operates in the same manner as the above-described in-loop filter unit 109, and performs in-loop filtering processes such as deblocking filtering and sample adaptive offset on the reconstructed image stored in the frame memory 206. The reconstructed image stored in the frame memory 206 is output by the control unit 250 as appropriate.

[0108] The decoding process in the image decoding device according to this embodiment will be described with reference to the flowchart in Fig. 4. In step S401, the separate decoding unit 202 acquires an encoded bitstream. The separate decoding unit 202 then separates encoded data of the quantization matrix from the acquired bitstream and supplies the encoded data to the decoding unit 209. The separate decoding unit 202 also separates encoded data of the input image from the bitstream and supplies the encoded data to the decoding unit 203.

[0109] In step S402, the decoding unit 209 reproduces the quantization matrix by decoding the coded data supplied from the separate decoding unit 202. In step S403, the decoding unit 203 decodes the coded data supplied from the separate decoding unit 202 to reproduce the quantization coefficients and prediction information of the sub-block to be decoded.

[0110] In step S404, the inverse quantization and inverse transform unit 204 identifies a "prediction mode corresponding to the quantization coefficients of the sub-block to be decoded" included in the prediction information decoded by the decoding unit 203. The inverse quantization and inverse transform unit 204 then selects a quantization matrix corresponding to the identified prediction mode from among the quantization matrices regenerated by the decoding unit 209. For example, if the prediction mode identified for the sub-block to be decoded is intra prediction, the quantization matrix for intra prediction shown in FIG. 8(a) is selected from the quantization matrices shown in FIGS. 8(a) to 8(c). If the prediction mode identified for the sub-block to be decoded is inter prediction, the quantization matrix for inter prediction shown in FIG. 8(b) is selected. If the prediction mode identified for the sub-block to be decoded is intra-inter mixed prediction, the quantization matrix for intra-inter mixed prediction shown in FIG. 8(c) is selected. The inverse quantization and inverse transform unit 204 then inverse quantizes the quantization coefficients of the sub-block to be decoded using the selected quantization matrix to regenerate orthogonal transform coefficients. The inverse quantization and inverse transform unit 204 then performs inverse orthogonal transform on the reconstructed orthogonal transform coefficients to reconstruct the prediction error of the sub-block to be decoded, and supplies the reconstructed prediction error to the image reconstruction unit 205.

[0111] In step S405, the image reproduction unit 205 generates a predicted image of the sub-block to be decoded by referring to the image stored in the frame memory 206 based on the prediction information decoded by the decoding unit 203. The image reproduction unit 205 then generates a reconstructed image of the sub-block to be decoded by adding the prediction error of the sub-block to be decoded, obtained by the inverse quantization and inverse transform unit 204, to the generated predicted image, and stores the generated reconstructed image in the frame memory 206.

[0112] In step S406, control unit 250 determines whether the processes of steps S403 to S405 have been performed for all sub-blocks. If the result of this determination is that the processes of steps S403 to S405 have been performed for all sub-blocks, the process proceeds to step S407. On the other hand, if there are still sub-blocks for which the processes of steps S403 to S405 have not been performed, the process proceeds to step S403 to perform the processes of steps S403 to S405 for those sub-blocks.

[0113] In step S407, the in-loop filter unit 207 performs in-loop filtering, such as deblocking filtering and sample adaptive offset, on the reconstructed image generated in step S405 and stored in the frame memory 206.

[0114] By this processing, it is possible to control quantization for each frequency component and decode a bitstream with improved image quality, even for sub-blocks generated using mixed intra-inter prediction in the first embodiment.

[0115] <Modification> In the second embodiment, separate quantization matrices are prepared for intra prediction, inter prediction, and mixed intra-inter prediction, and the quantization matrices corresponding to each prediction are decoded. However, some of these matrices may be shared.

[0116] For example, to dequantize the quantized coefficients of the orthogonal transform coefficients of the prediction error obtained based on intra-inter mixed prediction, a quantization matrix corresponding to intra prediction may be decoded and used instead of a quantization matrix corresponding to intra-inter mixed prediction. That is, for example, to dequantize the quantized coefficients of the orthogonal transform coefficients of the prediction error obtained based on intra-inter mixed prediction, the quantization matrix for intra prediction shown in FIG. 8(a) may be used. In this case, decoding of the quantization matrix corresponding to intra-inter mixed prediction can be omitted. That is, it is possible to decode a bitstream with a reduced amount of coded data of the quantization matrix included in the bitstream, and to obtain a decoded image with reduced image quality degradation caused by errors due to intra prediction, such as block distortion.

[0117] Furthermore, to dequantize the quantized coefficients of the orthogonal transform coefficients of the prediction error obtained based on intra-inter mixed prediction, a quantization matrix corresponding to inter prediction may be decoded and used instead of a quantization matrix corresponding to intra-inter mixed prediction. That is, for example, to dequantize the quantized coefficients of the orthogonal transform coefficients of the prediction error obtained based on intra-inter mixed prediction, the quantization matrix for inter prediction shown in FIG. 8(b) may be used. In this case, decoding of the quantization matrix corresponding to intra-inter mixed prediction can be omitted. In other words, it is possible to decode a bitstream with a reduced amount of coded data of the quantization matrix included in the bitstream, and to obtain a decoded image with reduced image quality degradation caused by errors due to inter prediction, such as jerky motion.

[0118] In addition, the quantization matrix to be used for inverse quantization of a subblock on which mixed intra-inter prediction has been performed may be determined depending on the respective sizes of the area of ​​"predicted pixels obtained by intra prediction" and the area of ​​"predicted pixels obtained by inter prediction" in the predicted image of the subblock on which mixed intra-inter prediction has been performed.

[0119] For example, suppose that the sub-block 1200 is divided into a division region 1200c and a division region 1200d as shown in Figure 12(e). Then, suppose that "predicted pixels obtained by intra prediction" are obtained as predicted pixels for the division region 1200c, and "predicted pixels obtained by inter prediction" are obtained as predicted pixels for the division region 1200d. Also, suppose that the size (area (number of pixels)) S1 of the division region 1200c: the size (area (number of pixels)) S2 of the division region 1200d = 1:3.

[0120] In this case, the size of the divided region 1200d to which inter prediction is applied is larger than the size of the divided region 1200c to which intra prediction is applied in the sub-block 1200. Therefore, the inverse quantization and inverse transform unit 204 applies a quantization matrix corresponding to inter prediction to inverse quantize the quantized coefficients of this sub-block 1200.

[0121] In addition, if the size of the divided area 1200d to which inter prediction is applied is smaller than the size of the divided area 1200c to which intra prediction is applied in the sub-block 1200, the inverse quantization / inverse transform unit 204 applies a quantization matrix corresponding to intra prediction to inverse quantize the quantized coefficients of the sub-block 1200.

[0122] This makes it possible to omit the decoding of quantization matrices corresponding to mixed intra-inter prediction while reducing the degradation of image quality in larger divided regions, thereby enabling decoding of a bitstream with a reduced amount of coded data of quantization matrices included in the bitstream.

[0123] Alternatively, a quantization matrix corresponding to mixed intra-inter prediction may be generated by combining a "quantization matrix corresponding to intra prediction" and a "quantization matrix corresponding to inter prediction" in accordance with the ratio of S1 to S2. For example, the inverse quantization and inverse transform unit 204 may generate a quantization matrix corresponding to mixed intra-inter prediction using the above formula (1).

[0124] In this way, quantization matrices corresponding to mixed intra- and inter-prediction can be generated as needed, so encoding of the quantization matrices can be omitted. This makes it possible to decode a bitstream with a reduced amount of encoded data for the quantization matrices included in the bitstream. Furthermore, it is possible to decode a bitstream with improved image quality by performing appropriate quantization control according to the ratio of the sizes of the regions where intra-prediction and inter-prediction are used.

[0125] Furthermore, in the second embodiment, the quantization matrix to be applied to a sub-block to which mixed intra-inter prediction is applied is uniquely determined, but similar to the first embodiment, a configuration in which the quantization matrix can be selected by introducing an identifier may be adopted. This makes it possible to decode a bitstream that selectively achieves a reduction in the amount of coded data of the quantization matrix to be included in the bitstream and unique quantization control for the sub-block to which mixed intra-inter prediction is applied.

[0126] Furthermore, in the second embodiment, a predicted image including predicted pixels (first predicted pixels) for one division area obtained by dividing a sub-block and predicted pixels (second predicted pixels) for the other division area is decoded. However, the predicted image to be decoded is not limited to this predicted image. For example, as in the modified example of the first embodiment, a predicted image may be one in which a third predicted pixel calculated by a weighted average of first predicted pixels and second predicted pixels included in an area near the boundary between one division area and the other division area (boundary area) is used as the predicted pixel for the boundary area. In this case, as in the first embodiment, the predicted image to be decoded has a first predicted pixel as the predicted pixel value for a corresponding area corresponding to the one division area, and a second predicted pixel as the predicted pixel value for a corresponding area corresponding to the other division area. Furthermore, a third predicted pixel as the predicted pixel value for a corresponding area corresponding to the boundary area. This makes it possible to suppress degradation of image quality in the boundary area between division areas in which different predictions are used, thereby enabling decoding of a bitstream with improved image quality.

[0127] Furthermore, in the second embodiment, three types of prediction, i.e., intra prediction, inter prediction, and intra-inter mixed prediction, have been described as examples. However, the types and number of predictions are not limited to these examples. For example, combined intra-inter prediction (CIIP) adopted in VVC may be used. In this case, the quantization matrix used for sub-blocks using combined intra-inter prediction may be the same as the quantization matrix used for sub-blocks using combined intra-inter prediction. This makes it possible to decode a bitstream in which quantization using a quantization matrix with the same quantization control properties is applied to sub-blocks using prediction methods that share the common feature of using both predicted pixels by intra prediction and predicted pixels by inter prediction within the same sub-block. Furthermore, it is also possible to decode a bitstream in which the code amount is reduced by the amount of the quantization matrix corresponding to the new prediction method.

[0128] In the second embodiment, an input image to be coded is decoded from a bit stream. However, the decoding target is not limited to an image. For example, a two-dimensional data array, which is feature data used in machine learning such as object recognition, may be decoded from a bit stream containing coded data obtained by coding the two-dimensional data array in the same manner as the input image. This enables efficient decoding of a bit stream that has coded feature data used in machine learning.

[0129] [Third embodiment] The image encoding device according to this embodiment encodes an image by performing a prediction process on a block-by-block basis. The image encoding device determines the strength of a deblocking filter process to be performed on the boundary between a first block in the image and a second block adjacent to the first block in the image, and performs deblocking filter process on the boundary according to the determined strength. The prediction process can be performed in one of the following modes: a first prediction mode (intra prediction) in which predicted pixels for the block to be encoded are derived using pixels in an image including the block to be encoded; a second prediction mode (inter prediction) in which predicted pixels for the block to be encoded are derived using pixels in an image different from the image including the block to be encoded; or a third prediction mode (mixed intra-inter prediction) in which predicted pixels for a portion of the block to be encoded are derived using pixels in an image including the block to be encoded, and predicted pixels for a portion of the block different from the portion of the block to be encoded are derived using pixels in an image different from the image including the block to be encoded. In determining the strength of the deblocking filter, the image encoding device determines the strength as the first strength when at least one of the first block and the second block is a block to which the first prediction mode is applied, and determines the strength as a strength based on the first strength when at least one of the first block and the second block is a block to which the third prediction mode is applied.

[0130] An example of the functional configuration of an image encoding device according to this embodiment will be described with reference to the block diagram of Fig. 13. In Fig. 13, the same functional units as those shown in Fig. 1 are assigned the same reference numerals, and descriptions of these functional units will be omitted.

[0131] Note that, although the transform / quantization unit 105 according to this embodiment will be described as using a predetermined quantization matrix when quantizing orthogonal transform coefficients, it may also use a quantization matrix corresponding to mixed intra / inter prediction, as in the above embodiment.

[0132] The in-loop filter unit 1309 performs in-loop filtering, such as deblocking filtering, on the image (sub-block) stored in the frame memory 108 in accordance with the filter strength (bS value) determined by the determination unit 1313. Then, the in-loop filter unit 1309 stores the image that has undergone the in-loop filtering in the frame memory 108.

[0133] The determination unit 1313 determines the filter strength (bS value) of the deblocking filter process to be performed on the boundary between two adjacent sub-blocks. More specifically, the determination unit 1313 determines the bS value, which is the filter strength of the deblocking filter process to be performed on the boundary between a sub-block P and a sub-block Q adjacent to the sub-block P, based on which of the following (Condition 1) to (Condition 6) is satisfied:

[0134] (Condition 1) If both subblocks P and Q are subblocks in a BDPCM mode that directly encodes the difference values ​​of adjacent pixels without performing conversion processing on the predicted difference image obtained by horizontal or vertical intra prediction, the bS value is set to 0. (Condition 2) If at least one of subblocks P and Q is a subblock that has undergone intra prediction or mixed intra-inter prediction, the bS value is set to 2. (Condition 3) If the boundary between sub-blocks P and Q is a boundary between sub-blocks, which are units of transform, and if the orthogonal transform coefficients of at least one of the sub-blocks include a non-zero orthogonal transform coefficient, the bS value is set to 1. (Condition 4) If the reference image for motion compensation is different between sub-block P and sub-block Q, or if the number of motion vectors is different, the bS value is set to 1. (Condition 5) If the absolute value of the difference between the motion vector in sub-block P and the motion vector in sub-block Q is 0.5 pixels or more, the bS value is set to 1. (Condition 6) If any of the conditions other than (Condition 1) to (Condition 5) is true, the bS value is set to 0. Here, deblocking filter processing is not performed on subblock boundaries (edges) where the bS value is 0, but a deblocking filter is determined based on the gradient and activity near the subblock boundary for subblock boundaries (edges) where the bS value is 1 or more. Basically, the larger the bS value, the stronger the correction strength of the deblocking filter processing that is applied.

[0135] In this embodiment, deblocking filtering is not performed on sub-block boundaries where the bS value is 0, and deblocking filtering is performed on sub-block boundaries where the bS value is 1 or more, but this is not limited to this. For example, the number of types of strengths of deblocking filtering may be greater or less.

[0136] Furthermore, the content of the processing may differ depending on the strength of the deblocking filter processing. For example, the bS value may take five levels from 0 to 4, as in the case of H.264 deblocking filter processing.

[0137] Furthermore, in this embodiment, the bS value of the deblocking filtering process for the boundaries of sub-blocks using intra- and inter-mixed prediction is the same as the bS value of the deblocking filtering process for the boundaries of sub-blocks using intra- and inter-mixed prediction. This bS value indicates 2, i.e., the maximum filter strength, but is not limited to this. For example, an intermediate bS value may be set between bS=1 (an example of another bS value) and bS=2 in this embodiment, and this intermediate bS value may be used when at least one of sub-blocks P and Q is a sub-block that has undergone intra- and inter-mixed prediction. In this case, it is possible to perform the same deblocking filtering process as when bS=2 is normally used on the luminance component, and to perform deblocking filtering with a weaker correction strength than when bS=2 on the chrominance component. This allows deblocking filtering with an intermediate correction strength to be performed on the boundaries of sub-blocks using intra- and inter-mixed prediction. Alternatively, the bS value may be determined for each boundary pixel based on whether each pixel in a sub-block using mixed intra-inter prediction is a predicted pixel by intra prediction or inter prediction. In this case, the bS value for predicted pixels by intra prediction is always 2, and the bS value for predicted pixels by inter prediction can be determined based on the above (Conditions 1) to (Condition 6).

[0138] In addition, although the present embodiment uses the bS value as the filter strength, this is not limiting. For example, other variables may be defined as the filter strength instead of the bS value, or the coefficients or filter length of the deblocking filter may be directly changed.

[0139] Next, the deblocking filter processing in the in-loop filter unit 1309 of this embodiment will be described in more detail. The deblocking filter processing is performed on the boundaries of sub-blocks, which are the units of prediction processing or transform processing. The filter length of the deblocking filter depends on the size (number of pixels) of the sub-block, and when the size of the sub-block is 32 pixels or more, the filter is applied to a maximum of seven pixels from the boundary. Similarly, when the size of the sub-block is four pixels or less, only the pixel values ​​of one pixel line adjacent to the boundary are updated.

[0140] In this embodiment, all sub-blocks are assumed to be 8x8 pixel in size and subjected to prediction and transform processing, but this is not limited thereto, and the size of the sub-block to be predicted and the size of the sub-block to be transformed may be different. For example, as in Subblock Transform (SBT) in VVC, the sub-block to which the transform processing is applied may be a sub-block that is further divided into the sub-block to which the prediction processing is applied. Alternatively, the sub-block may be larger, such as 32x32 pixels, or may not be square, such as 16x8 pixels.

[0141] 14 are adjacent sub-blocks of 8×8 pixels separated by a boundary, and also serve as units of orthogonal transform. p00 to p33 indicate pixels (pixel values) belonging to sub-block P, and q00 to q33 indicate pixels (pixel values) belonging to sub-block Q, with the pixel group of p00 to p33 and the pixel group of q00 to q33 adjacent to each other across the boundary. First, when the bS value for luminance is 1 or greater, in-loop filter unit 1309 determines whether or not to perform deblocking filter processing on the boundary between sub-block P and sub-block Q, for example, according to the following equation:

[0142] |p20-2×p10+p00|+|p23-2×p13+p03|+|q20-2×q10+q00|+|q23-2×q13+q03|<β Here, β is a value corresponding to the average value of the quantization step value in sub-block P and the quantization step value in sub-block Q. For example, among various βs registered in a table, the β registered in association with the average value is obtained. Only when this formula is satisfied, in-loop filter unit 1309 determines to perform deblocking filtering on the boundary between sub-block P and sub-block Q.

[0143] When it is determined that deblocking filtering is to be performed, in-loop filter unit 1309 determines whether to use a strong filter or a weak filter, which have different smoothing effects. For example, in-loop filter unit 1309 determines to use a strong filter if all of the following six equations (1) to (6) are satisfied. On the other hand, in-loop filter unit 1309 determines to use a weak filter if even one of the following six equations (1) to (6) is not satisfied. (1) 2×(|p20-2×p10+p00|+|q20-2×q10+q00|)<(β>>2) (2) 2×(|p23-2×p13+p03|+|q23-2×q13+q03|)<(β>>2) (3) |p30-p00|+|q00-q30|<(β>>3) (4) |p33-p03|+|q03-q33|<(β>>3) (5) |p00-q00|<((5×tc+1)>>1) (6) |p03-q03|<((5×tc+1)>>1) Here, >>N (N=1 to 3) means an N-bit arithmetic right shift operation, and tc is a parameter that determines the maximum amount of correction of the pixel value. tc can be obtained, for example, by the following process. That is, the average value qP of the quantization step value in sub-block P, the quantization step value in sub-block Q, and the bS value is calculated using the following formula: qP=qP+2x(bS-1) and obtains the tc registered in association with the corrected qP value from the various tc values ​​registered in the table. From this formula, it can be seen that when the bS value is 2, the qP value after correction will be larger. Since the table is set so that the tc value increases as the qP value increases, the larger the bS value, the more strongly a deblocking filter that corrects pixel values ​​will be applied.

[0144] Strong filtering for luminance (filtering processing using a strong filter) is expressed by the following equation (k=0 to 3), where the pixel values ​​after deblocking filter processing in sub-block P are p'0k, p'1k, p'2k, and the pixel values ​​after deblocking filter processing in sub-block Q are q'0k, q'1k, q'2k.

[0145] p'0k=Clip3(p0k-3×tc,p0k+3×tc,(p2k+2×p1k+2×p0k+2×q0k+q1k+4)>>3) p'1k=Clip3(p1k-2×tc,p1k+2×tc,(p2k+p1k+p0k+q0k+2)>>2) p'2k=Clip3(p2k-1×tc,p2k+1×tc,(2×p3k+3×p2k+p1k+p0k+q0k+4)>>3) q'0k=Clip3(q0k-3×tc,q0k+3×tc,(q2k+2×q1k+2×q0k+2×p0k+p1k+4)>>3) q'1k=Clip3(q1k-2×tc,q1k+2×tc,(q2k+q1k+q0k+p0k+2)>>2) q'2k=Clip3(q2k-1×tc,q2k+1×tc,(2×q3k+3×q2k+q1k+q0k+p0k+4)>>3) Here, Clip3(a, b, c) is a function that performs clipping processing so that the range of c is a≦c≦b. Furthermore, weak filtering for luminance (filtering processing using a weak filter) is expressed by the following formula.

[0146] Δ=(9×(q0k-p0k)-3×(q1k-p1k)+8)>>4 |Δ|<10×tc If the above conditions are not met, deblocking filter processing is not performed, but if they are met, processing is performed on p0k and q0k according to the following equations.

[0147] Δ=Clip3(-tc,tc,Δ) p'0k=Clip1(p0k+Δ) q'0k=Clip1(q0k-Δ) Here, Clip1(a) is a function that performs clipping so that the range of a is 0≦a≦(maximum value that can be expressed with the bit depth of the luminance or chrominance signal). For example, if the luminance is 8 bits (maximum value that can be expressed with the bit depth of the luminance), the value is 255, and if the luminance is 10 bits (maximum value that can be expressed with the bit depth of the luminance), the value is 1023.

[0148] Additionally, the following conditions |p20-2×p10+p00|+|p23-2×p13+p03|<(β+(β>>1))>>3) |q20-2×q10+q00|+|q23-2×q13+q03|<(β+(β>>1))>>3) When this condition is satisfied, deblocking filtering is performed on p1k and q1k according to the following formula:

[0149] Δp=Clip3(-(tc>>1),tc>>1,(((p2k+p0k+1)>>1-p1k+Δ)>>1) p'1k=Clip1(p1k+Δp) Δq=Clip3(-(tc>>1),tc>>1,(((q2k+q0k+1)>>1-q1k+Δ)>>1) q'1k=Clip1(q1k+Δq) In the deblocking filtering process for chrominance, when the filter length is 1, the deblocking filtering process is performed according to the following formula only when the bS value is 2.

[0150] Δ=Clip3(-tc,tc,(((q0k-p0k)<<2)+p1k-q1k+4)>>3)) p'0k=Clip1(p0k+Δ) q'0k=Clip1(q0k-Δ) In this embodiment, in the deblocking filtering process for chrominance, when the filter length is 1, the deblocking filtering process is performed only when the bS value is 2, but this is not limited to this. For example, the deblocking filtering process may be performed when the bS value is other than 0, or the deblocking filtering process may be performed only when the filter length is longer than 1.

[0151] In this embodiment, the bS value is used to determine whether to apply a deblocking filter and to calculate the maximum amount of pixel value correction by the deblocking filter. Separately, a strong filter with a high smoothing effect and a weak filter with a low smoothing effect are used depending on the pixel value conditions. However, this is not limited to this. For example, the filter length may be determined according to the bS value, or only the strength of the smoothing effect may be determined by the bS value.

[0152] The encoding process performed by the image encoding device described above will be described with reference to the flowchart in Fig. 15. Note that the process according to the flowchart in Fig. 15 is an encoding process for one input image. Therefore, when encoding each frame image in a moving image or multiple images captured periodically or irregularly, the process according to the flowchart in Fig. 15 is repeatedly performed for each image.

[0153] In step S1501, the integral encoding unit 111 encodes various header information necessary for encoding an input image to generate header code data.

[0154] In step S1502, the division unit 102 divides the input image into a plurality of basic blocks and outputs each of the divided basic blocks. The prediction unit 104 then divides each basic block into a plurality of sub-blocks.

[0155] In step S1503, the prediction unit 104 selects one of the unselected sub-blocks in the input image as a selected sub-block, determines a prediction mode for the selected sub-block, and performs prediction on the selected sub-block according to the determined prediction mode to obtain a predicted image, prediction error, and prediction information for the selected sub-block.

[0156] In step S1504, the transform / quantization unit 105 performs an orthogonal transform process on the prediction error of the selected sub-block obtained in step S1503 to generate orthogonal transform coefficients, and quantizes the orthogonal transform coefficients using a quantization matrix to obtain quantized coefficients.

[0157] In step S1505, the inverse quantization and inverse transform unit 106 generates orthogonal transform coefficients by inverse quantizing the quantized coefficients of the selected sub-block acquired in step S1504 using the quantization matrix described above. The inverse quantization and inverse transform unit 106 then performs inverse orthogonal transform on the generated orthogonal transform coefficients to generate (reproduce) prediction errors.

[0158] In step S1506, the image reproduction unit 107 generates a predicted image from the image stored in the frame memory 108 based on the prediction information acquired in step S1503, and regenerates an image of the sub-block by adding the predicted image to the prediction error generated in step S1505. The image reproduction unit 107 then stores the regenerated image in the frame memory 108.

[0159] In step S1507, the encoding unit 110 performs entropy encoding on the quantized coefficients acquired in step S1504 and the prediction information acquired in step S1503 to generate encoded data.

[0160] The integrated encoding unit 111 then generates a bit stream by multiplexing the header encoded data generated in step S1501 and the encoded data generated by the encoding unit 110 in step S1507.

[0161] In step S1508, control unit 150 determines whether all sub-blocks in the input image have been selected as selected sub-blocks. If the result of this determination is that all sub-blocks in the input image have been selected as selected sub-blocks, the process proceeds to step S1509. On the other hand, if one or more sub-blocks in the input image remain that have not yet been selected as selected sub-blocks, the process proceeds to step S1503.

[0162] In step S1509, the determination unit 1313 determines a bS value for each boundary between adjacent sub-blocks. Then, in step S1510, the in-loop filter unit 1309 performs in-loop filtering, such as deblocking filtering, with a filter strength based on the bS value determined in step S1509, on the image stored in the frame memory 108. More specifically, the in-loop filter unit 1309 performs in-loop filtering, such as deblocking filtering, with a filter strength based on the bS value determined by the determination unit 1313 for the boundary between sub-blocks that form the unit of orthogonal transform of the image stored in the frame memory 108. Then, the in-loop filter unit 109 stores the image that has undergone in-loop filtering in the frame memory 108.

[0163] As described above, according to this embodiment, a deblocking filter with a high distortion correction effect can be set for the boundaries of sub-blocks using mixed intra-inter prediction, thereby suppressing block distortion and improving image quality. Furthermore, since no new calculations are required to calculate the filter strength, the implementation complexity is not increased.

[0164] In this embodiment, the filter strength is used to determine whether to apply a deblocking filter and to calculate the maximum amount of correction of pixel values ​​by the deblocking filter. However, this is not limited to this. For example, when the filter strength is larger, a deblocking filter with a longer tap length and a higher correction effect can be used, and when the filter strength is smaller, a deblocking filter with a shorter tap length and a lower correction effect can be used.

[0165] Furthermore, in this embodiment, only three types of prediction, i.e., intra prediction, inter prediction, and intra-inter mixed prediction, are used. However, this is not limiting. For example, combined intra-inter prediction (CIIP) adopted in VVC may also be used. In this case, the bS value used for sub-blocks using combined intra-inter prediction may be set to the same bS value as when combined intra-inter prediction is used. This allows bitstream encoding to be performed using a deblocking filter of the same strength for sub-blocks using predictions that have in common the use of both intra-predicted pixels and inter-predicted pixels within the same sub-block.

[0166] [Fourth embodiment] The image decoding device according to this embodiment decodes an encoded image for each block. This image decoding device performs a prediction process for each block to decode the image. The image decoding device determines the strength of a deblocking filter process to be performed on the boundary between a first block and a second block adjacent to the first block, and performs a deblocking filter process on the boundary according to the determined strength. The image decoding device uses, as the prediction process, one of the following: a first prediction mode (intra prediction) in which predicted pixels for the block to be decoded are derived using pixels in an image including the block to be decoded; a second prediction mode (inter prediction) in which predicted pixels for the block to be decoded are derived using pixels in an image other than the image including the block to be decoded; or a third prediction mode (mixed intra-inter prediction) in which predicted pixels for a portion of the block to be decoded are derived using pixels in an image including the block to be decoded, and predicted pixels for a portion of the block other than the portion of the block to be decoded are derived using pixels in an image other than the image including the block to be decoded. Here, when determining the strength of the deblocking filter process, if at least one of the first block and the second block is a block to which the first prediction mode is applied, the strength is determined to be the first strength, and if at least one of the first block and the second block is a block to which the third prediction mode is applied, the strength is determined to be the strength based on the first strength.

[0167] In this embodiment, an image decoding device that decodes a bitstream encoded by the image encoding device according to the third embodiment will be described. First, an example of the functional configuration of the image decoding device according to this embodiment will be described using the block diagram of Fig. 16. In Fig. 16, the same reference numerals are assigned to the same functional units as those shown in Fig. 2, and descriptions of these functional units will be omitted.

[0168] The in-loop filter unit 1607 performs in-loop filtering, such as deblocking filtering, with a filter strength corresponding to the bS value determined for the sub-block boundary by the determination unit 1609, on the sub-block boundary of the reconstructed image stored in the frame memory 206. The determination unit 1609 determines the filter strength (bS value) of the deblocking filtering to be performed on the boundary between two adjacent sub-blocks, in the same manner as the determination unit 1313.

[0169] In this embodiment, as in the third embodiment, the bS value is used to determine whether to apply a deblocking filter and to calculate the maximum amount of pixel value correction by the deblocking filter. Separately, a strong filter with a high smoothing effect and a weak filter with a low smoothing effect are used depending on the pixel value conditions. However, this is not limited to this. For example, the filter length may be determined according to the bS value, or only the strength of the smoothing effect may be determined by the bS value.

[0170] The decoding process in the image decoding device according to this embodiment will be described with reference to the flowchart in Fig. 17. In step S1701, the separate decoding unit 202 acquires a bitstream. The separate decoding unit 202 then separates coded data of the input image from the bitstream, supplies the coded data to the decoding unit 203, and decodes header coded data in the bitstream.

[0171] In step S1702, the decoding unit 203 decodes the coded data supplied from the demultiplexing decoding unit 202 to reproduce the quantized coefficients and prediction information of the sub-block to be decoded.

[0172] In step S1703, the inverse quantization and inverse transform unit 204 inverse quantizes the quantized coefficients of the sub-block to be decoded using a quantization matrix to regenerate orthogonal transform coefficients. The inverse quantization and inverse transform unit 204 then performs inverse orthogonal transform on the regenerated orthogonal transform coefficients to regenerate prediction errors of the sub-block to be decoded, and supplies the regenerated prediction errors to the image regeneration unit 205.

[0173] In step S1704, the image reproduction unit 205 generates a predicted image of the sub-block to be decoded by referring to the image stored in the frame memory 206 based on the prediction information decoded by the decoding unit 203. The image reproduction unit 205 then generates a reconstructed image of the sub-block to be decoded by adding the prediction error obtained by the inverse quantization and inverse transform unit 204 to the generated predicted image, and stores the generated reconstructed image in the frame memory 206.

[0174] In step S1705, control unit 250 determines whether the processes of steps S1702 to S1704 have been performed for all sub-blocks. If the result of this determination is that the processes of steps S1702 to S1704 have been performed for all sub-blocks, the process proceeds to step S1706. On the other hand, if there are still sub-blocks for which the processes of steps S1702 to S1704 have not been performed, the process proceeds to step S1702 to perform the processes of steps S1702 to S1704 for those sub-blocks.

[0175] In step S1706, the determining unit 1609 determines the filter strength of the deblocking filtering process to be performed on the boundary between two adjacent sub-blocks, in the same manner as the determining unit 1313 described in the third embodiment. Since the type of prediction applied to each sub-block (intra prediction, inter prediction, or mixed intra-inter prediction) is recorded in the prediction information, the prediction applied to each sub-block can be identified by referring to the prediction information.

[0176] In step S1707, the in-loop filter unit 1607 performs in-loop filtering, such as deblocking filtering, on the sub-block boundaries of the reconstructed image stored in the frame memory 206, with a filter strength according to the bS value determined for the sub-block boundaries by the determination unit 1609.

[0177] In this way, according to this embodiment, an appropriate deblocking filter can be applied in decoding a bitstream including "sub-blocks coded using mixed intra-inter prediction" generated by the image coding device according to the third embodiment.

[0178] [Fifth embodiment] Each of the functional units shown in FIGS. 1, 2, 13, and 16 may be implemented by hardware, or each of the functional units except for the holding unit 103 and the frame memories 108 and 206 may be implemented by software (computer program).

[0179] In the former case, such hardware may be a circuit incorporated into a device that encodes and decodes images, such as an imaging device, or may be a circuit incorporated into a device that encodes and decodes images supplied from an external device, such as an imaging device or a server device.

[0180] In the latter case, such a computer program may be stored in a memory of a device that encodes and decodes images, such as an imaging device, or in a memory accessible by a device that encodes and decodes images supplied from an external device, such as an imaging device or a server device. A device (computer) that can read and execute such a computer program from a memory is applicable to the above-mentioned image encoding device and the above-mentioned image decoding device. An example of the hardware configuration of such a computer will be described using the block diagram of FIG.

[0181] The CPU 501 executes various processes using computer programs and data stored in the RAM 502 and the ROM 503. As a result, the CPU 501 controls the operation of the entire computer device, and also executes or controls various processes that have been described as being executed by the image encoding device and image decoding device in the above-described embodiments and modifications.

[0182] The RAM 502 has an area for storing computer programs and data loaded from an external storage device 506, and an area for storing data acquired from the outside via an I / F (interface) 507. The RAM 502 also has a work area (frame memory, etc.) used when the CPU 501 executes various processes. In this way, the RAM 502 can provide various areas as needed.

[0183] The ROM 503 stores setting data for the computer device, computer programs and data relating to the startup of the computer device, computer programs and data relating to the basic operation of the computer device, and the like.

[0184] The operation unit 504 is a user interface such as a keyboard, mouse, or touch panel, and allows the user to input various instructions to the CPU 501 by operating it.

[0185] Display unit 505 has a liquid crystal screen or a touch panel screen, and displays the processing results by CPU 501 as images, text, etc. Note that display unit 505 may also be a projection device such as a projector that projects images and text.

[0186] The external storage device 506 is a large-capacity information storage device such as a hard disk drive. The external storage device 506 stores an OS (operating system), computer programs and data for causing the CPU 501 to execute the various processes described above as being performed by the image encoding device and the image decoding device, etc. The external storage device 506 also stores information (encoding tables, tables, etc.) that has been treated as known information in the above description. The external storage device 506 may also store data to be encoded (input images, two-dimensional data arrays, etc.).

[0187] Computer programs and data stored in the external storage device 506 are loaded into the RAM 502 as appropriate under the control of the CPU 501, and are then processed by the CPU 501. The above-mentioned holding unit 103 and frame memories 108 and 206 can be implemented using the RAM 502, ROM 503, the external storage device 506, etc.

[0188] I / F507 can be connected to networks such as a LAN or the Internet, and other devices such as a projection device or display device, and this computer device can acquire and send various information via I / F507.

[0189] The CPU 501 , RAM 502 , ROM 503 , operation unit 504 , display unit 505 , external storage device 506 , and I / F 507 are all connected to a system bus 508 .

[0190] In the above configuration, when the computer is powered on, the CPU 501 executes a boot program stored in the ROM 503, loads the OS stored in the external storage device 506 into the RAM 502, and starts up the OS. As a result, the computer becomes capable of communication via the I / F 507. Under the control of the OS, the CPU 501 loads an encoding application from the external storage device 506 into the RAM 502 and executes it, thereby functioning as the functional units (excluding the storage unit 103 and the frame memory 108) shown in FIGS. 1 and 13 . In other words, the computer functions as the image encoding device described above. Meanwhile, under the control of the OS, the CPU 501 loads a decoding application from the external storage device 506 into the RAM 502 and executes it, thereby functioning as the functional units (excluding the frame memory 206) shown in FIGS. 2 and 16 . In other words, the computer functions as the image decoding device described above.

[0191] In this embodiment, it has been described that a computer device having the configuration shown in Fig. 5 is applicable to an image encoding device or an image decoding device. However, the hardware configuration of a computer device applicable to an image encoding device or an image decoding device is not limited to the hardware configuration shown in Fig. 5. Furthermore, the hardware configuration of a computer device applied to an image encoding device and the hardware configuration of a computer device applied to an image decoding device may be the same as or different from each other.

[0192] Furthermore, the numerical values, processing timing, processing order, processing subject, destination / source / storage location of data (information) used in each of the above embodiments and each of the modified examples are given as examples to provide a concrete explanation, and are not intended to be limited to such examples.

[0193] Furthermore, some or all of the above-described embodiments and modifications may be used in appropriate combination, and some or all of the above-described embodiments and modifications may be used selectively.

[0194] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0195] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0196] 102: Division unit 104: Prediction unit 105: Transformation and quantization unit 106: Inverse quantization and inverse transformation unit 107: Image reproduction unit 108: Frame memory 1309: In-loop filter unit 110: Encoding unit 111: Integrated encoding unit 1313: Determination unit 150: Control unit

Claims

1. An encoding means for encoding an image by performing a prediction process on a block-by-block basis; a setting means for setting a strength of a deblocking filter process to be performed on a boundary between a first block in the image and a second block adjacent to the first block in the image; a processing means for performing a deblocking filter process on the boundary in accordance with the strength set by the setting means; Equipped with The encoding means performs the prediction process for the block to be encoded, a first prediction mode for deriving a predicted pixel of the block to be coded using pixels in an image including the block to be coded; a second prediction mode in which a predicted pixel of the block to be encoded is derived using pixels in an image other than the image including the block to be encoded; a third prediction mode in which predicted pixels are derived for a partial area of ​​the block to be coded using pixels in an image including the block to be coded, and predicted pixels are derived for another area different from the partial area of ​​the block to be coded using pixels in an image different from the image including the block to be coded; [0033] It is possible to use one of a number of prediction modes, including The setting means is When at least one of the first block and the second block is a block to which the first prediction mode is applied, a first strength is set as the strength related to the deblocking filtering process for the boundary; When at least one of the first block and the second block is a block to which the third prediction mode is applied, a second strength is set as the strength related to the deblocking filtering process for the boundary.

1. An image encoding device comprising:

2. The image encoding device as described in Claim 1, characterized in that the second intensity is identical to the first intensity.

3. 2. The image encoding device according to claim 1, wherein the second intensity is an intensity between the first intensity and another intensity.

4. 2. The image encoding device of claim 1, wherein when at least one of the first block and the second block is a block to which the third prediction mode is applied, the second intensity is set to be the same as the first intensity for predicted pixels obtained by intra prediction at the boundary, and the second intensity is set to be a strength smaller than the first intensity for predicted pixels obtained by inter prediction at the boundary.

5. 5. The image encoding device according to claim 1, wherein the setting means sets an intensity for a luminance component and an intensity for a color difference component.

6. The image encoding device according to claim 1 , wherein the intensity is a bS value.

7. The setting means is When at least one of the first block and the second block is a block to which the first prediction mode is applied, a coefficient of a deblocking filter having a correction strength of the first strength is set; When at least one of the first block and the second block is a block to which the third prediction mode is applied, a coefficient of a deblocking filter having a correction strength of the second strength is set.

2. The image encoding device according to claim 1 .

8. The setting means is When at least one of the first block and the second block is a block to which the first prediction mode is applied, a filter length of a deblocking filter having a correction strength of the first strength is set; When at least one of the first block and the second block is a block to which the third prediction mode is applied, a filter length of a deblocking filter having a correction strength of the second strength is set.

2. The image encoding device according to claim 1 .

9. An image decoding device that decodes an encoded image for each block, comprising: A decoding means for decoding the image by performing a prediction process for each block; a setting means for setting a strength of a deblocking filter process to be performed on a boundary between a first block in the image and a second block adjacent to the first block in the image; a processing means for performing a deblocking filter process on the boundary in accordance with the strength set by the setting means; Equipped with The decoding means performs the prediction process for the block to be decoded, a first prediction mode in which a predicted pixel of the block to be decoded is derived using pixels in an image including the block to be decoded; a second prediction mode in which a predicted pixel of the block to be decoded is derived using pixels in an image other than the image including the block to be decoded; a third prediction mode in which predicted pixels are derived for a partial area of ​​the block to be decoded using pixels in an image including the block to be decoded, and predicted pixels are derived for another area different from the partial area of ​​the block to be decoded using pixels in another image different from the image including the block to be decoded; [0033] It is possible to use one of a number of prediction modes, including The setting means is When at least one of the first block and the second block is a block to which the first prediction mode is applied, a first strength is set as the strength related to the deblocking filtering process for the boundary; When at least one of the first block and the second block is a block to which the third prediction mode is applied, a second strength is set as the strength related to the deblocking filtering process for the boundary.

2. An image decoding device comprising:

10. The image decoding device of claim 9, wherein the second intensity is identical to the first intensity.

11. The image decoding device described in Claim 9, characterized in that the second intensity is an intensity between the first intensity and another intensity.

12. The image decoding device of claim 9, characterized in that when at least one of the first block and the second block is a block to which the third prediction mode is applied, the second intensity is set to be the same as the first intensity for predicted pixels obtained by intra prediction at the boundary, and the second intensity is set to be an intensity smaller than the first intensity for predicted pixels obtained by inter prediction at the boundary.

13. An encoding step of encoding an image by performing a prediction process on a block-by-block basis; A setting step of setting a strength of a deblocking filter process to be performed on a boundary between a first block in the image and a second block adjacent to the first block in the image; a processing step of performing a deblocking filter process on the boundary according to the strength set in the setting step; Equipped with In the encoding step, the prediction process for the block to be encoded includes the steps of: a first prediction mode for deriving a predicted pixel of the block to be coded using pixels in an image including the block to be coded; a second prediction mode in which a predicted pixel of the block to be encoded is derived using pixels in an image other than the image including the block to be encoded; a third prediction mode in which predicted pixels are derived for a partial area of ​​the block to be coded using pixels in an image including the block to be coded, and predicted pixels are derived for another area different from the partial area of ​​the block to be coded using pixels in an image different from the image including the block to be coded; [0033] It is possible to use one of a number of prediction modes, including In the setting step, When at least one of the first block and the second block is a block to which the first prediction mode is applied, a first strength is set as the strength related to the deblocking filtering process for the boundary; When at least one of the first block and the second block is a block to which the third prediction mode is applied, a second strength is set as the strength related to the deblocking filtering process for the boundary.

13. An image coding method comprising:

14. 1. An image decoding method for decoding an encoded image for each block, comprising the steps of: a decoding step of decoding the image by performing a prediction process for each block; A setting step of setting a strength of a deblocking filter process to be performed on a boundary between a first block in the image and a second block adjacent to the first block in the image; a processing step of performing a deblocking filter process on the boundary according to the strength set in the setting step; Equipped with In the decoding step, the prediction process for the block to be decoded is a first prediction mode in which a predicted pixel of the block to be decoded is derived using pixels in an image including the block to be decoded; a second prediction mode in which a predicted pixel of the block to be decoded is derived using pixels in an image other than the image including the block to be decoded; a third prediction mode in which predicted pixels are derived for a partial area of ​​the block to be decoded using pixels in an image including the block to be decoded, and predicted pixels are derived for another area different from the partial area of ​​the block to be decoded using pixels in another image different from the image including the block to be decoded; [0033] It is possible to use one of a number of prediction modes, including In the setting step, When at least one of the first block and the second block is a block to which the first prediction mode is applied, a first strength is set as the strength related to the deblocking filtering process for the boundary; When at least one of the first block and the second block is a block to which the third prediction mode is applied, a second strength is set as the strength related to the deblocking filtering process for the boundary.

2. An image decoding method comprising:

15. A computer program for causing a computer to function as each of the means of the image encoding device according to any one of claims 1 to 8.

16. A computer program for causing a computer to function as each of the means of the image decoding device according to any one of claims 9 to 12.

Images